Agent Observation Is the Second Half

For the past two years, the agentic AI race has been a race to build. Claude Code, Codex, Cursor, Hermes Agent, OpenClaw, OpenCode — long-running agent harnesses have become astonishingly capable. They connect LLMs to tools, files, messaging apps, and the broader internet. They run 24/7, get triggered by events, execute on schedules. They browse the web, manage calendars, spawn sub-agents, and chain complex multi-step tasks entirely on their own.

That first half — agent build — is rapidly becoming a solved problem. Pick a harness, plug in a frontier model, add skills and MCP servers, and you have a working agent in an afternoon. The marginal cost of standing up a capable agent is collapsing toward zero.

Which is exactly why it's no longer where the game will be decided.

Here is the uncomfortable truth about every agent in production today: it runs inside a loop you can't easily see into.

Ask the operator of a long-running agent some basic questions:

• Which tools did the agent actually use for this task?
• How many tokens did it burn, and on which step?
• How long did each tool call take? Where is the bottleneck in the loop?
• Which skills did it load? Which sub-agents did it spawn, and what did they do?
• What did it read from memory? And how do I lower the API bill?

Most teams cannot answer these questions today. There is no visibility into the agent's execution lifecycle. And that has hard consequences: you cannot debug failures without telemetry, and you cannot optimize or accelerate what you cannot observe.

We have seen this movie before. In the web and mobile era, the first half was shipping apps; the second half — the part that decided winners — was analytics, APM, and observability. Datadog was built on that second half. Agents are at the same inflection point now, with one difference: an agent's "request" is not a stateless HTTP call. It is a sprawling, branching trajectory of LLM calls, tool invocations, skill loads, memory accesses, and recursive sub-agents. The old observability toolbox doesn't natively speak that language.

The unit of value in the second half is the agent trajectory: the complete, structured record of everything an agent did during a run, with parent-child lineage intact.

Trajectory data is not just for pretty dashboards. It is the single upstream input for essentially everything you want to do with an agent after it ships:

• Observability & debugging — understanding runtime behaviour and failure modes.
• Evaluation — replaying and scoring real runs instead of synthetic benchmarks.
• Optimization (offline) — finding bottlenecks, wasted tokens, and redundant calls in the loop.
• Acceleration (online) — caching, routing, and hinting based on what previous runs revealed.
• Training — trajectories are exactly the shape of data that agentic RL and distillation pipelines want.

The problem: every harness records its own idea of a trace, in its own format, through its own hooks — when it records anything at all. Claude Code, Codex, Cursor, and Hermes each need the same thing: one shared layer for capturing sessions, model calls, tool calls, errors, and replay — and exporting the telemetry in standard formats. That layer didn't exist. This is the gap NeMo Relay was built to fill.

NVIDIA NeMo Relay is a Rust-based agent runtime layer that runs alongside your agent harness, sees the traffic that matters — LLM calls and tool calls — and gives you visibility and control over agent runs without requiring changes to the existing agent stack.

The positioning is deliberate. Relay does not replace your framework, your model provider, your observability backend, or your guardrail system. A production agent stack today might combine LangGraph, provider SDKs, custom harness code, NeMo Guardrails, a tracing backend, and an eval pipeline — and every one of those layers invents its own wrappers and its own trace vocabulary. Relay gives them one runtime contract: a common boundary where they all meet.

Under the hood, Relay is built from a small set of primitives. Rust is the source of truth for runtime behaviour; the Python and Node.js bindings mirror the same semantics.

Scopes — execution context with lineage

Everything in a run lives inside a hierarchy of scopes: runs, turns, tool calls, LLM calls, and sub-agents each get clear ownership, parent-child lineage, cleanup boundaries, and request isolation. This is what makes recursive sub-agent trees — the thing classic tracing struggles with — first-class citizens.

Managed LLM and tool calls

Call boundaries route through consistent lifecycle helpers, so the same lifecycle and middleware rules apply around every callback, no matter which framework or provider owns the actual execution. Scheduling, retries, memory, and result handling stay in the owning application.

Middleware — where policy meets execution

Middleware is the interception point where Relay can block, sanitize, transform, route, retry, or replace execution. Guardrails live here (pre-tool blocking), and so do payload sanitizers and routing logic. Observation and control share one mechanism.

Plugins, events, and subscribers

Reusable behavior — observability exporters, guardrails, adaptive optimization, PII redaction — ships as plugins enabled from configuration, not code changes. Everything emits into one canonical lifecycle event stream, and every output format is derived from that same stream by subscribers and exporters.

Relay's answer to "which trace format?" is: capture once, project into whatever your downstream tools speak.

The design point worth internalizing: raw ATOF events are the ground truth, and everything else is a projection. When a downstream trace looks wrong, you can always diff it against the raw event stream instead of guessing what the exporter did.

1 · The CLI: observe an agent you didn't write

The fastest path. The Relay CLI is a transparent wrapper: it starts a local Relay gateway, injects host-specific hook and provider settings for the launched process, and shuts the gateway down when the agent exits. No changes to the agent, no code — wrap the coding agent you already use, and the raw canonical events plus the normalized trajectory land on disk.

Claude Code, Codex, and Hermes are supported today for observability, with hook-backed security (pre-tool blocking) where the host exposes blocking hooks.

2 · The SDK: instrument code you own

If you're writing the application that calls models and tools, route those boundaries through Relay directly, in Python, Node.js, or Rust — open a scope around your agent, emit events at the LLM and tool boundaries, and register subscribers to consume the stream.

3 · Prebuilt integrations: frameworks

For LangChain, LangGraph, and Deep Agents, Relay ships integrations that wrap tool and LLM calling with full observability, security, and optimization support. OpenClaw is covered through hook-backed telemetry with pre-tool guardrails.

A fair objection: agents are already slow, and now you want another layer in the path. The relevant question is not just overhead — it's how quickly a finished run becomes usable telemetry. Agent completion does not guarantee trace availability. We measured two things:

• Visible latency — time until a run's trace can first be queried in the backend.
• Complete latency — time until the expected observation count has fully arrived.

In a benchmark we ran in July 2026 — same agent, same model, same Langfuse backend, 4 benchmarks × 20 tasks, in serial and 20-way concurrent regimes — we compared two telemetry paths into that shared backend: a transcript-hook path (after each response: transcript delta → rebuild → flush OTel spans) versus the Relay path (capture LLM and tool boundaries during execution, export spans on close):

Every reported Relay mean stayed under 0.9 seconds, across all eight benchmark/regime combinations. The gap is architectural rather than a tuning artifact: a hook that reconstructs transcripts after each response batches work at the end of the pipeline, while a runtime that captures events at the call boundary has essentially already done the work when the run closes. Sub-second trace availability is what makes observation usable inside operational loops — watch-dogs, live evals, budget guards — rather than as after-the-fact archaeology.

If Relay stopped at trace collection it would still be useful, but the more interesting part is that observation and control share the same substrate. Because Relay sits at the call boundary, it can act, not just record:

• Adaptive optimization — the nemo-relay-adaptive crate lets the runtime learn from observed executions, inject runtime hints, persist adaptive state, and cache repeated LLM responses.
• Routing — integration with NeMo Switchyard's Decision API adds routing-aware LLM execution intercepts, while Relay keeps ownership of credentials, dispatch, retries, and fallbacks.
• Privacy — a first-party pii-redaction plugin applies deterministic, config-driven redaction to tool and LLM observability payloads, instead of every team hand-writing sanitize middleware.
• Guardrails — middleware hooks give guardrail systems a real enforcement point: block a tool call before it runs, not flag it afterwards.

That's the full second-half flywheel on one substrate: observe → evaluate → optimize → accelerate — collecting trajectory data not as an archive but as the fuel for making agent harnesses faster and cheaper, run after run.

In the first half, the moat was the harness. In the second half, the harness is table stakes — the moat is what you learn from every run. Teams that treat agent runs as ephemeral executions will keep debugging by vibes and paying API bills they can't explain. Teams that capture standardized trajectories turn every run into a compounding asset: eval sets from production, caches that hit, routers that learn, RL data that's already in the right shape.

Agent build was the first half. Agent observation is the second half — and it has already started.

github.com/NVIDIA/NeMo-Relay · docs.nvidia.com/nemo/relay

Latency numbers are from internal comparison runs (July 2026) by Yuxiao Yi, Jinsong Zhou, and Harvey Li, measuring two telemetry paths into one shared Langfuse backend. NeMo Relay is open source under Apache-2.0.