--- title: "Trace voice agents with Datadog Agent Observability" description: "Use Agent Observability to replay voice interactions and investigate latency, cost, interruptions, and transcripts across OpenAI Realtime sessions." author: "Zach Groves, Will Potts" date: 2026-10-29 tags: ["agent observability", "openai", "amazon nova", "ai"] blog_type_id: the-monitor locale: en --- A voice agent can return the right words and still deliver a poor experience. It might leave a caller in silence, speak over them, or stop midway through a response. A transcript may look correct even when the audio the user heard was delayed, incomplete, or different from the text on screen. These problems are difficult to diagnose from text traces alone. Voice conversations also involve more than a single request and response. Audio streams, transcripts, model interactions, and tool calls all contribute to each turn. To understand why a conversation failed, teams need to examine those signals together and in the order the user experienced them. [Datadog Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/) brings audio into traces for supported voice agents. With these signals in one trace, you can listen to captured audio and connect a user-visible problem to the model or application behavior behind it. In this post, we'll show how you can: - [Listen to captured voice turns alongside trace data](#listen-to-captured-voice-turns-alongside-trace-data) - [Find slow or high-usage voice interactions](#find-slow-or-high-usage-voice-interactions) - [Investigate interruptions and incomplete transcripts](#investigate-interruptions-and-incomplete-transcripts) - [Start tracing supported voice agents](#start-tracing-supported-voice-agents) ## Listen to captured voice turns alongside trace data A transcript tells you what text was recorded, but not always what the caller heard. The agent might have selected the right response but delivered it after an uncomfortable pause. A user might have interrupted the response, or the generated audio might differ from what the transcript suggests. Investigating these cases requires the audio and the surrounding agent activity in one place. For supported applications built with the [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime), automatic instrumentation into Agent Observability captures each turn's input and output audio, transcripts, token usage, and tool calls. Datadog converts raw PCM16 and G.711 audio to WAV so you can play it directly from the trace. Agent Observability also adds tracing support for applications built with [Amazon Nova 2 Sonic](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-amazon-nova-2-sonic.html) in the Python and Node.js tracers. From an affected turn, you can listen to the exchange while inspecting its transcript and tool calls. This helps you determine whether the agent chose the wrong action, returned unexpected content, or was interrupted before it finished speaking. ![Screenshot of an Agent Observability trace where a user’s recorded audio plays above its transcript and the tool result that followed.](https://web-assets.dd-static.net/42588/1793111971-tool-results-on-span_multi_modal_fa-1.png) Audio also preserves details that a transcript cannot, including pauses, overlapping speech, and the point at which a response was interrupted. Viewing those details with the rest of the trace lets you investigate the conversation as the user experienced it instead of treating each model response as an isolated text completion. ## Find slow or high-usage voice interactions Response latency is immediately noticeable in a spoken conversation. A delay that might be tolerable in a text interface can leave a caller waiting in silence. Uneven audio delivery can also make an otherwise correct response difficult to follow. For OpenAI Realtime applications, Datadog's automatic instrumentation emits the `ml_obs.trace.time_to_first_agent_audio distribution` metric, which measures how long a user waits for the agent's audio response. Charting this metric over time can show whether slow initial responses are isolated or widespread. When percentile aggregations are enabled for the metric, you can also compare values such as p50, p95, and p99. For more information, see the [Datadog distribution metrics documentation](https://docs.datadoghq.com/metrics/distributions.md). When the metric shows a latency increase, use the “Average time to first agent audio” facet in Agent Observability to scope the span list to turns with the slowest initial audio responses. You can then open an affected trace and inspect the surrounding turns and tool calls to determine whether the delay is associated with a model interaction, a downstream action, or another part of the conversation. ![Screenshot of the Agent Observability Traces list filtered to voice turns where time to first agent audio is at least 360 ms.](https://web-assets.dd-static.net/42588/1793112003-time-to-first-agent_multi_modal_fa-1.png) Voice agents can also have different usage patterns than text-only agents have because providers count audio and text tokens separately. Agent Observability captures token usage from OpenAI Realtime interactions, so you can review high-usage turns alongside latency and other operational signals. Opening the associated traces helps you determine whether higher usage is connected to longer conversations, particular workflows, agent routing, or external tool calls. ## Investigate interruptions and incomplete transcripts Interruptions are part of natural conversation, but repeated interruptions can also point to a problem. An agent might give answers that are too long or that begin with irrelevant information, or it might rely on voice activity detection that mistakes a pause for the end of the user's turn. Voice traces preserve the sequence of events around these interactions. When a user interrupts an OpenAI Realtime response, you can open the corresponding turn and listen to what the agent was saying at that moment. The surrounding context helps you distinguish normal turn-taking from a pattern that warrants further investigation. ![Screencast of a voice session timeline marking when the user, agent, or both are speaking, with each turn’s transcript below.](https://web-assets.dd-static.net/42588/1793112043-session-to-trace_multi_modal_fa.mp4) Audio context is also important when you review transcripts. OpenAI Realtime input transcription runs separately from the model's processing of the audio, and an output transcript can be incomplete when a user interrupts a response. Comparing the transcript with the corresponding audio helps prevent an incomplete transcript from being treated as a complete record of the interaction. Across traces, these signals help you identify conversations that need closer inspection and determine whether a problem originated in the model response, tool execution, turn-taking behavior, or another part of the application. ## Start tracing supported voice agents Datadog provides automatic multimodal instrumentation for supported voice applications through the Agent Observability SDK. Python applications that use the [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime) with `ddtrace 4.13.0` or later can automatically capture per-turn input and output audio, transcripts, token usage, and tool calls without manually constructing audio spans. [ElevenLabs Realtime API](https://elevenlabs.io/docs/api-reference/speech-to-text/v-1-speech-to-text-realtime) support will be available soon with `ddtrace 4.17.0`. Agent Observability also adds tracing support for applications built with [Amazon Nova 2 Sonic](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-amazon-nova-2-sonic.html) in the Python and Node.js tracers. To configure Agent Observability, follow the [Agent Observability instrumentation documentation](https://docs.datadoghq.com/llm_observability/instrument.md). For supported audio formats, SDK versions, automatic OpenAI Realtime capture, and media size limits, see the [multimodal instrumentation documentation](https://docs.datadoghq.com/llm_observability/instrument/multimodal.md). Applications that already emit spans that comply with the OpenTelemetry GenAI semantic conventions (version 1.37 or later) can also send supported audio message parts to Agent Observability. Datadog's SDK provides the automatic OpenAI Realtime instrumentation described in this post. To learn about the OpenTelemetry path and its supported GenAI message structure, see the [Agent Observability OpenTelemetry instrumentation documentation](https://docs.datadoghq.com/llm_observability/instrument/otel_instrumentation.md). ## Bring voice context into your agent investigations Agent Observability connects the audio a user experiences with the model, tools, latency, and token usage behind the conversation. Instead of reconstructing an exchange from separate transcripts, metrics, and logs, you can move from a latency signal or reported problem to the specific turn and trace that produced it. To get started, read the [Agent Observability documentation](https://docs.datadoghq.com/llm_observability.md) and the [multimodal instrumentation documentation](https://docs.datadoghq.com/llm_observability/instrument/multimodal.md). If you're new to Datadog, [start with Agent Observability for free](https://www.datadoghq.com/products/ai/agent-observability/). The free plan includes up to 40,000 LLM spans per month.