--- title: "Compare AI agent changes with Online Experiments in Agent Observability" description: "Learn how Datadog Online Experiments, available through Agent Observability, enables you to test AI agent changes against real production traffic so you can compare response quality, cost, latency, and errors." author: "Alex Barksdale, Yakov Shapiro, Demetri Pananos, Miguel Tulla Lizardi" date: 2026-11-04 tags: ["agent observability", "experiments", "ai", "testing"] blog_type_id: the-monitor locale: en --- Improving an AI agent often involves changing its prompts, models, retrieval methods, or tools. But determining whether those changes actually improve performance can be difficult. A revised prompt might produce better answers for a handful of test cases but perform poorly on unexpected user requests. A cheaper model might reduce token costs while introducing tool-calling errors or longer execution times. Before rolling out a change, teams need to understand how it affects agent quality, reliability, and cost under real usage. Datadog Online Experiments, available in [Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/), enables teams to compare agent variants by using live production traffic and measure how a change impacts agent behavior and performance. By bringing [agent evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md) together with [Datadog Experiments](https://www.datadoghq.com/product/experiments/), Online Experiments helps you assess response quality alongside operational metrics such as latency, cost, token usage, and errors. You can also investigate the underlying traces to understand why an agent change caused variations in behavior. In this post, we’ll walk you through: - [The differences between offline and online experiments](#offline-versus-online-experiments-for-ai-agents) - [How Datadog Online Experiments helps you compare agent variants using real production traffic](#compare-agent-variants-using-real-production-traffic-with-datadog-online-experiments)  - [How we used Online Experiments internally to evaluate a model change](#how-we-used-online-experiments-to-evaluate-a-model-change) ## Offline versus online experiments for AI agents Offline experiments are an integral part of agent development. They let teams run versions of their agents against predefined datasets, compare outputs, and catch known regressions before deployment, without waiting for them to appear in live traffic. In Datadog Agent Observability, [offline experiments](https://docs.datadoghq.com/llm_observability/improve/experiments/setup.md) support testing for both individual components and complete agent workflows. But [building comprehensive offline evaluation datasets](https://www.datadoghq.com/blog/offline-llm-evaluations.md) takes time and product knowledge. Teams need to choose representative requests, agree on what good behavior looks like, and decide which checks would reveal a regression. A model-based judge needs a rubric your team understands and a human to check its assessments. The work continues as your product changes: New workflows expose gaps in the dataset, tools return different information, and a judge that captures one failure mode may miss another. Early in development, you may still be discovering the requirements that would let you choose the right cases and criteria. Online experiments test performance and behavior in agent variants based on real user interactions. They rely on online evaluations, which score agent behavior in production, rather than predefined test cases. This allows you to compare how different versions of your agent behave under production workloads, so you can catch patterns that you may not otherwise have included in your offline evaluation datasets.  For example, consider a support agent that retrieves documents to answer customer questions. Your team might discover that its search tool frequently returns long passages, causing the agent to perform repeated searches before generating an answer. With offline evaluations, you could modify the tool to return shorter excerpts and check whether the agent still cites relevant evidence. But these tests might not capture the full range of requests users make or reveal whether the change helps users resolve their issues.  Online evaluations help you see, for example, whether users bring requests your team did not anticipate or ask follow-up questions, so you can better determine if the change you’ve made helps users resolve their issues. You can then run online experiments that use those scores to compare different versions of your agent. You keep the existing version as a control and test a changed version over the same period, using random assignment to separate the effect of your change from existing differences in usage (e.g., a revised search tool looks better because it receives easier questions). This helps you establish which version of your agent performs best in real production situations and, in turn, enables you to build better offline evaluation datasets as well. ## Compare agent variants using real production traffic with Datadog Online Experiments Running online experiments requires teams to connect each agent variant they are testing to the appropriate measurements and analyze the differences in performance. [Datadog Online Experiments](https://docs.datadoghq.com/llm_observability/improve/online_experiments.md) brings Agent Observability data together with Datadog Experiments to help you manage this process. You can use [evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md) you’ve created in Agent Observability and connect their scores to experiment metrics, alongside the operational measures you choose to collect. With assignment and measurement configured, Experiments calculates the relative changes and confidence intervals. ![Setup page in Online Experiments, showing decision metrics and other prefilled options to choose from.](https://web-assets.dd-static.net/42588/1793809302-online-experiments-setup-v2.png) You also choose how much exposure a change should receive. Once the variants are wired into your application, [Datadog Feature Flags](https://docs.datadoghq.com/feature_flags.md) can manage the traffic split and let you adjust exposure without deploying the application again. You can begin with a small share of users while retaining a control group for the comparison. The measures should reflect the decision you're making. A revised tool might improve evidence quality while adding extra searches. A cheaper model might require more attempts to complete a task. Comparing evaluation scores with duration, cost, and errors helps you see those trade-offs.  From your experiments, you can also view traces in Agent Observability to inspect the individual executions: which tool the agent called, what arguments it supplied, and how it responded to the result. This provides context that helps you dig deeper when an experiment leaves you with questions. For example, a promising cost result may justify pursuing a change, while uncertainty about its other effects warrants a closer look. Next, we’ll show you how we used Online Experiments internally at Datadog to assess this exact type of change in one of our own agents. ## How we used Online Experiments to evaluate a model change We used Online Experiments to compare OpenAI's GPT-5.6 Luna and GPT-6 Luna in our [Insights agent](https://www.datadoghq.com/blog/improving-insights-agent-observability.md), which investigates agent behavior by using trace evidence. We wanted to test whether [Luna 6's lower model prices](https://openai.com/index/introducing-gpt-6-sol-and-luna/) translated into cheaper investigations and understand what the change meant for investigation quality and tool behavior. We split traffic evenly between the models, keeping Luna 5.6 as the reference while evaluating Luna 6 under real usage. Our primary measure was investigation quality. An LLM judge scored investigations from 1 to 5 using the request, tool inputs and outputs, and the agent's conclusions. The rubric covered accuracy, completeness, and how well conclusions followed from the available evidence, including whether the agent acknowledged uncertainty or missing data. We collected duration, estimated cost, estimated token usage, and workflow errors from the agent's traces as separate measures. Online Experiments brought those scores and operational measures into the same comparison in Agent Observability. We could assess cost alongside the work the agent performed to complete an investigation. ![An experiment result breakdown of a comparison between Luna 5.6 and Luna 6 in Datadog Product Analytics.](https://web-assets.dd-static.net/42588/1793809392-online-experiments-results.png) *A is Luna 5.6 and B is Luna 6. Only the estimated-cost interval lies entirely below zero.* Luna 6's estimated cost was 44.3% lower. The confidence interval—a range of differences consistent with the data—ran from 37.5% to 51.2% lower. This result supported our hypothesis that the model change could reduce investigation cost. Quality scores were close at 3.84 for Luna 5.6 and 3.86 for Luna 6, although close averages alone do not establish equivalent quality. Duration and workflow errors left more uncertainty. The best estimates were 12.8% higher duration and 13.6% more workflow errors. But the duration interval ranged from 4% lower to 30% higher, and the error interval from 16% lower to 43% higher—effectively, a range between little-to-no difference and a meaningful increase. We inspected the traces to understand the risk before deciding whether to proceed. In the traces we reviewed, Luna 6 supplied invalid tool parameters more often than Luna 5.6. It usually corrected those errors after retries. Our quality judge considered tool use, but its overall score did not tell us how often calls failed or how much work recovery required. We were thus able to use traces to expose a specific behavior in the agent's handling of tool calls that could be improved in the Luna 6 version of the agent. We addressed the tool parameter issue that our investigation uncovered before proceeding with the Luna 6 version that helped us lower costs. The experiment also revealed next steps that could help us improve our agent in the future. The failed calls we found are useful candidates for [offline regression tests](https://www.datadoghq.com/blog/from-traces-to-experiments-a-loop-for-improving-ai-agents.md). A follow-up online experiment could test a change to tool definitions or error feedback while continuing to measure quality, duration, and cost. ## Improve AI agents faster with Online Experiments in Agent Observability Datadog Online Experiments helps teams evaluate AI agent changes under real production conditions, compare quality and operational performance, and investigate unexpected behavior by using traces. Whether you're testing a different model, refining retrieval, or modifying a tool, controlled experiments provide evidence to help you determine which changes improve your agent and which require further work. The findings from each experiment can also help improve offline evaluation coverage. As your team discovers new failure modes and user workflows, you can turn those observations into regression tests and use subsequent experiments to measure whether your fixes work under real usage. Online Experiments is free for Agent Observability customers submitting up to 40,000 LLM spans per month. To learn more about getting started with Agent Observability, see the [product page](https://www.datadoghq.com/products/ai/agent-observability/) and [documentation](https://docs.datadoghq.com/llm_observability.md). New to Datadog? . Startup? Eligible companies get a year of free Datadog credits. [Apply here](https://www.datadoghq.com/partner/datadog-for-startups/).