Teaching a 9B model to investigate production alerts

A deployment, feature flag, or configuration change may trigger an alert, but identifying which recent change most likely contributed to the alert can require a time-consuming investigation across multiple services and systems. Rules-based approaches can identify potentially relevant changes quickly and inexpensively, but their accuracy is limited on complex incidents. Agentic investigations can reason more deeply about the available evidence, but using frontier models for every alert is too expensive at scale.  

To see whether a smaller, specialized model could close that gap, we fine-tuned Qwen3.5-9B on traces from investigations generated by GLM-5.3. The resulting model achieved 87% of GLM-5.3’s recall. Its self-hosted LLM serving cost was $0.003 per investigation, compared with $0.06 in API charges for GLM-5.3, a roughly 20× decrease under our evaluated deployment conditions. At $0.003 per investigation, a single 40 GB A100 GPU can support approximately 100,000 investigations per week. This is enough to investigate a subset of the millions of unique monitors customers interact with each week. We are pursuing further optimizations to make this approach practical for a much larger share of those monitors.

In this post, we explain how we built the training-data flywheel, what it changed about the model’s investigative behavior, and how fine-tuning smaller models could make agentic investigations practical across a much larger volume of production alerts.

Two approaches to change attribution

Change Tracking currently supports incident investigation in two complementary ways. The first is the Relevant Changes tab, which overlays recent changes on a monitor’s alert timeline and highlights those most likely to have contributed to the alert. This gives engineers a fast way to identify potentially relevant deployments, feature flags, and configuration changes. 

Change Tracking also exposes its data through a Model Context Protocol (MCP) tool that agents, including Datadog’s Bits Investigation, can use during incident investigations. According to internal Datadog telemetry from July 2026, the Change Tracking tool contributes to thousands of Bits investigations each week, and 20% of Bits Investigation conclusions reference a change captured by Change Tracking.

The two approaches offer different trade-offs. Relevant Changes returns results within seconds and is inexpensive enough to make available for free to Application Performance Monitoring (APM) customers, but its rules-based retrieval limits its accuracy on complex incidents. Agentic investigations using the MCP tool can reason more deeply about the available evidence, but multi-turn investigations with frontier models are too expensive to run for every alert at scale.

A smaller model specialized for change attribution offered a potential way to combine these strengths: deeper agentic investigation at a cost that could support a much larger volume of alerts.

The approach: The change attribution flywheel

Prompt engineering alone wasn’t enough to make the smaller model reliable. Even after repeated iterations on the system prompt and tool descriptions, Qwen3.5-9B tended to search too broadly instead of narrowing its investigation around the most promising evidence. Rather than continue adding rules to compensate for that behavior, we explored whether we could teach the smaller model the investigative behavior of a more capable model.

We adapted NVIDIA’s data flywheel blueprint for change attribution: Generate investigation traces with a larger teacher model, use successful traces to fine-tune a smaller student model, and repeat the process as new production investigations become available. NVIDIA demonstrated this approach by fine-tuning a Llama 3.2 1B model on tool-calling traces from a 70B teacher, reaching 98% of the teacher’s accuracy with a roughly 70× reduction in parameter count. We adapted that approach to test whether the same idea could make agentic change attribution inexpensive enough to run across a large volume of alerts.

Change attribution is well suited to this approach because its investigations have a consistent structure: Each investigation uses the same set of tools and works toward the same objective of identifying the change most likely responsible for an incident. Bits Investigation conclusions also let us derive labeled examples from production investigations without manual annotation. Together, the repeatable workflow and a continuously growing set of labeled examples make change attribution a strong candidate for a specialized model:

1. Identify the change: Bits Investigation writes a free-form conclusion for every incident it investigates. We use GLM-5.3 to parse the conclusion, identify any changes it references, and map them to the change IDs produced by our system. We treat each referenced change as a proxy label: the change Bits associated with the incident, rather than independently verified causality. This gives us labeled examples without requiring manual annotation.

2. Run the teacher agent: GLM-5.3 investigates the same alert using eight turns, five tools, and a 65,536-token context window. It returns a ranked list of possible changes along with confidence scores. We intentionally constrain the investigation process so that a much smaller model can learn to reproduce it.

3. Generate the dataset: We run the teacher model three times on each alert. We keep only the examples that return changes that match the proxy labels extracted from the Bits Investigation conclusion and discard the rest. This technique is known as rejection sampling fine-tuning.

We applied this process to 348 internal incidents that occurred between May 26 and June 24, 2026. From these incidents, we created an initial dataset of 100 teacher traces, each from a different investigation and selected based on which traces scored their proxy label the highest. No customer data was used to generate this dataset.

4. Fine-tune the student: We fine-tuned Qwen3.5-9B using 16-bit low-rank adaptation (LoRA), which updates a small set of adapter parameters rather than all of the model’s weights. We calculated training loss only on the assistant responses in each selected trace.

5. Repeat the cycle: Both the teacher and student models run in production. When the teacher’s prediction matches the proxy label and the student’s does not, we add the teacher’s investigation trace to the training dataset and retrain the student on the expanded dataset, allowing it to learn from new production investigations over time. 

We have completed two rounds of this process using additional internal incidents, adding 86 training examples and increasing the dataset from 100 to 186 examples.

Results: 87% of the teacher’s Recall@5 at roughly 5% of the investigation cost

The following results are based on a sample of 326 production incidents, comprising 187 internal incidents and 139 customer incidents collected between August 11 and August 25, 2026. A daily cron job replays the previous day’s production incidents and runs an investigation with each model. For this evaluation, Recall@5 measures whether each model’s top five ranked changes include the proxy label extracted from the corresponding Bits Investigation conclusion. Recall@5 measures agreement with the change identified in the Bits Investigation conclusion, but it does not independently verify that the change caused the incident.

Every evaluated incident occurred after the training data was generated, so the evaluation set was fully held out from the training set. The fine-tuned model was trained only on internal incidents, making the 139 customer incidents a useful test of whether the learned behavior transfers beyond the population used for training. Because GLM-5.3 is currently enabled at Datadog only for internal use, however, our direct student-teacher comparison is limited to the internal incidents.

On customer incidents, the fine-tuned model reached 0.62 Recall@5, compared with 0.52 for the base model and 0.51 for the heuristics-based ranker.

Four results jump out.

The fine-tuned model retained much of the teacher’s Recall@5 at 5% of the cost: The fine-tuned student achieved 0.55 Recall@5, compared with 0.63 for the teacher. Its self-hosted serving cost was $0.003 per investigation, compared with $0.06 in API charges for GLM-5.3. In other words, the student achieved 87% of the teacher’s Recall@5 at 5% of the inference cost.

Fine-tuning changed how the model investigated alerts: Prompting alone did not correct the base model’s tendency to search too broadly. Across the same evaluation set, the base Qwen3.5-9B model exhausted its turn limit or context window in 25% of investigations, and trace analysis helped explain why: It searched broadly for services and changes without narrowing its investigation quickly enough. After fine-tuning, that behavior changed. The fine-tuned model averaged 5.8 search calls per trace, compared with 6.8 for the base model. Instead, it gathered more evidence from logs, spans, and metrics, averaging 8.3 calls compared with 4.7. This shift from broad change discovery toward focused evidence collection helped the fine-tuned model improve Recall@5 while using fewer tokens.

Fine-tuning outperformed our handwritten rules: The fine-tuned student improved Recall@5 from 0.46 to 0.55 compared with our production rules-based system while costing approximately $0.003 per investigation. Rather than continuing to expand a growing collection of specialized rules, fine-tuning let us learn investigative behavior from production-derived examples.

The main trade-off is the number of tokens processed per investigation. The fine-tuned model uses substantially more tokens than the heuristics-based ranker because it performs a multi-turn agentic investigation instead of applying a fixed set of rules.

Extra intelligence has a price: While more capable models such as Opus 5.0 continue to improve Recall@5, their costs quickly become prohibitive, with an average cost per investigation of $0.32. For our use case, where we need to run investigations across a large volume of alerts, that cost compounds quickly.

The cost estimates are calculated over the same 187 internal incidents used for the direct model comparison. For the self-hosted Qwen models, we estimate serving costs based on measured investigation throughput and allocated GPU instance costs. The fine-tuned model generates an average of 1,700 output tokens per investigation and achieves an aggregate throughput of 280 output tokens per second on a single NVIDIA A100 40 GB GPU. At an effective cost of $1.74 per GPU-hour for an AWS p4d.24xlarge instance, this corresponds to an estimated serving cost of approximately $0.003 per investigation.

Beyond supervised fine-tuning

Supervised fine-tuning gets us a strong student, but it has a ceiling: The student is limited by the behaviors represented in the teacher’s traces. To continue improving, we plan to explore reinforcement learning (RL).

Our problem has a useful property for reinforcement learning: Bits Investigation results provide a signal that we can evaluate automatically. Whenever Bits Investigation identifies a change associated with an incident, we can compare the model’s predictions against that result and potentially use the match as a reward signal for reinforcement learning. This could let the model learn from production investigations without depending on explicit human feedback such as thumbs-up and thumbs-down ratings.

This is similar to how Cursor continuously improves its Tab model using feedback from accepted and rejected code completions. In our case, the feedback signal would come from the changes identified in Bits Investigation results rather than explicit user interactions. The challenge is that this feedback signal is imperfect. Bits Investigation can sometimes identify the wrong change. As Bits Investigation improves, we expect the quality of the labels it provides to improve as well. The advantage is that this approach would not require a separate human grader or learned reward model. The same production investigations that power the data flywheel could also provide the feedback needed for future reinforcement learning.

Making agentic investigation practical at scale

This process points to a repeatable approach for tasks with the right ingredients: a consistent agentic workflow, a growing source of useful labels, and an evaluation signal that can identify successful investigations. For change attribution, those ingredients let us generate successful investigation traces with a capable teacher model, use them to specialize a smaller model, and continue expanding the training set as new production investigations become available.

We believe this will become an increasingly common way to build AI systems. Frontier models remain essential for solving the hardest problems, but they can be too expensive to run for every request at production scale. Lower inference costs can reduce spending and make new product experiences possible. As inference costs continue to fall, we expect AI to enable new observability workflows that would be impractical at higher costs.

Change attribution is one example of that shift. To investigate which changes may have contributed to an incident in your own environment, run Bits Investigation or ask Bits Chat a change-related question.

If you’re interested in building and fine-tuning AI systems for production-scale engineering challenges, check out our open roles at Datadog.