--- title: "How we investigated prompt cache misses with Datadog" description: "An investigation of prompt cache misses in internal AI applications by using Datadog Cloud Cost Management and Agent Observability." author: "Bruno Volpato, Will Potts, JC Mackin" date: 2026-11-06 tags: ["agent observability", "cloud cost management", "ai", "debugging", "cache"] blog_type_id: the-monitor locale: en --- Our internal AI applications repeatedly send the same system instructions, tool definitions, and conversation history to large language models (LLMs). Prompt caching lets models reuse this data from previously processed input, reducing the cost of handling repeated content. But while investigating our LLM costs, we found applications that weren’t benefiting from caching, even when it appeared to be enabled. We used Datadog [Cloud Cost Management](https://www.datadoghq.com/product/cloud-cost-management/) to examine provider spending, then investigated individual model requests with [Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/). We found that small changes between requests could prevent the provider from reusing work it had already done. Some applications also enabled caching without sending the information the provider needed to use it. In this post, we'll explain: - [How we traced LLM costs to individual requests](#how-we-traced-llm-costs-to-individual-requests) - [Why our applications weren't reusing cached prompts](#why-our-applications-werent-reusing-cached-prompts) - [How some caching settings failed to reach the provider](#how-some-caching-settings-failed-to-reach-the-provider) - [How we evaluated opportunities to reduce costs](#how-we-evaluated-opportunities-to-reduce-costs) ## **How we traced LLM costs to individual requests** We started by examining provider spending in Cloud Cost Management. We then used Agent Observability to investigate particular applications and the API routes they used to reach models. An agent can call a model several times to complete one task, so we compared the first call with later calls from that task. Each request can include instructions for the model, descriptions of tools it can call, and earlier messages. The application often sends that material again when it adds a new message or a tool result. We wanted to determine whether the provider reused work from processing the repeated material. We examined how many tokens each request used, its estimated cost, and how long the model took to begin responding. Cache writes showed input used to populate the cache; cache reads showed input served from it. Comparing those measurements across calls helped us assess whether later requests benefited from earlier ones. The example below shows three requests with no recorded cache hits. Each included more than 180,000 input tokens, with estimated input costs ranging from $0.74 to $0.82 per request. ![Three LLM requests with zero prompt cache hits, with each request using over 180,000 input tokens and costing $0.74–$0.86 in total.](https://web-assets.dd-static.net/42588/1793978610-1-agent-observability-llm-requests-without-prompt-cache-hits.png) In the caching example below, successive requests read more input from the cache and wrote less new content to it. Estimated total costs ranged from $0.02 to $0.14 per request, compared with a total of $0.74 to $0.86 per request in the example without cache hits. ![Three successive LLM requests show increasing cache reads, decreasing cache writes, and estimated costs falling from $0.14 to $0.02.](https://web-assets.dd-static.net/42588/1793978687-2-agent-observability-prompt-cache-reads-writes-and-request-costs.png) Before drawing conclusions, we checked what our traces recorded. The application and its gateway could both record the same model call, which would cause us to count its cost twice. Conversely, a record missing cache fields could make caching appear unused. We checked the recorded usage against provider responses and looked for complete request content. Some records omitted system instructions or tools, leaving us unable to explain why the provider hadn’t reused earlier input. We then inspected the code that built the requests and sent them. ## **Why our applications weren’t reusing cached prompts** Prompt caching depends on matching the beginning of a request, called its prefix, with input processed earlier. A difference near the beginning can prevent reuse of otherwise identical content that follows. Our investigation showed how applications could introduce those differences even when they sent the same underlying instructions or tools. ### **Changing details appeared within reusable instructions** One chat service combined its system instructions with the current time, organization details, feature flags, and incident context. Some of those values could change between requests. If a changing value appeared early, the provider might be unable to reuse work for the instructions that followed it. For example, imagine requests beginning with these two versions of the same instruction: - First request: “The time is 10:00. Help the engineer investigate this incident.” - Later request: “The time is 10:05. Help the engineer investigate this incident.” The instruction stays the same, but the timestamp changes before the provider reaches it.  Our proposed change put stable instructions first and request-specific details afterward. Where supported, a cache boundary would mark the end of the reusable portion. The model would still receive the context it needed, with the same restrictions on which information it could access. We also checked how often values changed before attributing misses to them. One application’s system prompt included the UTC date. That could explain a miss across a date change, but not repeated misses throughout the day. Another apparent leading timestamp came from recorded user content, so editing the system instructions wouldn’t have addressed it. ### **The same tools arrived in different orders** The chat service built its tool list by iterating over a Go map, which doesn’t guarantee the order in which entries appear. It could send tools A, B, and C in one request, then B, A, and C in another. Because tool descriptions can be part of the cached prefix, that changed sequence could prevent reuse. The associated patch sorted tool names before building the list. A test added the same tools to maps in different orders, then checked that the resulting lists matched. The test checked the code’s behavior; subsequent model requests would still be needed to confirm whether the provider reused more cached input. ![Go map iteration produces different tool orders between requests, while sorting by tool name produces a consistent order.](https://web-assets.dd-static.net/42588/1793978749-3-prompt-caching-go-map-tool-definition-order-comparison.jpeg) A service that generated prompts from source code had a similar problem when it listed files from a map. Different file orders changed both the prompt and which content remained after the prompt was shortened. Sorting file paths addressed that inconsistency, although any caching benefit still depended on the backend supporting reuse. ## **How some caching settings failed to reach the provider** We also followed requests through the software that prepared them for the provider. An application could enable a caching option while sending a request that lacked the required controls or that used an unsupported API. ### **An enabled option left out required controls** One chat service had an option called `WithPromptCaching`. It was enabled, but the Anthropic requests we inspected showed no cache activity. The option added a beta header but omitted controls in the request body that told the provider what to cache. Another application didn’t pass its SDK’s caching option to the component that prepared the provider request. These findings gave us specific code paths to examine and change. To verify a fix, we would need to inspect the outgoing controls and confirm that a later request read previously cached input. ### **One API route didn’t support caching** Our gateway could send requests through Anthropic’s native Messages API or its compatibility endpoint, which accepted requests formatted for OpenAI’s API. We observed cache reads on native Messages traffic and no cache activity on the compatibility route. During the investigation, we confirmed in Anthropic’s documentation that the compatibility route didn’t support prompt caching. Moving an application to the native API became a potential way to enable caching. Before recommending that change, we needed to check that the application would still: - Send messages with the correct roles and content - Call tools and receive their results correctly - Stream responses as expected We also needed to measure how much input each application could reuse. Some applications already used both routes, and their cache usage on the native route varied. The route comparison identified an opportunity to investigate, but it didn’t establish how much any application would save. ## **How we evaluated opportunities to reduce costs** Our findings identified changes to evaluate, including sorting tools, separating instructions from changing context, and passing caching controls through to the provider. Each needed checks beyond whether the request succeeded. The application could continue answering correctly while repeatedly paying to process input that might otherwise be reused. We also needed to evaluate whole tasks. Some providers charge extra to write input to the cache, so repeatedly writing content that is rarely reused can increase costs. Keeping longer conversation histories can increase the amount of input sent, even if more of it is cached. To evaluate a change, we needed to compare the same application route and task type before and after, asking: - Did later requests read input cached by earlier requests, including after the agent called a tool? - Did each completed task cost less, including output tokens, retries, and calls to fallback models? - How quickly did the model start responding, and how long did the task take? - Did the application still answer accurately and use tools successfully? Providers report cached input differently, so we checked what each token count included. Our normalized Agent Observability input total already includes cache reads and writes. Adding those categories again would count the same input twice and distort the estimated cost. [DDSQL](https://docs.datadoghq.com/ddsql_editor.md#use-sql-syntax-ddsql) can group model calls by application, provider, model, and time window to help repeat comparisons after a change. Those queries still require us to identify which records represent the provider calls we want to measure. Estimated token costs can help explain request behavior, while provider bills are needed to confirm financial savings. ## **What we learned about investigating LLM costs** Inspecting requests helped us understand why enabled caching options didn’t always work as expected. Some prompts mixed changing details into reusable instructions, and some requests listed tools in different orders. Other requests omitted caching controls or used routes that didn’t support them. Fixing these issues helped us reuse cached input and reduce LLM costs. Cloud Cost Management helped us examine provider spending, while Agent Observability helped us investigate how applications called models and used tokens. Other teams can follow the same approach to check whether their applications benefit from prompt caching.  To begin, see the[ Agent Observability documentation](https://docs.datadoghq.com/llm_observability.md) and[ Cloud Cost Management documentation](https://docs.datadoghq.com/cloud_cost_management.md). [Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/) offers a free tier with up to 40,000 LLM spans per month. Ready to instrument your apps and agents?[ Read the quickstart guide](https://docs.datadoghq.com/llm_observability/quickstart.md). Are you a startup? Eligible companies get a year of free Datadog credits. [Apply to the Datadog for Startups program](https://www.datadoghq.com/partner/datadog-for-startups/).  If you’re not yet a Datadog customer, .