Jacob Simpher
Software Engineer
Will Potts
Senior Product Marketing Manager
Prompts are an important part of LLM application behavior, but teams often manage them alongside application code or across separate notebooks and configuration files. This can make prompt changes dependent on application releases and make it difficult to understand which version is running in each environment. When a change affects application quality, latency, or cost, teams also need a way to connect that behavior back to the prompt version that produced it.
Prompt Management in Datadog Agent Observability provides a centralized registry where you can create, version, deploy, and retrieve prompts independently of your application release cycle. Prompt Management works with Prompt Tracking to connect managed prompt versions with the LLM calls that use them, helping you understand how prompt changes relate to application behavior.
In this post, we’ll explain how you can:
Manage prompt versions in one place
As teams iterate on LLM applications, prompt development can become difficult to coordinate. Different versions may live in application code, configuration files, or development environments, making it harder to identify the prompt that is running in production or compare it with an earlier version.
Prompt Management gives teams a centralized registry for prompts used by their LLM applications. Each managed prompt has a stable Prompt ID and a version history. You can update the prompt while applications continue to reference the same identifier, and you can inspect earlier versions to understand how the prompt has changed.
The Prompts page brings managed prompts and prompts captured through Prompt Tracking into the same catalog. If Agent Observability already tracks a prompt in your application, you can register it for management instead of recreating it separately. You can also create prompts directly in Datadog and define variables for content that your application supplies at runtime.
As you edit a managed prompt, Datadog preserves previous versions instead of overwriting them. Version history and comparisons help teams review prompt changes and return to earlier versions when necessary.
Deploy prompt changes independently of application releases
Prompt Management separates prompt deployment from application deployment. Instead of embedding each prompt revision in application code, your application retrieves a managed prompt at runtime by using its stable Prompt ID. Datadog resolves that request to the appropriate deployed version.
This separation gives teams more flexibility to iterate on prompt behavior, as they can publish a new prompt version and deploy it to an environment without shipping a new application build. If a prompt change produces unwanted results, you can deploy an earlier version without rolling back unrelated application changes.
Applications can also retain a local prompt as a fallback for cases when managed prompt retrieval fails. This lets teams introduce managed prompts while maintaining application-defined fallback behavior.
Prompt Management supports runtime retrieval in Python. For prerequisites and implementation details, including supported ddtrace versions, authentication, and fallback behavior, see the Prompt Management documentation.
Control how prompt versions reach your applications
Different stages of prompt development often need different versions of the same prompt. A team might want one version in development, another in staging, and a stable version in production without maintaining separate prompt identifiers for each environment.
Prompt Management lets you deploy versions of the same Prompt ID to different environments. Applications can retrieve the version associated with their environment, while developers can also request an exact version when they need to reproduce or validate specific behavior.
For more granular control, Prompt Management builds on Datadog Feature Flags to support targeting rules. Within an environment, you can keep one prompt version as the default while serving another version to requests that match attributes you define. This gives teams a way to expose prompt changes selectively without creating separate prompts or changing application deployments.
These controls provide a consistent path for moving prompt changes through development and production. Teams can keep a stable Prompt ID while deciding which version different environments or targeted requests receive.
Connect prompt changes to application performance
Managing prompt versions addresses one part of the prompt development life cycle. Teams also need to understand what happens after a version reaches their application.
When Agent Observability is enabled, Prompt Management works with Prompt Tracking to associate managed prompt versions with LLM spans. Datadog can then connect an individual prompt with information about its usage and performance.
From the Prompts page, you can compare versions by call volume, latency, token usage, and cost. You can also inspect version history, review differences between prompts, and jump to traces that used a particular version. This context helps teams investigate whether changes in application behavior coincide with a prompt update.
For example, if latency or token usage changes after a prompt update, you can identify the version associated with affected LLM calls and inspect the corresponding traces. Prompt Tracking also lets you filter Trace Explorer by prompt name, ID, or version, providing another way to investigate requests associated with a change.
Prompt retrieval and Prompt Tracking remain separate capabilities. Applications can retrieve managed prompts without enabling Agent Observability, while Agent Observability provides the LLM spans and prompt metadata needed to analyze prompt usage and performance. See the Prompt Tracking documentation for instrumentation and correlation details.
Manage prompt changes with Datadog Agent Observability
Prompt Management gives LLM application teams a centralized, versioned registry for changing prompts independently of application releases. Teams can deploy different versions across environments, control how versions reach application traffic, and return to earlier versions without changing the Prompt ID used by the application.
Combined with Prompt Tracking, Prompt Management also connects those versions to LLM calls and performance data, helping teams investigate how prompt changes relate to application behavior.
Datadog Agent Observability is free for Datadog customers submitting up to 40,000 LLM spans per month. To learn more about getting started with Agent Observability, see the product page and documentation.
If you don’t have a Datadog account, sign up for a 14-day free trial to manage and monitor prompts for your LLM applications.
