Get Started with Datadog

The Monitor

Golden Paths for AI agents: What changes when platform users aren’t human?

Published

Read time

14m

Golden Paths for AI agents: What changes when platform users aren’t human?
Candace Shamieh

Candace Shamieh

Technical Content Writer

Shlomo Benyaminov

Shlomo Benyaminov

Senior Software Engineer

James Eastham

James Eastham

Senior Technical Advocate

Agents are already calling your self-service APIs, querying your catalog, and independently provisioning resources around the clock. According to Gartner’s 2026 Hype Cycle for Agentic AI report, agents have had the most aggressive adoption curve of any emerging technology. Gartner even considers agents to be a formal user persona, referring to the agent experience (AX) throughout the report. The question that platform teams are asking has shifted from “Should agents use the platform?” to “How should the platform evolve now that they do?”

For teams using Golden Paths to encode supported development workflows, this shift introduces a new challenge. Paths originally designed to guide human developers must adapt to guide autonomous agents.

In this post, we’ll discuss how to adapt Golden Paths so that agents can use your platform effectively and within constraints. Specifically, we’ll cover the following:

How should you design Golden Paths for AI agents?

An agent-facing Golden Path should be designed around an execution pattern and define which steps must remain under deterministic control.

Match AI agent execution patterns to workload requirements

The term execution pattern describes how and where an agent’s process runs. Platform teams should select an execution pattern based on the workload’s latency, durability, isolation, and API requirements. A Golden Path designed for one execution pattern may suit some workloads but not others.

We recommend that you start by choosing an execution pattern that addresses the workload’s primary constraint. Use a synchronous, low-latency pattern for real-time interactions and a durable pattern for asynchronous or resumable tasks. You can supplement the execution pattern with supporting capabilities as needed, such as an isolated sandbox for development tasks. 

You may recognize the underlying principle driving our recommendation. For more than two decades, methods like Attribute-Driven Design, developed by the Carnegie Mellon Software Engineering Institute, have taught engineers to select an execution pattern based on workload requirements. The unique challenge when working with agents is the workload profile, which includes more variable execution time, dynamically selected actions, and potentially greater requirements for durability or isolation.

Real-time conversational AI agents need a low-latency execution pattern

A real-time conversational agent operates within a tight latency budget that typically ranges from seconds to a few minutes. These agents need Golden Paths that use a synchronous, low-latency execution pattern because the agent receives a request, invokes models and tools, and returns or streams a response within a single interaction.

The Golden Path must account for time-to-first-output and total completion time. Streaming reduces the wait for initial output but doesn’t guarantee that the agent will complete its work within the overall latency budget. Low latency and a responsive user experience have always been priorities for synchronous services.

To keep this work within the latency budget, the Golden Path should set an end-to-end deadline, apply timeouts to individual calls, and define what happens if the work cannot finish before the deadline. 

Long-running, proactive AI agents need a durable execution pattern

Long-running agents operate asynchronously on tasks that continue for hours or days. Because their work can outlive a single request or application process, the Golden Path should use a durable, resumable execution pattern.

At Datadog, we learned this firsthand during a routine rolling deployment of AI Gateway, our internal-facing service that acts as a proxy for LLM providers. Its graceful shutdown period was too short for in-flight agent requests to complete, which increased the number of request timeouts until we extended it appropriately.

With durable execution, workflow state must persist independently of model context and in-memory agent state to keep the record of the workflow’s progress intact. The record should include completed steps, intermediate outputs, and pending approvals so that work can resume after an application restart or infrastructure interruption. 

Add isolated execution when AI agents need to run code

Agents that operate in a development environment inherit that environment’s permissions. Model-generated commands and repository code run with the same access to files, credentials, and network endpoints as the agent process. The Golden Path should isolate these operations from the primary agent service.

When possible, keep the agent harness in a standard, observable service and expose an isolated sandbox as a remote tool. Restrict the sandbox’s filesystem and network access, resource usage, and execution time to fit what the operation requires. The Golden Path should also specify the sandbox’s lifetime and cleanup behavior to prevent state from persisting between runs.

Separate probabilistic agent work from deterministic control

Golden Paths can include probabilistic and deterministic steps. Probabilistic steps allow the agent to perform work that requires interpretation or judgment. Deterministic steps can act as controls by using CI workflows, policy checks, authorization, approvals, and deployment rules to determine whether work can proceed.

When designing an agent-facing Golden Path, identify where agent judgment produces an output that could lead to a consequential action. To reduce the risk of unverified results being executed, place deterministic controls between a probabilistic output and the resulting action.

Weave Intelligence, a platform engineering analytics organization, has predicted that the most innovative organizations will have hybrid Golden Paths in production by mid-2027. Hybrid Golden Paths allow probabilistic operations in a workflow where progression remains under deterministic control. For example, in a bug-fixing path, the agent can diagnose the problem and generate a code change. Tests, security scans, approvals, and merge permissions are the deterministic gates that decide whether the output progresses. The agent can revise an output that fails these gates, but it can’t override them or decide that its own work satisfies the release conditions. 

How can I help AI agents use my platform capabilities?

Golden Paths will tell the agent what to call and when, but they won’t make your platform’s capabilities usable by agents. Each capability’s interface must be legible so that your agent can reason about it independently, and appropriate security measures need to be enforceable at the platform level.

Platform capabilities exposed to agents should have reusable, machine-consumable contracts that define how agents can use them safely. For each workflow, the Golden Path can impose stricter permissions, limits, or approval requirements as necessary. A Golden Path should use authoritative service metadata to determine which resources the agent can act on and which requirements apply. Since guardrails at the orchestration level do not transfer to the capability itself, the platform—not the agent—must enforce both the capability-level contracts and the Golden Path-specific constraints.

Expose capabilities through structured interfaces 

Every action exposed to an agent should have a machine-consumable contract that contains the following: 

  • A clear description

  • Typed inputs and outputs

  • Stable and versioned schemas

  • Machine-readable limits and error categories

  • Declared side effects, preconditions, and approval requirements

Datadog Action Catalog output schema for a chat completion action, defining typed fields for container data and message content returned to an AI agent.
Datadog Action Catalog output schema for a chat completion action, defining typed fields for container data and message content returned to an AI agent.

Rate limits, concurrency limits, execution timeouts, spending limits, and compute or memory boundaries should be enforced at the interface, gateway, or workflow orchestrator. Errors should be machine-readable and differentiate between invalid input, authorization failure, rate limiting, temporary unavailability, and non-retryable policy rejections. For retryable errors, specify any delay or backoff requirements. For non-retryable errors, clearly identify whether the agent must change its input, request authorization, obtain approval, or stop the workflow.

Because agents might repeat a request even after the first attempt succeeds, state-changing operations should be idempotent where possible. When inherent idempotency is not an option, the platform can accept an idempotency key or unique operation identifier so that repeated requests return the original result instead of performing the action again.

At Datadog, we apply these principles at both the runtime-enforcement and tool-design layers. Datadog AI Guard evaluates prompts, model responses, and tool calls against configured security policies. When blocking is enabled, AI Guard can stop a tool call before it executes, keeping enforcement outside the agent’s own judgment.

Datadog AI Guard blocking a tool call flagged for data exfiltration and jailbreak attack categories before the action executed.
Datadog AI Guard blocking a tool call flagged for data exfiltration and jailbreak attack categories before the action executed.

For data-intensive capabilities, machine-consumable contracts should also define how agents can request only the data necessary for the task. Query-oriented tools in the Datadog MCP Server supports context-efficient retrieval by allowing agents to select fields, limit records, and request aggregated results instead of raw data. In some evaluation scenarios, these tools reduced run costs by approximately 40% because agents used fewer tokens to reach an answer.

Keep the service catalog authoritative for AI agent decisions

Agents can consult wikis and runbooks, but they cannot reliably reconcile information that is incomplete, outdated, or contradictory. An agent-facing Golden Path should be able to direct agents to an authoritative catalog for ownership, dependencies, service criticality, environments, and supported actions. The catalog can combine telemetry data, service definitions, repositories, and other systems of record. As the canonical interface for the platform, the catalog should establish which source takes precedence when records conflict.

Catalog accuracy affects the decisions that an agent is permitted to make. An incorrect owner can route an approval to the wrong team, a missing dependency can cause the agent to underestimate the impact of a change, and an outdated environment or criticality classification can allow an inappropriate action. In order to identify gaps before autonomous workflows rely on inaccurate metadata, platform teams should use automated discovery, schema and freshness validation, scorecards, CI checks, or regular reviews. 

The Golden Path should verify that required catalog metadata is present and trustworthy before allowing a consequential action to proceed. If that metadata is missing, stale, or contradictory, the path should restrict the agent to read-only operations, require human review, or block the action according to its potential impact. As the canonical interface for the platform, the catalog should establish which source takes precedence when records conflict or refuse to execute without accurate metadata.

Using products like Datadog Software Catalog and Datadog Scorecards minimizes manual overhead for platform teams as they maintain an authoritative service catalog. Software Catalog automatically discovers entities and dependencies and populates each entry with declared metadata, such as ownership and documentation. Scorecards evaluate catalog entries for production readiness based on requirements for observability, performance, security, and documentation. 

How can I control autonomous AI agent work?

An agent-facing Golden Path should define the dispatch and workflow controls that govern the majority of autonomous work, including why a run may begin, what the agent may access, where it may execute, and how its results must be recorded.

Make dispatch an explicit step in agent-facing Golden Paths

Dispatch is the platform step that converts a signal into a bounded, authorized agent run. A signal can come from an alert, schedule, code change, human request, or other sources. The signal identifies potential work but doesn’t authorize the agent to perform that work. Before dispatch, the platform should verify that the signal comes from an approved source, the proposed task is within scope, and an equivalent task isn’t already underway.

Formalize this transition within the Golden Path as: signal → curated context → task identity → allowed capabilities → execution target → recorded output. At the curated-context step, the platform combines authoritative catalog metadata with evidence from the triggering signal, such as alert details or request parameters. It then assigns a unique task identity that connects the run to its source and enables attribution, deduplication, and audit. Based on the task, the platform grants access only to the required capabilities and selects the appropriate execution pattern and environment. The resulting changes, evidence, approvals, and final status remain associated with the task identity and are added to the relevant system of record.

Dispatch converts an incoming signal into a bounded agent run by checking curated content, task identity, allowed capabilities, and execution target. The bounded run then produces recorded output including changes, evidence, approvals, and final status.
Dispatch converts an incoming signal into a bounded agent run by checking curated content, task identity, allowed capabilities, and execution target. The bounded run then produces recorded output including changes, evidence, approvals, and final status.

Credentials that are issued at dispatch should be scoped to the components, environments, and operations required for that run, and expire when the task ends. This model enables the same agent software to support different use cases without carrying the same authority into each one. For example, one dispatch can permit a code-fixing agent to inspect logs and repository files, create a branch, and run tests for a specific service. Another can allow the same agent to investigate a different service without modifying its repository.

Proactive agents also need explicit boundaries for self-generated work. Define which signals they treat as potential tasks, which components and environments they can investigate, and what follow-up work they can initiate. Limit how frequently tasks can be created, how deeply one task can generate additional tasks, and how duplicate or recurring work is suppressed. When a finding falls outside of these boundaries, the agent should record or recommend the work for review instead of dispatching it independently.

Build checkpoints and retry budgets into AI agent workflows

An autonomous workflow needs explicit points where the platform evaluates progress and determines whether the agent can continue. Build approval states, dry-run or preview modes, and precondition checks into the Golden Path before actions that modify production systems or create other consequential effects. At each checkpoint, the workflow should preserve its state and record the evidence used to continue, pause, reject, or escalate the work. These decisions should remain under deterministic control.

Individual requests and autonomous correction loops require different retry controls. An interface can tell the agent whether a failed request is safe to repeat, but a retry budget limits how long the overall workflow can continue attempting to make progress. Bound each correction loop by its number of attempts, elapsed time, cost, or a combination of these factors. For example, a code-fixing agent might be permitted to revise its change and rerun tests three times or until it reaches a defined time or token budget. Without these limits, an agent can repeatedly consume model, tool, and infrastructure capacity without achieving a successful outcome.

The Golden Path should also define terminal states—such as succeeded, failed, cancelled, expired, approval denied, and escalated—to help manage workflow life cycle. When a workflow reaches its retry limit or another terminal condition, it should stop safely, preserve its intermediate outputs and supporting evidence, and notify a human operator or initiate a separate escalation workflow.

Equally important is defining how interrupted or unsuccessful operations are handled. When a workflow resumes from a checkpoint, retain its task identity and idempotency keys so that it doesn’t repeat state-changing operations that already succeeded. For reversible actions, specify how to roll back the change. When reversal isn’t possible, define a compensating or containment action that limits the impact and leaves the affected resource in a known state.

Preserve visibility and accountability in AI agent workflows

An agent-facing Golden Path should define the execution telemetry data and audit evidence that every run must produce. Capture model calls, tool selection, retrieved context, latency, errors, retries, evaluations, and cost. Datadog Agent Observability consolidates this telemetry data for you in unified traces.

Datadog Agent Observability trace showing an Investment Agent span that failed after exceeding its maximum turn limit, with agent manifest and instructions visible for debugging.
Datadog Agent Observability trace showing an Investment Agent span that failed after exceeding its maximum turn limit, with agent manifest and instructions visible for debugging.

A trace explains how the agent run unfolded, while an audit event records an action taken against a platform resource. For each external action, record the task identity, credential or workload identity used, and the affected resource, requested operation, approval state, timestamp, and outcome. 

At Datadog, when an AI agent such as Bits Chat performs a supported action on a user’s behalf, Audit Trail records the agent as the actor and the user as the delegator. This distinction identifies who performed the action and under whose authority.

Datadog Audit Trail entry recording an AI agent’s workflow action with both the delegating user and the agent identified as co-authors of the event.
Datadog Audit Trail entry recording an AI agent’s workflow action with both the delegating user and the agent identified as co-authors of the event.

Correlating audit events and agent traces through the task identity created at dispatch gives teams a continuous record from the signal that initiated the work to the changes the agent ultimately made.

Key takeaways: Evolving Golden Paths for AI agents

Golden Paths do not disappear when platform users become agents. They multiply because conversational, long-running, and code-executing agents have different requirements for latency, durability, and isolation. Supporting their execution patterns safely requires machine-consumable capability contracts, authoritative service context, controls enforced outside the agent, and evidence that includes the details of each run. Begin with one bounded path, verify that the agent selects appropriate tasks and operates within its constraints, and then expand its authority incrementally.

To start adapting a path, use the Datadog MCP Server to give agents structured access to Datadog capabilities and set up Datadog Agent Observability to trace how they use them. 

To start monitoring the systems and workflows that support your AI agents, .

Start monitoring your metrics in minutes