---
title: "Build and troubleshoot Observability Pipelines with AI agents"
description: " Use AI agents and the Datadog MCP Server to build and validate pipelines, troubleshoot pipeline health, and measure log volume reduction."
author: "Zara Boddula"
date: 2026-11-20
tags: ["observability pipelines", "ai", "mcp server"]
blog_type_id: the-monitor
locale: en
---

Managing a telemetry pipeline often involves recurring investigation and verification tasks. When you onboard a new log source, you look up which components exist and which fields each one requires. When logs stop arriving, you trace where events drop. Before you change a pipeline, you validate the new configuration. After you add a filter to cut costs, you measure the volume it removed and check for unintended drops. An AI agent can help with each task if it can access your pipeline configurations and metrics.

[Datadog Observability Pipelines](https://docs.datadoghq.com/observability_pipelines.md) tools are now generally available in the [Datadog MCP Server](https://www.datadoghq.com/product/ai/mcp-server/). Observability Pipelines lets you collect, transform, and route logs and metrics in your own environment before they reach their destinations. With the new `observability-pipelines` toolset, AI agents in clients such as Claude Code, Cursor, Codex, and VS Code can inspect your existing pipelines and help you draft, validate, and create new ones.

This post covers four workflows:

- [Create a pipeline from a prompt](#create-a-pipeline-from-a-prompt)
- [Check the health of a pipeline](#check-the-health-of-a-pipeline)
- [Validate pipeline changes before you apply them](#validate-pipeline-changes-before-you-apply-them)
- [Verify volume reduction](#verify-volume-reduction)

We'll also explain the guardrails that help your team [stay in control of what gets deployed](#stay-in-control-of-what-gets-deployed).

## Create a pipeline from a prompt

Onboarding a new log source usually means knowing which source, processor, and destination types exist, which fields each one requires, and how they connect to form a valid pipeline. With the Observability Pipelines MCP tools, you describe the outcome, and your agent can assemble the configuration.

### Describe the pipeline you need

For example, when a new source sends Splunk HTTP Event Collector (HEC) logs that contain payment data, you might ask your agent:

```
Build a pipeline that collects Splunk HEC logs, redacts credit card numbers, and routes them to Amazon S3 and Splunk.
```

Your agent works through three steps:

1. **Discover components**: Without arguments, [`get_observability_pipeline_component_schema`](https://docs.datadoghq.com/mcp_server/tools.md#get_observability_pipeline_component_schema) returns the available source, processor, and destination types. When your agent requests specific components, such as the splunk_hec source and the `amazon_s3` destination, the tool returns their fields, including which are required, their types, and allowed values.
1. **Validate the draft**: [`validate_observability_pipeline`](https://docs.datadoghq.com/mcp_server/tools.md#validate_observability_pipeline) checks the configuration without creating or deploying anything. It returns all validation errors at once, so your agent can fix the configuration and validate it again.
1. **Create the pipeline**: [`create_observability_pipeline`](https://docs.datadoghq.com/mcp_server/tools.md#create_observability_pipeline) returns the new pipeline’s ID and a link to open it in Datadog. Creating a pipeline doesn’t affect your existing pipelines. The new pipeline doesn’t process data until you [install the Observability Pipelines Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md) for it.

### Review the new pipeline

In the following example, the agent drafts and validates the configuration and summarizes each component.

![An agent drafts a pipeline that redacts credit card numbers from Splunk HEC logs and routes them to S3 and Splunk.](https://web-assets.dd-static.net/42588/1795111490-2-agent-drafts-splunk-hec-pipeline.png)

After the agent creates the pipeline, you can review its source, processor, and destinations in the pipeline builder.

![The pipeline builder shows an inactive pipeline that redacts credit card numbers and routes logs to S3 and Splunk.](https://web-assets.dd-static.net/42588/1795111559-3-splunk-hec-pipeline-in-builder.png)

### Build pipelines for other use cases

You can use the same approach for other common use cases. For example, you can ask your agent to [dual ship](https://www.datadoghq.com/blog/observability-pipelines-dual-ship-logs.md) logs to Datadog and a SIEM or [archive raw logs](https://docs.datadoghq.com/observability_pipelines/configuration/explore_templates.md?tab=logs#archive-logs) to cloud storage. You can also have it send security logs and application logs to different destinations. [Supported destinations](https://docs.datadoghq.com/observability_pipelines/destinations.md?tab=logs) include Datadog, Splunk, Amazon S3, Google Security Operations, and Microsoft Sentinel.

## Check the health of a pipeline

When a dashboard goes flat or an alert stops firing, a common first question is whether the logs are still arriving. Your agent can use the `observability-pipelines` and `core` [toolsets](https://docs.datadoghq.com/mcp_server/setup.md?tab=antigravity#available-toolsets) together. With both toolsets, it can pair a pipeline’s configuration with the metrics that its Observability Pipelines Workers send to Datadog.

When you’re on call, you might ask your agent:

```
Is the Prod Agent Logs pipeline healthy? Are any events being dropped before they reach their destinations?
```

Your agent first finds the pipeline with [`list_observability_pipelines`](https://docs.datadoghq.com/mcp_server/tools.md#list_observability_pipelines). It then maps the pipeline’s sources, processors, and destinations with [`describe_observability_pipeline`](https://docs.datadoghq.com/mcp_server/tools.md#describe_observability_pipeline). Finally, it queries the[ pipeline usage metrics](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md) tagged with the pipeline’s `pipeline_id` and each `component_id`. These metrics give your agent the same signals you’d find on the pipeline’s [Health and Workers tabs](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines.md):

- Events received and sent by each component
- Events discarded intentionally and unintentionally
- Errors for each destination, and requests for HTTP-based destinations
- Destination buffer size 
- Worker CPU and memory usage

Your agent can summarize these findings in plain language. It can explain which destinations are returning errors, whether buffers are filling up, and whether event drops began after a configuration change. In the following example, the agent reports no dropped events and no errors over 24 hours, and notes that 28 percent of events in the last four hours were rerouted to an S3 overflow bucket.

![An agent reports zero dropped events, zero errors, and 28% of events rerouted to S3 over the last four hours.](https://web-assets.dd-static.net/42588/1795111718-4-agent-pipeline-health-summary.png)

## Validate pipeline changes before you apply them

Pipelines often need updates. For example, a new compliance rule might require you to redact another field, or a noisy service might need a filter. A misconfigured change can break the dashboards, alerts, and detections that depend on the pipeline. Your agent can draft and validate each change before you review and apply it.

### Describe the change

To filter a noisy service and redact a new field in the Splunk HEC pipeline from the previous example, you might ask your agent:

```
Add a step to the Splunk HEC pipeline that drops debug logs from the checkout service, and redact email addresses before the S3 archive. Show me what would change.
```

Your agent retrieves the current configuration with `describe_observability_pipeline`, explains the current flow, and drafts the updated configuration with the new filter and redaction processors. It then checks the draft with `validate_observability_pipeline` and returns a before-and-after summary.

From there, you decide how to apply it. You can review and deploy the change in the pipeline builder, apply it through your existing [Terraform or API workflow](https://docs.datadoghq.com/observability_pipelines/guide/manage_pipelines_with_terraform_and_api.md), or have your agent create it as a new pipeline with `create_observability_pipeline` and test it side by side before cutting over.

### Review the proposed change

In the following example, the agent shows a configuration diff and the pipeline before and after the change. The agent adds a [filter](https://docs.datadoghq.com/observability_pipelines/processors/filter.md) that drops debug logs from the checkout service before they reach either destination. It also adds a separate S3 route with a [Sensitive Data Scanner](https://docs.datadoghq.com/observability_pipelines/processors/sensitive_data_scanner.md?tab=libraryrules) rule that redacts credit card numbers.

![An agent shows a before-and-after diagram and configuration diff that adds a filter and an email redaction rule.](https://web-assets.dd-static.net/42588/1795186688-5-agent-shows-config-diff.png)

The agent then confirms that the proposed edit passes validation and that nothing in the deployed pipeline has changed. It asks whether to apply the edit as shown.

![An agent reports that a proposed pipeline edit passes validation and asks which version to apply.](https://web-assets.dd-static.net/42588/1795111851-6-agent-asks-to-apply-edit.png)

### Apply the change

After you approve the edit, the agent creates the updated configuration as a new pipeline. In the pipeline builder, you can review what each Sensitive Data Scanner rule looks for, which part of the event it scans, and how it redacts what it finds.

![The Edit Scanning Rule panel shows a credit card rule’s regex, scan target, action, and replacement text.](https://web-assets.dd-static.net/42588/1795111927-7-edit-scanning-rule-panel.png)

## Verify volume reduction

A telemetry pipeline helps you control the volume that reaches your most expensive destinations. [Common strategies for reducing log volume](https://docs.datadoghq.com/observability_pipelines/guide/strategies_for_reducing_log_volume.md) include dropping low-value events, sampling noisy sources, and converting high-volume logs into metrics. Measuring the result is harder. You need to show how much each change reduced volume and check that the pipeline isn’t dropping events unintentionally. 

### Add a pack to reduce log volume

[Observability Pipelines Packs](https://docs.datadoghq.com/observability_pipelines/packs.md) are predefined configurations for specific log sources, such as AWS CloudTrail, Palo Alto Firewall, Zscaler, Cloudflare, Akamai, and Fastly. Each pack applies Datadog-recommended processing that defines which logs to drop, which to keep or parse, and which fields to remove. To [reduce the log volume you send to your SIEM or data lake](https://www.datadoghq.com/blog/manage-high-volume-logs-with-observability-pipeline-packs.md), add a pack to a new or existing pipeline. Then ask your agent to measure how much volume the pack removed.

### Measure what the pack removed

After you add the [Fortinet Firewall pack](https://docs.datadoghq.com/observability_pipelines/packs/fortinet_firewall.md) to a pipeline, you might ask your agent:

```
Compare how many events the Fortinet pipeline sent to Splunk today versus yesterday. How much did the pack remove, and were any unintended events discarded?
```

![An agent compares events removed today and yesterday by filter, showing a 44% reduction and no unintended discards.](https://web-assets.dd-static.net/42588/1795123737-8-agent-volume-reduction-report.png)

Your agent queries events received and sent for each `component_id`, then compares the two periods. Each pipeline usage metric also carries a `rc_version` tag, which increments each time the pipeline is updated. With this tag, your agent can trace the reduction to the configuration version that introduced it. 

The agent then reports how many events each of the pack’s filters removed and whether any unintended discards occurred. In the example above, the pack's filters removed 44 percent of events today, with no unintended discards. You can check the numbers in a Datadog dashboard or notebook by graphing the same [pipeline usage metrics](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md) that the agent queried.

## Stay in control of what gets deployed

The [Observability Pipelines toolset](https://docs.datadoghq.com/mcp_server/tools.md#observability-pipelines) lets your agent explain, draft, validate, and create pipelines. Your team decides when a new pipeline starts processing data. The toolset works within these limits:

- **The toolset doesn’t change existing pipelines**: It can read existing pipelines, but it can’t update or delete them.
- **New pipelines process no data until you install the Worker**: `create_observability_pipeline` returns a link to the new pipeline in Datadog, so you can review it before you [install the Observability Pipelines Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md) to run it.
- **Your agent can validate before it creates**:** **Your agent can check a configuration with `validate_observability_pipeline` without creating anything. If a configuration fails validation, `create_observability_pipeline` returns the errors. 
- **Your permissions apply**: Your agent has the same [access controls](https://docs.datadoghq.com/observability_pipelines/configuration/access_control.md) as the user who connects it. Creating a pipeline requires the Observability Pipelines Deploy permission, and the other tools require Observability Pipelines Read. Your role also needs the [MCP Read permission](https://docs.datadoghq.com/mcp_server/setup.md#required-permissions) for the read tools and the MCP Write permission to create pipelines.

## Get started with Observability Pipelines in the Datadog MCP Server

The Observability Pipelines MCP tools give your AI agent access to the configurations and metrics behind your telemetry pipelines. Your agent can draft new pipelines and changes, check pipeline health, and measure volume reduction. To try the tools:

1. Connect the Datadog MCP Server to your AI client by following the [setup instructions](https://docs.datadoghq.com/mcp_server/setup.md) for that client.
1. Enable the `core` and `observability-pipelines` [toolsets](https://docs.datadoghq.com/mcp_server/setup.md#toolsets). For most clients, add `?toolsets=core,observability-pipelines` to your MCP Server endpoint URL.
1. Try a prompt such as "List our pipelines modified in the last week" or "Create a logs pipeline that receives logs from the Datadog Agent, drops debug logs, and sends the rest to Datadog."

See the [Datadog MCP Server tools documentation](https://docs.datadoghq.com/mcp_server/tools.md#observability-pipelines) for the full list of Observability Pipelines MCP tools and example prompts.

If you’re not already a Datadog customer, <!-- Sign-up trigger (sign up for a free 14-day trial) omitted -->.