Feature Overview

Datadog Observability Pipelines helps Security and DevOps teams easily control, optimize, and refine logs and metrics in their environment before routing them to any SIEM, data lake, or storage platform. In just a few clicks, teams can control costs and reduce noise while retaining critical visibility needed for threat detection, investigations, and compliance. Built-in integrations with popular vendors, AI-guided configuration, and performance monitoring make it simple for teams to build flexible and reliable pipelines at scale.

Easily control costs and noise on the stream

Control costs by automatically filtering, sampling, and refining noisy data before it reaches downstream tools with ready-to-use Packs for common sources
Quickly identify trends in log data with AI-based pattern recognition so you can respond before they impact your budget or performance
Optimize storage costs and maintain visibility by sending critical events to your SIEM and archiving lower-priority logs in cost-efficient cold storage or a searchable data lake
Prevent cost overruns by controlling metric volume and applying tag governance rules to eliminate noisy, incomplete, or non-compliant data

Cost-effective processing and routing of all your logs

Migrate to new SIEMs without losing visibility

Easily adopt your preferred SIEM, data lake, or logging solution at your own pace without sacrificing visibility by dual shipping your logs to multiple destinations.
Take advantage of best-of-breed solutions by classifying and routing logs based on use case—for example, sending security logs to a SIEM and DevOps logs to a log management platform
Integrate seamlessly with new solutions by automatically converting logs to open standards like the Open Cybersecurity Schema Framework (OCSF), OTEL, Splunk CIM, or custom mappings
Route security logs directly to any data lake, such as Datadog CloudPrem, ClickHouse, Snowflake, or Databricks for long-term, searchable storage

Adopt your preferred SIEM without losing visibility

Pinpoint threats faster with on-stream detection

Automatically standardize logs for faster analysis with AI-assisted Grok parsing, 150+ built-in parsing rules, and custom Grok patterns
Surface new or anomalous changes in log volume before they’re routed to your preferred destinations
Speed up security investigations and improve downstream visibility by adding valuable context—such as GeoIP data, threat intelligence, and network metadata—before logs are indexed
Send high-quality, standardized security logs to your preferred SIEM or data lake with automatic mapping to OCSF

Enforce compliance and governance across tools

Discover, classify, and manage sensitive data in your on-premises logs with 150+ built-in detection rules for PII, PCI, and other regulated data.
Help meet data residency laws and support Data Loss Prevention initiatives with built-in or user-defined rules to support compliance with PCI, GDPR, HIPAA, CCPA, and more
Limit gaps in access control and ensure schema standardization by easily adding, copying, or dropping relevant attributes and tags

Enforce compliance and governance across environments

Quickly create, deploy, and monitor pipelines

Set up and optimize pipelines fast with ready-to-use templates and Packs that streamline data processing, transformation, and routing
Monitor pipeline health in real time with Live Capture and centralized dashboards to diagnose and resolve configuration issues before they affect data delivery
Deploy high-performance pipelines built on a Rust-based engine to handle petabyte-scale workloads