Name: Datadog Observability Pipelines
Brand: Datadog

Feature Overview

Datadog Observability Pipelines helps Security and DevOps teams easily control, optimize, and refine logs, metrics, and traces in their environment before routing them to any SIEM, data lake, or storage platform. In just a few clicks, teams can control costs and reduce noise while retaining the visibility needed for threat detection, investigations, and compliance. Built-in integrations with popular vendors, AI-guided configuration, and performance monitoring make it simple for teams to build flexible and reliable pipelines at scale. Observability Pipelines are OTEL and OCSF compatible.

Easily control telemetry costs and noise on the stream

Control costs by automatically filtering, sampling, and dropping low value data safely before it reaches downstream tools with 100+ ready-to-use Packs for common sources
Quickly identify trends in log data with AI-based pattern recognition so you can respond before they impact your budget or performance
Optimize storage costs and maintain visibility by sending critical events to your SIEM while archiving lower-priority logs in S3 with archival search built in
Optimize storage costs and maintain visibility by sending critical events to your SIEM while archiving lower-priority logs in S3 with Archive Search

Cost-effective processing and routing of all your logs

Migrate to new SIEMs without losing visibility

Easily adopt your preferred SIEM, data lake, or logging solution at your own pace without sacrificing visibility by dual shipping your logs to multiple destinations
Take advantage of best-of-breed solutions by classifying and routing logs based on use case—for example, sending security logs to a SIEM and DevOps logs to a log management platform
Integrate seamlessly with new solutions by automatically converting logs to open standards like the Open Cybersecurity Schema Framework (OCSF), OTEL, Splunk CIM, or custom mappings
Route security logs directly to any data lake, such as Datadog BYOC Logs, ClickHouse, Snowflake, or Databricks for long-term, searchable storage

Adopt your preferred SIEM without losing visibility

Pinpoint threats faster with on-stream detection

Automatically standardize logs for faster analysis with AI-assisted Grok parsing, 150+ built-in parsing rules, and custom Grok patterns
Surface new or anomalous changes in log volume before they’re routed to your preferred destinations
Speed up security investigations and improve downstream visibility by adding valuable context—such as GeoIP data, threat intelligence, and network metadata—before logs are indexed
Send high-quality, standardized security logs to your preferred SIEM or data lake with automatic mapping to OCSF

Enforce compliance and governance across tools

Discover, classify, and manage sensitive data in your on-premises logs with 150+ built-in detection rules for PII, PCI, and other regulated data
Help meet data residency laws and support Data Loss Prevention initiatives with built-in or user-defined rules to support compliance with PCI, GDPR, HIPAA, CCPA, and more
Limit gaps in access control and ensure schema standardization by easily adding, copying, or dropping relevant attributes and tags
Accelerate security data normalization with packs that support MITRE ATT&CK mappings and ServiceNow CMDB context
Standardize security telemetry with packs that transform data into Microsoft Sentinel and GoogleSecOps UDM format

Enforce compliance and governance across environments

Quickly create, deploy, and monitor pipelines

Set up and optimize pipelines fast with ready-to-use templates and Packs that streamline data processing, transformation, and routing
Monitor pipeline health in real time with Live Capture and centralized dashboards to diagnose and resolve configuration issues before they affect data delivery
Reduce deployment risk by validating pipeline behavior with before-and-after transformations in Pipeline Simulation before deploying changes to production
Transform raw data into any desired format with AI-assisted Grok parsing, then export to Terraform with a single click for Infrastructure-as-Code (IaC) deployment
Standardize security telemetry with Packs that transform data into Microsoft Sentinel and GoogleSecOps UDM format

Create, deploy, and monitor high-performance pipelines from an intuitive UI

Process metrics and traces with greater control

Collect, process, and route metrics or traces from OpenTelemetry and Datadog through the same pipeline infrastructure you use for logs
Enforce consistent metric tagging across teams by standardizing, adding, copying, or removing metric tags in flight, while controlling metric cardinality to reduce downstream costs
Protect sensitive data with trace redaction and reduce trace volume using head- and tail-based sampling
Apply consistent tagging and enrichment policies across logs, metrics, and traces before routing them to downstream destinations