Use OpenTelemetry with Observability Pipelines for vendor-neutral log collection and cost control

Today, many DevOps and security teams operate in a world of complex, hybrid, or multi-vendor environments. As more teams look to avoid lock-in by adopting open standards, OpenTelemetry (OTel) is quickly gaining adoption as the primary open source method for DevOps and security teams to instrument and aggregate their telemetry data. However, OTel alone may lack the advanced processing functions, native volume control rules, and hybrid environment support that large organizations need. This results in a fragmented setup, where teams struggle to reduce the noise sent to their downstream destinations and enforce standardized tagging.

Datadog Observability Pipelines now supports the OTel Collector as a logs source, enabling your teams to instrument and collect telemetry data with OTel before processing and routing to your preferred destinations. Using OTel with Observability Pipelines helps teams maintain a vendor-neutral architecture and ensure their telemetry data is shaped for their desired taxonomy.

In this post, we’ll describe how to:

• Collect, process, and route data with a vendor-neutral architecture
• Unify OTel collection with pre-ingestion volume control and parsing rules
• Automatically handle the OTLP protobuf for simplified data manipulation

Collect, process, and route data with a vendor-neutral architecture

For DevOps and security teams, managing growing data volumes while staying within budget is a never-ending battle with various sources and different formats. As a result, many teams are turning to vendor-neutral tooling, schemas, or instrumentation. Observability Pipelines already supports open source frameworks by providing out-of-the-box normalization to the Open Cybersecurity Schema Framework (OCSF), a vendor-neutral data format for security teams that use vendors such as Splunk, Datadog Cloud SIEM, Amazon Security Lake, Google SecOps (Chronicle), Microsoft Sentinel, and CrowdStrike. Additionally, Datadog enables DevOps teams to take advantage of Vector Remap Language (VRL) using the Custom Processor for advanced data transformation and historical logs migrations.

Now, with OTel and OCSF support, Observability Pipelines gives you a single solution to collect and transform your logs in open standards for both DevOps and security use cases. This enables you to easily instrument telemetry collection with OTel, process and remap to OCSF with Observability Pipelines, and route data to your preferred tools under a fully vendor-agnostic architecture.

Close dialog

For example, let’s say you’re an architect at a large fintech organization responsible for standardizing your telemetry instrumentation with OTel for on-premises observability and security. Your company uses Datadog CloudPrem for DevOps troubleshooting along with Splunk for security. While OTel enables you to minimize vendor lock-in for telemetry collection, you still need the ability to preprocess and split logs for ingestion into your preferred tools. With Observability Pipelines, your team can easily ingest logs from OTel and route without compromising visibility, sacrificing compliance, or breaking any of your current security or DevOps workflows.

Unify OTel collection with pre-ingestion volume control and parsing rules

While collecting logs, metrics, and traces with OTel enables teams to use an open source schema and data export method, using OTel alone does not meet all of the modern DevOps requirements. For example, OpenTelemetry offers minimal data normalization capabilities, lacks sophisticated volume management, and struggles to operate efficiently across hybrid environments. This increases the barrier of adoption and forces teams to further rely on legacy or proprietary tooling to accomplish their goals.

To help mitigate conflicts between OTel and third party instrumentation in hybrid environments, Observability Pipelines enables you to centralize data aggregation across OTel and non-OTel source integrations—including Splunk HEC, HTTP, Syslog, and more. This flexibility allows you to centralize data processing across hybrid and multi-cloud environments, regardless of whether data originates from VMs, Kubernetes, Docker, on-prem forwarders, or cloud services.

Once you’ve connected your sources to Observability Pipelines, you can take advantage of out-of-the-box processors—including 150+ preset parsing rules in the Grok Parser—to help you parse unstructured data on-stream before routing to your preferred tools. Additionally, you can enforce compliance requirements by redacting sensitive data on-premises before routing to your preferred destinations.

Close dialog

Once data is normalized, you can generate metrics from higher volume datasets like CDN logs from popular vendors such as Akamai, Fastly, and Cloudflare. This ensures any KPI is extracted, while the log can be dropped or routed directly to cloud storage archives such as Amazon S3, Azure Blob Storage, or Google Cloud Storage.

For example, let’s say you’re the Head of Platform Engineering at a large telecommunications organization. Your security team runs CrowdStrike in Kubernetes, while your DevOps team has a large on-prem footprint in Splunk. As your organization standardizes on OTel, you need a way to reliably collect telemetry data from both your on-prem and cloud-native environments. With Observability Pipelines, you can unify data collection across VMs, clusters, and legacy forwarders using both OTel and non-OTel sources. This ensures your teams can use one tool to process and route data to their destinations without disrupting existing workflows or sacrificing visibility.

Automatically handle the OTLP protobuf for simplified data manipulation

As teams look to standardize on OTel, manipulating logs requires data to be in a transformable format. The OTel Collector natively outputs log data through gRPC or HTTP in a Protocol Buffers (protobuf) format called OTLP (OpenTelemetry Protocol), resulting in a nested structure:

1
ResourceLogs: object containing a single Resource and 1+ ScopeLogs
2
    ScopeLogs: object containing a single Scope and 1+ LogRecords
3
     LogRecords: log event

Because of the nested format, teams are forced to parse and flatten upon ingestion, creating extra effort and delay. However, Observability Pipelines automatically extracts each LogRecord as its own event and lifts contextual metadata such as resource.attributes and scope.name into the log as a field or metadata. This use case is similar to splitting nested arrays, where each unique LogRecord entry retains the resource and scope from its parent objects. To visualize this, here is an example of an Okta User Session Start log event that is commonly used by security teams to understand potential IAM threats.

The following log reflects an example of a web store app in the OTLP protobuf:

1
logs_data {
2
  resource_logs {
3
    resource {
4
      attributes { key: "service.name"  value { string_value: "web-store" } }
5
      attributes { key: "host.name"     value { string_value: "i-082a019146cfa596a" } }
6
      attributes { key: "env"           value { string_value: "prod" } }
7
      attributes { key: "team"          value { string_value: "shopist-web" } }
8
      attributes { key: "k8s.pod.name"  value { string_value: "rails-storefront-main-7b9cb88476-gvsdr" } }
9
      attributes { key: "k8s.container.name" value { string_value: "rails-storefront" } }
10
    }
11

12
    scope_logs {
13
      scope {
14
        name: "datadog.op.pipeline"
15
      }
16

17
      log_records {
18
        time_unix_nano: 1763599982812000000
19
        severity_number: SEVERITY_NUMBER_ERROR
20
        severity_text: "ERROR"
21

22
        body { string_value: "/checkout.json" }
23

24
        attributes { key: "dd.id"          value { string_value: "AwAAAZqeLaTc..." } }
25
        attributes { key: "log.level"      value { string_value: "ERROR" } }
26
        attributes { key: "logger.name"    value { string_value: "Rails" } }
27
        attributes { key: "controller"     value { string_value: "ShoppingCartController" } }
28
        attributes { key: "action"         value { string_value: "checkout" } }
29

30
        attributes { key: "http.method"        value { string_value: "POST" } }
31
        attributes { key: "http.status_code"   value { string_value: "503" } }
32
        attributes { key: "url.path"           value { string_value: "/checkout.json" } }
33

34
        attributes { key: "duration.ms"        value { double_value: 6041.41 } }
35

36
        attributes {
37
          key: "error.stack"
38
          value {
39
            array_value {
40
              values { string_value: "PaymentServiceUnavailableError" }
41
              values { string_value: "Payment service reported 503 Unavailable." }
42
            }
43
          }
44
        }
45

46
        attributes {
47
          key: "params"
48
          value { string_value: "{\"checkout\":{...},\"format\":\"json\"}" }
49
        }
50
      }
51
    }
52
  }
53
}

The following log reflects an example of a web store app in parsed JSON:

1
{
2
  "id": "AwAAAZqeLaTcduNdewAAABhBWnFlTGFyMkFBQXRSRm9TUHdzVFFRUGMAAAAkZjE5YTllMmQtYTRkZC00NTZlLWE1NDMtNjA2NjkyYzE1NWZhAAAAAQ",
3
  "content": {
4
    "timestamp": "2025-11-19T22:13:02.812Z",
5
    "tags": [
6
      "env:prod",
7
      "kube_replica_set:rails-storefront-main-7b9cb88476",
8
      "swarm_service:web-store",
9
      "kube_container_name:rails-storefront",
10
      "image_id:172597598159.dkr.ecr.us-east-1.amazonaws.com/rails-storefront@sha256:38d30266f98f38f8cbdb872ebbebbe51aca7eb705f2574c37bb7f6281d398dd1",
11
      "chart_name:rails-storefront",
12
      "container_name:rails-storefront",
13
      "short_image:rails-storefront",
14
      "deployment_group:main",
15
      "tags.datadoghq.com/service:web-store",
16
      "filename:117.log",
17
      "kube_qos:burstable",
18
      "app:rails-storefront",
19
      "pod_name:rails-storefront-main-7b9cb88476-gvsdr",
20
      "image_tag:3875212b",
21
      "kube_namespace:default",
22
      "kube_deployment:rails-storefront-main",
23
      "swarm_namespace:web",
24
      "service:web-store",
25
      "team:shopist-web",
26
    ],
27
    "host": "i-082a019146cfa596a",
28
    "service": "web-store",
29
    "message": "/checkout.json",
30
    "attributes": {
31
      "controller": "ShoppingCartController",
32
      "level": "ERROR",
33
      "logger": {
34
        "name": "Rails"
35
      },
36
      "format": "json",
37
      "source_type": "datadog_agent",
38
      "source": " nginx",
39
      "params": {
40
        "shopping_cart": {
41
          "checkout": {
42
            "cvc": 736,
43
            "card_number": "fa55e253024cdcff",
44
            "exp": "10/21"
45
          }
46
        },
47
        "format": "json",
48
        "checkout": {
49
          "cvc": 736,
50
          "card_number": "fa55e253024cdcff",
51
          "exp": "10/21"
52
        }
53
      },
54
      "error": {
55
        "stack": [
56
          "PaymentServiceUnavailableError",
57
          "Payment service reported 503 Unavailable."
58
        ]
59
      },
60
      "ddsource": "ruby",
61
      "duration": 6041.41,
62
      "view": 0,
63
      "allocations": 1740522,
64
      "action": "checkout",
65
      "http": {
66
        "url_details": {
67
          "path": "/checkout.json"
68
        },
69
        "status_code": "503",
70
        "method": "POST",
71
        "status_category": "ERROR"
72
      },
73
      "exception_object": "Payment service reported 503 Unavailable.",
74
      "timestamp": "2025-11-19T22:13:02.812408Z",
75
      "status": "ERROR"
76
    }
77
  }
78
}

Aggregate from OTel and process with Observability Pipelines

Datadog Observability Pipelines can aggregate telemetry data from the OTel Collector and supports your taxonomy requirements with out-of-the-box volume control and no vendor lock-in. Observability Pipelines also enables Grok parsing with over 150 preconfigured parsing rules and custom parsing capabilities, GeoIP enrichment, JSON parsing, and more.

Observability Pipelines can be used without any subscription to Datadog Log Management or Datadog Cloud SIEM. For more information, visit our documentation. If you’re new to Datadog, you can sign up for a 14-day free trial.