Observability Pipelines: a guide to sizing, scaling, and performance

September 9, 2026

Introduction

Every organization that deploys Datadog Observability Pipelines (OP) asks the same question: how much compute do I need? The answer depends on what your pipeline does. A pipeline that routes logs untouched uses a fraction of the CPU that a pipeline running complex transformations and processing consumes. This guide provides the data and configuration guidance you need to size your deployment correctly, scale it as volumes grow, and monitor it in production.

We tested three representative pipeline tiers on Observability Pipelines Worker (OPW), each progressively more compute-intensive. The results establish concrete throughput baselines per vCPU, identify the configuration choices with the largest performance impact, and provide prescriptive Kubernetes and VM deployment guidance.

This guide covers:

• Throughput benchmarks across three pipeline processing tiers
• Sensitive Data Scanner (SDS) performance optimization, the single highest-impact configuration choice
• Vertical and horizontal scaling guidance with measured data
• Kubernetes and VM deployment recommendations
• Buffering and backpressure configuration
• What to monitor in production

Key takeaways

1. Always plan conservatively. Datadog’s standard guidance is 1 vCPU per 1TB of logs per day. Organizations may find they achieve better performance than this, however when first starting out it is better to over provision, monitor, and optimize later.
2. Pipeline complexity determines throughput. A basic processing pipeline achieves 4 TB per vCPU per day. Adding SDS with 40 rules and custom VRL drops that to under 1 TB. Size for your actual pipeline, not a generic estimate.
3. SDS scoping is the single highest-leverage optimization. SDS one of the most widely used and compute intensive processors OP offers. Restricting which events SDS scans from all events to only the services that handle sensitive data improves throughput by 22% with 10 rules and 149% with 40 rules. Most organizations can identify the 10-20% of services that actually handle PII.
4. Scale horizontally, not vertically, for SDS-heavy pipelines. Two pods at 1 vCPU each outperform one pod at 2 vCPU by 10%. Independent runtimes eliminate the internal queue contention that limits vertical scaling with high SDS rule counts.
5. For Kubernetes based environments - do not set CPU limits on OPW pods. CFS throttling reduces OPW throughput. Set CPU requests for scheduling and let the pod burst above requests when capacity is available.
6. CPU-based autoscaling can fail for SDS-heavy pipelines. When SDS saturates, workers become I/O-bound waiting for the scanner to drain, and CPU drops. The autoscaler interprets falling CPU as surplus capacity and scales down, amplifying the problem. Use pipeline-aware scaling signals for pipelines with high SDS rule counts.

Understanding pipeline processing tiers

We define three tiers based on the type and volume of processing work the pipeline performs. These are not product categories. They represent the range of pipeline complexity we see in production deployments, from basic routing and enrichment to heavy regex scanning and custom transforms.

Basic Processing

A pipeline focused on routing, filtering, sampling, and field manipulation. No SDS. No custom VRL.

Processors: filter, sample, add fields, JSON parse, rename fields, grok parse (nginx CLF pattern), remove fields, reduce (aggregation on a specific service), tag enrichment.

This tier represents the floor of production pipeline complexity. Most organizations run at least this level of processing. Throughput is CPU-bound by JSON parsing and grok pattern matching.

Medium Processing + SDS

Extends Basic Processing with Sensitive Data Scanner, metrics generation, and deduplication.

Additional processors: SDS with 10 credit card detection rules, generate metrics, and dedupe.

This tier represents a common production pattern: organizations that scan logs events for PII while extracting operational and business metrics from the pipeline. We tested this tier in two configurations:

• Targeted SDS: SDS scans only services that handle sensitive data (approximately 20% of events). This is the recommended configuration.
• Blanket SDS: SDS scans every event regardless of service. This is the default when no filter condition or event boundaries are set, and it carries a significant performance cost.

Heavy Processing + SDS

Extends Medium Processing with a larger SDS rule set, CPU-intensive VRL transforms, and additional enrichment.

Additional processors: SDS expanded to 40 rules (credit card, network, device, and PII pattern sets), additional metrics generation, two custom VRL processors (timestamp format conversion, base64 encoding, user-agent parsing, URL decomposition), and additional tag enrichment.

Organizations at this level typically have regulatory or compliance requirements that demand broad-spectrum PII scanning alongside custom data transformation logic. As with the Medium tier, we tested both targeted and blanket SDS configurations.

Custom VRL processors are added in the OP pipeline by creating a custom processor and writing VRL code. See Remap processor documentation for syntax and examples.

The cost of blanket scanning

The difference between targeted and blanket SDS scanning is the single largest performance variable in these benchmarks. The following table shows the throughput impact of each configuration relative to the Basic Processing baseline:

Blanket scanning with 40 rules consumes 83% of the throughput available to a no-SDS pipeline. A pipeline that processes 100 TB per day at the Basic tier would need over 6x the compute to handle the same volume with 40-rule blanket SDS.

This is why SDS scoping, rule auditing, and field targeting are the most impactful optimizations covered in the companion “” guide.

Sizing reference

The “1 TB per vCPU per day” planning baseline

Datadog’s scaling documentation quotes approximately 1 TB per vCPU per day as a general planning baseline for a 12-processor pipeline. Our testing shows most pipeline configurations achieve more than 1 TB per vCPU per day, making this a safe lower bound for initial capacity planning.

Use the 1 TB figure as a starting point when you do not yet know your pipeline’s exact processor composition. Once you have a pipeline defined, use the tier-specific throughput numbers from the table above for more accurate sizing.

Throughput per vCPU

All results measured at ~1 vCPU, running on AWS EKS with c7a.2xlarge instances (AMD EPYC Genoa). The test workload consisted of seven event types with a weighted average of approximately 2,127 bytes per event, sustained over 30-minute steady-state windows.

“Targeted” vs “blanket” SDS: Targeted SDS uses a filter condition to scan only the services that handle sensitive data (approximately 20% of events in our test workload). Blanket SDS scans every event and log event field. The performance difference ranges from 22% at 10 rules to 149% at 40 rules. We cover additional optimization strategies (SDS “rule-splitting”) in detail in the SDS section.

Sizing formula

We always recommend starting with the 1 vCPU per 1 TB/day formula. This is what we quote all customers on and how we determine licensing. This is conservative, and pipeline complexity can go beyond the high tier outlined in these performance tests. It is best to oversize, then monitor performance using self-reported OPW metrics, and optimize after observing the system for a period of time.

To estimate required vCPU for your deployment:
Required vCPU = (daily volume in TB) / (TB per vCPU per day for your tier)

For example, to process 50 TB per day through a Medium Processing + SDS (targeted) pipeline:
50 TB / 3.61 TB per vCPU per day = 14 vCPU

We recommend adding 25% headroom above this baseline to absorb traffic spikes without triggering autoscaling.

Accounting for diurnal traffic patterns

The sizing formula above treats daily volume as a flat sustained rate. In practice, most log traffic follows a diurnal pattern: business-hours peaks produce significantly more volume than overnight steady state. Most environments see a 2x peak-to-trough ratio, but can be an even higher ratio. Inspect your volume patterns before making decisions about how to scale your fleet.

When planning capacity, size for the peak rate, not the daily average; an example:

Sizing for diurnal patterns:

• Set minReplicas to handle below peak/steady-state at 60% CPU target
• Set maxReplicas to handle peak rate at 60% CPU target
• HPA scales from minReplicas to maxReplicas as traffic transitions from steady state to peak

This approach avoids over-provisioning for when traffic is below the daily average rate, while ensuring the fleet scales to handle peak traffic.

For organizations with known traffic patterns (e.g., higher ratio business-hours peak, batch processing windows, or deploy-driven spikes), measure your actual steady state to peak(s) ratio(s) and adjust accordingly.

Deployment sizing table

These are minimums. Always enable autoscaling for your OPW fleet. We recommend the maximum be set to 1.25 - 2x the baseline to absorb daily fluctuations and unforeseen spikes. Size appropriately based on what you know about your own log volume seasonality.

These configurations assume targeted SDS scanning and include 25% capacity headroom for traffic spikes. The recommendations are intentionally conservative and are weighted toward general planning guideline of 1 vCPU per 1 TB/day, rather than relying solely on the higher throughput observed in our benchmarks. This is especially relevant for the Basic Processing tier, whose benchmarked throughput is significantly higher than the conservative planning baseline.

To determine the number of pods or VMs, we divide the adjusted vCPU requirement by the number of vCPUs assigned to each worker and round up to the next whole number. We recommend a minimum of three workers for high availability.

These are planning baselines. Your actual throughput will vary with event size, event structure, and the specific processors in your pipeline. Observe your pipeline’s actual CPU utilization in production and adjust.

vCPU per pod can be tweaked slightly for better binpacking in kubernetes (say 3.5 vCPU to fit two pods on an 8 vCPU instance).

If your pipeline uses blanket SDS scanning, multiply the Heavy + SDS pod count by approximately 2.5x.

Event size and shape matters

These benchmarks used a heterogeneous workload averaging approximately 2,127 bytes per event. Throughput in bytes per second is relatively stable across event sizes, but throughput in events per second varies. If your events are smaller (under 512 bytes), expect higher events-per-second rates at similar bytes-per-second throughput. If your events are larger (over 4 KB), expect lower events-per-second rates.

The scaling documentation provides two reference points:

• Unstructured logs (approximately 512 bytes per event): approximately 10 MiB/s per vCPU
• Structured logs (approximately 1.5 KB per event): approximately 25 MiB/s per vCPU

Observability Pipelines: Sensitive Data Scanner optimization

For strategies and configurations that help improve SDS performance see the companion guide: OP Sensitive Data Scanner optimization

High availability and scaling strategies

Run every production deployment with high availability (HA) as the default. Whenever possible deploy at least three OPW workers, running as pods or VMs, before you scale for volume. Three is the practical minimum: it lets one worker fail while the others keep serving traffic, and a replacement joins soon after.

Where possible, match that number to your failure domains. On VMs, use one worker per availability zone in the cloud, or one per rack, power source, or network path on-premises. On Kubernetes, spread workers across nodes and availability zones using pod anti-affinity and topology spread constraints.

Once HA is in place, you add capacity two ways:

• Vertical scaling: give each worker more vCPU.
• Horizontal scaling: add more workers.

Horizontal scaling is the better default. It outperforms vertical scaling at every pipeline tier. The advantage grows as SDS load increases.

Vertical scaling still works well for Basic and Medium Processing. Throughput there grows close to linearly with vCPU. For Heavy Processing + SDS, skip vertical scaling. Start at 1 vCPU per worker, then add workers as volume grows.

The three sections below cover the data behind this: vertical scaling limits, horizontal scaling gains, and how a fleet behaves during a worker failure. Kubernetes-specific autoscaling is covered in the next section.

Vertical scaling

For Basic Processing and Medium Processing + SDS pipelines, throughput scales linearly with vCPU. Allocating 2 vCPU to a Basic Processing pod delivers approximately 2x the throughput of 1 vCPU.

For Heavy Processing + SDS pipelines, vertical scaling breaks down quickly: head-of-line blocking creates a throughput ceiling at around 3 vCPU per pod. At 4 vCPU allocated, OPW consumed only 2.82 vCPU under full load. The remaining 1.18 vCPU went unused. Per-vCPU efficiency drops from 1.97 TB/day/vCPU at 1 vCPU to 1.26 TB/day per consumed vCPU at 4 vCPU, a 36% efficiency loss.

• Only 2.82 of 4 allocated vCPU consumed due to queue contention.

** Rule splitting is the act of breaking up a single SDS processor into multiple SDS processors. For more information on rule splitting see the OP Sensitive Data Scanner optimization guide. Recommendation: For unsplit Heavy Processing + SDS pipelines cap vCPU at 3. If you need more than 3 vCPU per pod for Heavy pipelines, split your SDS rules first (see above), then scale pods horizontally.

Horizontal scaling

Horizontal scaling outperforms vertical scaling for all pipeline tiers, and the advantage is most significant for SDS-heavy pipelines.

Each OPW worker, whether a pod or a VM, runs an independent Tokio async runtime. Multiple workers eliminate the shared queue contention that limits single-worker vertical scaling. For this reason, we recommend scaling by adding workers rather than increasing per-worker vCPU, particularly for pipelines with SDS.

High availability

Always deploy at least three OPW workers, running as pods or VMs. Where possible, match that number to your failure domains: one worker per availability zone in the cloud, or one per rack, power source, or network path on-premises.

In our HA simulation, killing one pod in a three-replica deployment resulted in:

• Zero events dropped
• The surviving pods absorbed 100% of load within 36 seconds
• Full throughput recovered when the replacement pod started

Kubernetes deployment

Resource configuration

resources:
  requests:
    cpu: "3"
    memory: "6Gi"
  # Do not set limits.cpu - CFS throttling reduces OPW throughput
  limits:
    memory: "6Gi"

Choosing pod CPU size for node binpacking

The CPU request determines how many OPW pods fit on each node. Kubernetes schedules pods based on requests, and kubelet, DaemonSets (e.g. Datadog Agent, kube-proxy, etc.), and the OS consume a portion of each node’s capacity. If the OPW CPU request is too large, nodes have wasted capacity that cannot be allocated.

For 8-vCPU compute-optimized nodes (c7i.2xlarge, c4a-standard-8, F8s_v2):

At cpu: "4", Kubernetes cannot fit 2 pods on an 8-vCPU node after accounting for kubelet reservations (typically 60-100m) and DaemonSets (e.g. Datadog Agent: 200-400m). The node runs 1 pod and wastes ~4 vCPU.

We recommend cpu: "3" as the default (shown in the examples). For environments that want tighter binpacking with less node overhead, cpu: "3.5" fits 2 pods on an 8-vCPU node with 1.0 vCPU remaining for overhead. Verify your DaemonSet footprint before using 3.5.

Memory follows the same logic: at 2 GiB per vCPU, a 3-vCPU pod requests 6 GiB. Two pods = 12 GiB on a 16 GiB node, leaving 4 GiB for OS and DaemonSets. At 3.5 vCPU: 2 x 7 GiB = 14 GiB, leaving 2 GiB.

CPU requests: Set to the number of vCPU you want OPW to use. The Kubernetes scheduler uses this value for pod placement. We recommend 3 vCPU as the standard pod size.

CPU limits: Do not set. OPW benefits from bursting above its request during traffic spikes. CFS throttling from CPU limits creates artificial backpressure that is difficult to diagnose. CFS only guarantees the CPU requests, and only allows bursting if capacity exists on the node. For more details on defining CPU limits, see this blog post.

Memory requests: 2 GiB per vCPU. OPW’s actual memory consumption is modest, but the 2 GiB figure provides headroom for destination buffers, jemalloc page retention, and kernel caches.

Memory limits: Set equal to requests. This provides predictable scheduling, avoids memory overcommit, and protects against eviction under node memory pressure (Guaranteed QoS on memory). OPW’s actual memory consumption is modest. The 2 GiB per vCPU allocation already provides headroom for jemalloc page retention and in-memory buffer spikes.

Why not sub-1 vCPU per pod?

Kubernetes allows fractional CPU requests (e.g., cpu: "500m"), and in theory you could run OPW pods at less than 1 vCPU each. In practice, this is not recommended for several reasons:

• OPW is CPU-bound. At sub-1 vCPU, each pod processes proportionally less data, so you need many more pods for the same total throughput. The scheduling overhead and PVC count scale linearly with pod count.
• SDS performance data shows that >= 1 vCPU pods can actually help eliminate head-of-line blocking (HOL blocking requires multiple threads to manifest). This makes 1 vCPU the minimum unit for SDS-heavy pipelines.
• The memory overhead per pod (jemalloc, destination buffers, runtime) is roughly fixed regardless of CPU allocation. At sub-1 vCPU, the memory-to-CPU ratio becomes inefficient.

If your use case genuinely requires lower resource consumption per pod, use 1 vCPU pods with more aggressive horizontal scaling rather than fractional CPU. Each 1-vCPU pod runs an independent runtime with no shared queue contention.

Choose your autoscaling strategy

OPW supports two autoscaling approaches on Kubernetes. One is CPU-based, using the Horizontal Pod Autoscaler (HPA). The other is metric-based, using KEDA (Kubernetes Event-Driven Autoscaling) or the Datadog Pod Autoscaler (DPA). The right choice depends on your pipeline’s SDS configuration.

Autoscaling with HPA

CPU-based HPA works well:

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
        - type: Pods
          value: 5
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 900
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
      selectPolicy: Min

Scale-up policy: React within 1-2 minutes to load increases. The selectPolicy: Max ensures the HPA adds whichever is larger: 50% of current pods or 5 pods.

Scale-up pod count should match fleet size. The Pods policy in the example adds 5 pods per minute, which is appropriate for fleets under 25 pods. For larger fleets, increase proportionally: at 75+ pods, use 25 pods/min. A fleet of 100 pods adding only 5/min takes 10+ minutes to double, which is too slow for sudden traffic spikes.

Scale-down policy: 15-minute stabilization window with a conservative 10% reduction per minute. This prevents flapping as traffic spikes subside.

Target CPU: We recommend 70%, which provides 30% burst headroom per pod. The official scaling documentation recommends 85% with a 5-minute stabilization window. Both are valid approaches. Use 85% if your traffic is predictable with gradual ramps. Use 70% (or less) if your traffic is spiky or bursty with sharp increases that need to be absorbed before the autoscaler reacts.

KEDA for SDS-heavy pipelines

For pipelines with more than 20 SDS rules, replace CPU-based HPA with KEDA scaling on pipeline-aware metrics. CPU can be a misleading signal when SDS saturates, as described in the case study found in the companion guide: OP Sensitive Data Scanner optimization.

Recommended KEDA scaling signals, in order of reliability:

1. pipelines.source_buffer_utilization_mean > 50% of capacity: The most reliable backpressure indicator. This EWMA gauge rises under all backpressure conditions, including SDS saturation.
   1. Important: This metric reports raw event counts, not a 0-1 ratio. The maximum value equals vCPU x 1,000 (e.g., 3,000 for a 3-vCPU pod, 4,000 for a 4-vCPU pod). Set KEDA thresholds as absolute values: for a 3-vCPU pod, 50% of capacity = 1,500.
2. kubernetes.cpu.usage > 60%: Complements buffer utilization for compute-bound scenarios.
3. pipelines.component_discarded_events_total{intentional:false} > 0: Emergency signal indicating active data loss.

Do not use pipelines.utilization as a scaling signal. This metric drops to 0 during pipeline stalls and does not reliably indicate backpressure.

The following is a complete KEDA ScaledObject manifest. Adjust minReplicaCount, maxReplicaCount, scaleTargetRef.name, and the <PIPELINE_ID> placeholders to match your deployment:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-keda-secret
  namespace: observability-pipelines
type: Opaque
data:
  apiKey: <BASE64_DD_API_KEY>
  appKey: <BASE64_DD_APP_KEY>
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: datadog-auth
  namespace: observability-pipelines
spec:
  secretTargetRef:
    - parameter: apiKey
      name: datadog-keda-secret
      key: apiKey
    - parameter: appKey
      name: datadog-keda-secret
      key: appKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: opw-scaledobject
  namespace: observability-pipelines
spec:
  scaleTargetRef:
    kind: StatefulSet
    name: opw-observability-pipelines-worker
  minReplicaCount: 3
  maxReplicaCount: 20
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 60
          policies:
            - type: Percent
              value: 50
              periodSeconds: 60
            - type: Pods
              value: 5
              periodSeconds: 60
          selectPolicy: Max
        scaleDown:
          stabilizationWindowSeconds: 900
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60
          selectPolicy: Min
  triggers:
    # Primary: backpressure (buffer utilization)
    # IMPORTANT: source_buffer_utilization_mean reports raw event counts, NOT 0-1.
    # Max value = vCPU x 1000. For 3-vCPU pods: 50% of 3000 = 1500.
    # Adjust queryValue when changing pod vCPU size.
    - type: datadog
      metadata:
        query: "avg:pipelines.source_buffer_utilization_mean{pipeline_id:<PIPELINE_ID>}"
        queryValue: "1500"
        queryAggregator: "avg"
        age: "120"
        metricUnavailableValue: "0"
      authenticationRef:
        name: datadog-auth
    # Secondary: CPU saturation
    - type: datadog
      metadata:
        query: "avg:kubernetes.cpu.usage{kube_stateful_set:opw-observability-pipelines-worker}"
        queryValue: "1800000000"
        queryAggregator: "avg"
        age: "120"
        metricUnavailableValue: "0"
      authenticationRef:
        name: datadog-auth
    # Emergency: active data loss
    - type: datadog
      metadata:
        query: "sum:pipelines.component_discarded_events_total{pipeline_id:<PIPELINE_ID>,intentional:false}.as_rate()"
        queryValue: "0"
        queryAggregator: "max"
        age: "60"
        metricUnavailableValue: "0"
      authenticationRef:
        name: datadog-auth

When using KEDA, set autoscaling.enabled: false in your OPW Helm values to prevent conflicts between the HPA and the KEDA ScaledObject.

Datadog Pod Autoscaler (alternative to KEDA)

The Datadog Pod Autoscaler (DPA) is a Kubernetes-native autoscaler that queries Datadog metrics directly, without requiring KEDA as an intermediary. DPA uses the Datadog Cluster Agent to evaluate scaling rules against any metric in your Datadog account.

For OPW, DPA can scale on the same pipeline-aware metrics recommended for KEDA (source_buffer_utilization_mean, CPU, discarded events) but with a simpler operational footprint: no KEDA installation, no separate TriggerAuthentication secrets, and native integration with the Datadog Cluster Agent you may already be running.

Status: DPA is a newer option. Evaluate whether it meets your scaling precision requirements alongside KEDA.

Pod disruption budget

podDisruptionBudget:
  enabled: true
  minAvailable: 1

For deployments with three or more replicas, increase minAvailable to 2.

Pod anti-affinity

Spread replicas across nodes to avoid correlated failures:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - observability-pipelines-worker
          topologyKey: kubernetes.io/hostname

For multi-AZ deployments, add topology spread constraints:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: observability-pipelines-worker

Cost note: Multi-AZ topology spread means cross-zone traffic between agents and OPW pods. AWS charges approximately $0.01/GB for cross-AZ data transfer within the same region. GCP does not charge for cross-zone traffic within the same region. Azure varies by region. At high log volumes, this cost can be meaningful. If cost is a concern, consider co-locating OPW pods in the same AZ as the majority of your agent traffic, or running a decentralized deployment with OPW per AZ.

Instance selection

Use compute-optimized instances with at least 8 vCPU per node:

Avoid burstable instances (AWS t-family, Azure B-series). OPW under sustained load will exhaust CPU credits and throttle.

Additional recommendations

• podManagementPolicy: Parallel in the StatefulSet spec enables fast HPA scale-up. The default OrderedReady starts pods one at a time.
• Pin to a specific image tag. Control when you adopt new releases. Review the changelog.
• Dedicated node pool. Isolate OPW from application workloads to prevent resource contention.
• jemalloc tuning: Set MALLOC_CONF="thp:never,dirty_decay_ms:1000,muzzy_decay_ms:1000" to force jemalloc to return pages faster after traffic bursts.
• Update strategy: Set type: RollingUpdate with maxUnavailable: "10%" in the StatefulSet to limit disruption during Helm upgrades.

Karpenter and node provisioning

OPW is compatible with Karpenter for dynamic node provisioning. Key considerations:

• PVC zone affinity: OPW uses a StatefulSet with PersistentVolumeClaims. When Karpenter provisions a new node for a rescheduled OPW pod, the node must be in the same availability zone as the pod’s existing PVC (ReadWriteOnce volumes are zone-bound). Configure Karpenter’s NodePool with topology constraints that match your storage zones.
• Instance selection: Use Karpenter’s requirements to constrain instance types to compute-optimized families and exclude burstable types. Example:

requirements:
  - key: karpenter.k8s.aws/instance-family
    operator: In
    values: ["c7i", "c7g", "c7a"]
  - key: karpenter.k8s.aws/instance-size
    operator: In
    values: ["xlarge", "2xlarge"]

• Consolidation: Karpenter’s consolidation feature may attempt to pack OPW pods onto fewer nodes during low-traffic periods. This is generally safe given OPW’s shared-nothing architecture, but can conflict with pod anti-affinity preferences. Monitor for unexpected co-location after consolidation events.

Reference configuration: Kubernetes (10 TB/day)

The following Helm values are a complete, production-ready starting point for a 10 TB/day deployment using the conservative guidance of 1TB/day per vCPU. Every value is commented with rationale and scaling instructions.

• Conservative baseline: 1 TB/day per vCPU
• 10 TB / 1 TB/vCPU = 10 vCPU + 50% headroom = 15 vCPU
• maxReplicas = 2x minReplicas for diurnal (or other) peaks
• Assumes dedicated node groups for OPW pods (recommended)
  • For centralized or hybrid approaches, a dedicated OP cluster is recommended

See this values.yaml for 10 TB/day. 100 TB/day and 300 TB/day references are also available.

To scale this config to a different volume: multiply pod counts linearly, 50 TB/day = 5x this config, however there are other configuration options to consider at higher throughput (see references).

For lower volumes the risk is yours as to whether or not to run fewer than 3 replicas and fewer max replicas. You may also adjust the number of vCPU per pod to something lower as referenced in the earlier sizing tables (see “Deployment sizing table”).

VM deployment

Instance sizing

Use compute-optimized instances. OPW is CPU-bound, and memory consumption is modest.

Graviton, Axion, and Cobalt instance types also work well.

Instance group and autoscaling

Deploy OPW in a managed instance group (AWS ASG, GCP MIG, Azure VMSS) behind a network load balancer (L4). Do not use an application load balancer (L7). OPW’s traffic is high-throughput, low-latency TCP, and L7 inspection adds unnecessary overhead and latency.

Configure the autoscaler to:

• Scale up at 70% average CPU utilization. This provides 30% headroom per instance for traffic spikes. For SDS-heavy pipelines the recommended approach is to use KEDA or DPA on pipeline metrics described above.
• Minimum 3 instances for high availability.
• Cap individual instances at 50% of total pipeline volume. No single instance should process more than half the total traffic.
• Enable the OP API for health checks: set DD_OP_API_ENABLED=true and DD_OP_API_ADDRESS=0.0.0.0:8686. Point the load balancer health check at port 8686.

Architecture

OPW is a share-nothing architecture. Every instance processes events independently. There are no leader nodes, no consensus protocols, and no instance-to-instance communication. Datadog’s backend handles coordination for features that require it (such as quota processors).

This means you can scale the instance group freely without worrying about cluster state or rebalancing.

Load balancer configuration

• Protocol: TCP (L4). Do not use an application load balancer (L7) - OPW traffic is high-throughput and L7 inspection adds unnecessary overhead.
• Distribution: Cloud NLBs (AWS NLB, GCP Network LB, Azure Standard LB) use a flow hash algorithm based on the connection’s 5-tuple (source IP/port, destination IP/port, protocol). With many logging clients the hash distributes connections evenly across OPW targets. OPW recycles connections every 300 seconds with 10% jitter. The server sends Connection: close after 5 minutes, forcing the client to reconnect with a new source port and new flow hash ensuring periodic redistribution. This is already built in and enabled by default. For on-premises L4 load balancers (HAProxy, NGINX), use round-robin or least-connections.
• TCP idle timeout: Ensure this exceeds your client-side keep-alive setting to prevent the NLB from closing connections before the client does, which causes connection resets.
  • AWS NLB: defaults to 350 seconds (configurable 60-6000s). 350s is appropriate. Sends TCP RST on expiry.
  • GCP Internal NLB: defaults to 600s (configurable 60-600s). 600s is appropriate. No RST on expiry - stale connections silently reroute. Client keepalives are critical
  • GCP External NLB: defaults to 60s (not configurable). Set client keepalive to 20-25s.
  • Azure Standard LB: 4 min (configurable 4-100 min). Default is silent drop. Increase to at least 6 min so the timeout exceeds OPW’s connection recycling (330s max)
    • Enable TCP Reset (–enable-tcp-reset true) - the default silent drop causes half-open connections.
• Rule of thumb: client keepalive interval < LB timeout / 2
• Low number of clients: If you have a small number logging clients (e.g., syslog aggregators, Splunk Heavy Forwarders), or traffic passes through a NAT gateway that collapses source IPs, the flow hash may produce uneven distribution. In this case: configure sources to use multiple concurrent connections, and right-size OPW pods so each pod can absorb the largest single-client connection.
• Cross-zone load balancing: Enable cross-zone. Even target distribution across OPW pods is critical - a single overloaded pod can trigger saturation and buffer backpressure. Uneven pod-per-AZ distribution is common during scaling events, node failures, and PVC zone affinity
  • AWS NLB: Cross-zone is disabled by default. Enable it. AWS charges ~$0.01/GB for cross-zone data transfer. At high volumes (50+ TB/day), this cost can be significant. If cost is a concern, enforce strict even pod distribution across AZs using topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule (hard constraint) before disabling cross-zone.
  • GCP: Cross-zone is the default behavior and free within a region.
  • Azure: Zone-redundant distribution is the default and free within a VNet
• Client-side load balancing: DNS round-robin, application-level is not recommended. Use a network load balancer for health-aware distribution.

Network requirements

OPW requires outbound HTTPS (port 443) access to several Datadog domains for configuration delivery, metrics reporting, and operational logs. If your environment restricts outbound traffic, allowlist the following domains:

• api.<DD_SITE> - API key and pipeline ID validation at startup
• config.<DD_SITE> - Remote Configuration delivery (polled every 5 seconds)
• http-intake.logs.<DD_SITE> - OPW operational logs
• *.agent.<DD_SITE> - Metrics (subdomain changes with each Worker version)
• obpipeline-intake.<DD_SITE> - Live Capture

For the complete domain list including Linux package installation domains, see Network traffic configuration.

Buffering and backpressure

OPW uses a three-layer buffer chain to manage flow control between the source, processors, and destinations.

Buffer architecture

When a destination is unavailable, events accumulate in the destination buffer. Once the buffer fills, backpressure propagates upstream through the processor chain to the source. The source stops accepting new events, and HTTP clients receive delayed responses (held open) or, if send_timeout_secs is configured, HTTP 503 responses.

Multi-destination caveat: If your pipeline fans out to multiple destinations, backpressure from any single destination blocks all destinations. To prevent one slow destination from blocking your entire pipeline, set when_full: drop_newest on non-critical destination buffers.

Disk buffer configuration

For pipelines where data durability during destination outages is important, configure disk-based destination buffers. In-memory buffers lose all buffered events on pod restart or crash.

Sizing formula:

buffer_per_pod = throughput_per_vCPU x vCPU x buffer_duration_seconds PVC_size = buffer_per_pod x 1.10 (10% filesystem overhead)

Example: a 3 vCPU pod at 10 MiB/s/vCPU with 20 minutes of runway:

3 x 10 MiB/s x 1,200 seconds = 36 GiB buffer 36 GiB x 1.10 = 40 GiB PVC

For 1 hour of runway at the same configuration, budget 120 GiB per pod.

Disk buffer limits: Minimum 256 MB, maximum 5 TB (Worker 2.20.0 and later; 500 GB on earlier versions). The on-disk format uses 128 MB data files with fsync every 500 ms. Data written within the last 500 ms is at risk on an unexpected crash.

Disk buffer corruption from partial writes: If a disk buffer volume reaches 100% capacity (ENOSPC), partial writes can corrupt the buffer. OPW will loop attempting to drain, emitting Events dropped with unprocessable_events errors. To prevent this, always provision the PVC at least 10% larger than the pipeline’s configured max_size. Monitor pipelines.data_dir_available_bytes and alert when free space drops below 15% of capacity.

Drain time and graceful shutdown

When a pod terminates, OPW attempts to drain buffered events to the destination before exiting. The default terminationGracePeriodSeconds of 60 seconds is too short for large buffers.

When OPW receives SIGTERM, it executes a graceful shutdown in this order:

1. The health/readiness API is marked not-serving. Kubernetes stops routing new traffic to the pod.
2. All HTTP sources stop accepting new TCP connections immediately (hyper graceful shutdown). In-flight requests on existing connections are completed.
3. No new events enter the pipeline after in-flight requests finish.
4. Transforms drain naturally as their input channels close.
5. Sinks flush remaining events from their buffers to the destination.
6. Once all components finish, OPW exits.

The drain time formula only needs to account for events already in the pipeline at the moment of SIGTERM, not continuous live ingestion. OPW closes the listener before draining begins.

OPW has an internal graceful shutdown deadline, configurable via DD_OP_GRACEFUL_SHUTDOWN_LIMIT_SECS (default: 60s). If components do not finish within this deadline, they are force-killed. This value must be less than the pod’s terminationGracePeriodSeconds, otherwise kubelet sends SIGKILL before OPW finishes draining. Set terminationGracePeriodSeconds to at least DD_OP_GRACEFUL_SHUTDOWN_LIMIT_SECS + 10.

A second SIGTERM during graceful shutdown triggers immediate exit with no further draining.

Drain time formula:

drain_time = buffer_size_bytes / drain_rate_bytes_per_second

Observed drain rates vary. At 2 vCPU draining to Datadog intake, drain rates are approximately 60 MB/s. A 50 GiB buffer at 60 MB/s takes approximately 15 minutes to drain.

Set terminationGracePeriodSeconds to cover your maximum expected drain time plus margin. For production deployments with disk buffers at 2+ vCPU per pod, 300 seconds (5 minutes) covers most buffer sizes up to 50 GiB. Undrained events persist in the PVC and will be picked up when the pod restarts or scale-out happens again (ordinal pod comes back).

Monitoring your pipeline

Key metrics

These are the metrics we recommend monitoring for every OPW deployment. All pipelines.* metrics are emitted by OPW itself to Datadog.

Throughput:

• pipelines.component_received_bytes_total{component_kind:source} (.as_rate()): Ingest throughput in bytes per second. This is your primary sizing validation metric.
• pipelines.component_sent_event_bytes_total{component_kind:sink} (.as_rate()): Egress throughput. Compare against ingest to measure the effect of filtering and sampling.
• pipelines.component_received_events_total{component_kind:source} (.as_rate()): Ingest throughput in events per second.

CPU and resource utilization:

• pipelines.cpu_usage_seconds_total (.as_rate()): OPW process CPU usage in vCPU. Compare against pod CPU requests to assess capacity.
• pipelines.resident_memory_used_bytes: RSS memory. Should be stable under steady-state load.
• container.cpu.throttled: Must be zero. Non-zero values indicate CFS throttling from CPU limits.

Backpressure:

• pipelines.source_buffer_utilization_mean (EWMA gauge): The single best backpressure indicator. This metric reports raw event counts, not a 0-1 ratio. The maximum value equals vCPU x 1,000 (e.g., 3,000 for a 3-vCPU pod, 16,000 for a 16-vCPU pod). To compute a utilization percentage, divide by pipelines.source_buffer_max_size_events. Values above 50% of capacity indicate the pipeline is not keeping up with ingest. Available in OPW 2.13 and later.
• pipelines.source_lag_time_seconds (distribution): Time between event timestamp and receipt by OPW. Rising lag indicates backpressure. Available in OPW 2.16 and later.
• pipelines.buffer_size_events / pipelines.buffer_size_bytes: Destination buffer fill level. Rising values indicate destination slowness.

Errors and drops:

• pipelines.component_errors_total: Must be zero in a healthy pipeline. Investigate any non-zero value.
• pipelines.component_discarded_events_total{intentional:false}: Active unintentional data loss. This should trigger a critical alert.
• pipelines.component_discarded_events_total{intentional:true}: Intentional discards from filter, sample, quota processors, and other volume control processors. Expected and normal.

SDS-specific:

• pipelines.utilization{component_type:sensitive_data_scanner}: SDS processor saturation. Values consistently above 0.9 indicate SDS is the pipeline bottleneck. Note: this metric can drop to 0 during pipeline stalls and should not be used as an autoscaling signal.
• pipelines.component_received_events_total{component_type:sensitive_data_scanner} (.as_rate()): Events per second entering SDS. Compare against total source events to validate your SDS filter scope.
• pipelines.component_latency_seconds{component_type:sensitive_data_scanner} (distribution): Per-event processing time through SDS. Use this to measure the impact of SDS optimizations (rule reduction, field targeting, rule splitting). Enable percentiles in Datadog Metrics Summary before querying.

Recommended monitors

Metrics catalog

For the complete metric reference including HTTP server/client metrics, adaptive concurrency metrics, and version dependencies, see Pipeline usage metrics.

Datadog provides a built-in dashboard: Observability Pipelines Overview. It covers throughput, component health, buffers, errors, CPU/memory, and SDS matches. No configuration required.

Best practices checklist

Sizing:

• Start with the conservative 1 TB/vCPU/day estimate, observe, then size for your actual pipeline tier
• Add 25% headroom above calculated vCPU requirements
• Cap pods at 4 vCPU; scale horizontally when more capacity is needed
• Budget 2 GiB memory per vCPU

SDS optimization:

• Scope SDS to only the services that handle sensitive data
• Limit which fields SDS scans to reduce per-event regex work
• Audit enabled rules routinely; looking over weeks of data, disable rules with zero matches
• Split rule sets larger than 20 rules across multiple SDS processors
• Place SDS after volume reduction processors when pipeline ordering permits
• Use pipelines.component_latency_seconds{component_type:sensitive_data_scanner} to measure the impact of optimizations

Pipeline design:

• Place filter, reduce, throttle, quota, and sample processors before computationally expensive ones (SDS, grok parse, custom VRL)
• Add processors incrementally, not in large batches; observe the impact of each addition

Deployment:

• Always deploy at least 3 replicas for high availability
• Do not set CPU limits on OPW pods
• Use compute-optimized, non-burstable instance types
• Use L4 network load balancers, not L7 application load balancers
• For Kubernetes: enable PodDisruptionBudget, pod anti-affinity, and topology spread constraints
• Allowlist required Datadog domains for outbound HTTPS (port 443) if your network restricts egress

Autoscaling:

• Use CPU-based HPA at 70% target for standard pipelines
• Use KEDA with source_buffer_utilization_mean for SDS-heavy pipelines (20+ rules)
• Configure aggressive scale-up (react in 1-2 minutes) and conservative scale-down (15-minute stabilization)

Buffering:

• Use disk buffers when data durability during destination outages matters
• Set when_full: drop_newest on non-critical destination buffers to prevent multi-destination blocking
• Set terminationGracePeriodSeconds to cover your maximum buffer drain time

Monitoring:

• Alert on component_discarded_events_total{intentional:false} (critical: any non-zero value)
• Alert on source_buffer_utilization_mean (warning at 70% of capacity, critical at 90%); this metric reports raw event counts (max = vCPU x 1,000), not a 0-1 ratio. Use a formula monitor (utilization_mean / max_size_events) for pod-size-independent thresholds.
• Monitor SDS utilization; sustained values above 0.9 indicate SDS is the bottleneck
• Validate throughput against sizing calculations using component_received_bytes_total{component_kind:source}

How we tested

Environment

• Platform: AWS EKS, Kubernetes
• Instance type: c7a.2xlarge (AMD EPYC Genoa, 8 vCPU, 16 GB)
• Pod configuration: StatefulSet, requests: {cpu: 1, memory: 2Gi}, no CPU limit
• Test duration: 30-minute steady-state windows per configuration

Workload

The test workload consisted of seven event types designed to represent a heterogeneous production log pipeline:

Weighted average event size: approximately 2,127 bytes. SDS match rate: 22.5% of events carried Luhn-valid credit card numbers, ensuring SDS performed actual match-and-redact operations, not scan-only.

Methodology

Each test pushed a single OPW pod to approximately 100% CPU utilization to measure maximum throughput per vCPU. The generator ran a configurable number of replicas and worker threads, increasing load until the OPW pod reached saturation. Throughput was measured using OPW’s self-reported pipelines.component_received_bytes_total{component_kind:source}, and CPU was measured using pipelines.cpu_usage_seconds_total (both as rates).

The TB/day/vCPU metric was calculated as:

(avg_bytes_per_second x 86,400 / 1,000,000,000,000) / avg_cpu_vCPU

All tests confirmed zero errors (pipelines.component_errors_total), zero unintentional discards (pipelines.component_discarded_events_total{intentional:false}), and zero CPU throttling (container.cpu.throttled).

Production considerations

These benchmarks represent maximum throughput under controlled conditions. Your production throughput will vary based on:

• Event size and structure (field count, nesting depth, value sizes)
• Specific processors in your pipeline and their configuration
• SDS rule count, rule complexity, and scan scope
• Destination latency and availability
• Network conditions and load balancer configuration

Use these numbers as planning baselines, not guarantees. We recommend starting with the deployment sizing table, monitoring your pipeline’s actual performance in production, and adjusting based on observed metrics.

Need help designing your OP deployment?

Every Observability Pipelines deployment is different. Data volumes, pipeline complexity, SDS requirements, infrastructure topology, and availability goals can all affect the architecture and capacity you need. If you’d like help translating these benchmarks into a production-ready design for your environment, Datadog Services and Enablement can work with your team to plan, architect, and implement a deployment tailored to your requirements.

Authors

Chris Kelner - Senior Product Solutions Architect

References

• Buffering and backpressure
• Best practices for scaling Observability Pipelines
• Pipeline usage metrics
• Network traffic configuration
• VM deployment architecture
• Kubernetes deployment architecture