Observability Pipelines Sensitive Data Scanner optimization

September 9, 2026

Introduction

Sensitive Data Scanner (SDS) is the single most impactful processor on pipeline throughput. Understanding how it behaves and how to configure it efficiently can mean the difference between 0.79 TB and 4.7 TB per vCPU per day.

This is companion guide to Observability Pipelines: a guide to sizing, scaling, and performance which also covers the impact of the SDS processor. It is recommended to read both to understand the full scope of SDS, its impact, and how to optimize it.

How SDS affects throughput

SDS evaluates regex rules against every scannable field of every event it processes. The cost scales with three factors:

1. Number of rules: Each additional 25 rules reduces maximum throughput by approximately 30-40%. This is the dominant cost driver.
2. Number of fields per event: Cost scales linearly with field count. Events with 100 fields take roughly 3x longer to scan than events with 30 fields.
3. Field value size: Cost scales logarithmically with the byte length of field values. Doubling the field size does not double the scan time.

The following factors have no measurable effect on scan time: redaction method, match action type, key name size, and nesting depth. The scanner does not examine key names, only field values.

Scope SDS to the services that need it

The highest-leverage optimization is restricting which events SDS scans. In most organizations, only a fraction of services handle sensitive data: payment processing, user authentication, email, and similar services. Scanning every event from every service wastes CPU on events that will never contain PII.

The improvement grows with rule count because every additional rule multiplies the cost of scanning events that do not contain sensitive data.

In the OP pipeline, configure the SDS processor’s filter condition to target only relevant services:

service:payments OR service:customer-lookup

You can also limit which fields SDS scans. Instead of scanning every field in every event, target only the fields that could contain sensitive data: message, body, request_body, and specific custom attributes. This reduces the per-event regex work without changing your rule set.

Audit your rule set

Many organizations enable SDS rule categories broadly without evaluating whether each rule is necessary. The Secrets and Credentials category alone contains 111 rules. Enabling it in its entirety when your pipeline only handles application logs, not infrastructure secrets, is a significant and unnecessary throughput cost.

To identify which rules produce matches, check the match counts in the SDS Processor section of the OP pipeline overview dashboard (expand the processor and click on “view scanning rules”). Rules with zero or near-zero matches are potential candidates for removal.

The current OP pipeline UI shows SDS rule match counts, but the visible window is limited to recent data (24 hours). A 24-hour window is not sufficient to confidently identify a rule as unused, since some PII patterns (e.g., international identification numbers) may only appear in logs during specific business events, time zones, or seasonal patterns.

Recommended approach for auditing:

1. Query pipelines.sds_rule_matched_total over a 30-day rolling window grouped by rule name.
2. Rules with zero matches over 30 days are strong candidates for removal. Rules with fewer than 10 matches warrant manual review - they may be false positives.
3. Re-audit quarterly. Traffic patterns change as new services are deployed, new data sources are onboarded, and business operations evolve.

Alternatively if your logs are going to Datadog, you can search in the Logs Explorer by Fields grouped on Sensitive Data Category scoped to logs coming from your OP Pipeline (we suggest adding a field to your logs via OP to make them easily identifiable as to the OP pipeline they flowed through). However this method is less reliable than the metrics based approach due to shorter log retention periods, but it does allow for easy inspection of the logs themselves to look for false positives. Example:

To measure the latency impact of SDS on your pipeline, query pipelines.component_latency_seconds{component_type:sensitive_data_scanner} . This distribution metric shows per-event processing time through the SDS processor. You must enable percentiles for this metric in the Datadog Metrics Summary UI before it is available for graphing.

Throughput degradation at high rule counts

For organizations with more than 40 SDS rules, expect throughput to degrade at the following approximate rate (these multipliers approximate targeted scanning; blanket scanning decays roughly twice as fast):

Split large rule sets across multiple SDS processors

When running more than 20 SDS rules at 3 or more vCPU per pod, splitting the rules across multiple SDS processor instances with disjointed rule sets delivers significant throughput improvements.

Why splitting works

OPW processes events concurrently across multiple threads. However, a single SDS processor with many rules creates long per-event processing times that cause head-of-line blocking in OPW’s internal task queue. Events are dispatched as concurrent tasks, but the queue requires tasks to complete in order. When a task at the head of the queue takes a long time (because it is scanning 40+ regex rules), faster-completing tasks wait idle. Worker threads go unused, and CPU utilization drops even under full load.

Splitting rules across multiple smaller SDS processors reduces per-event scan time per processor, which shortens the queue head’s hold time and allows more threads to stay active.

To measure the impact of per-event scan time with these changes look at pipelines.component_latency_seconds{component_type:sensitive_data_scanner} - this distribution metric shows per-event processing time through each SDS processor instance. Compare p50/p95 before and after splitting to measure the reduction. You need to enable percentiles for this metric in Datadog Metrics Summary before it’s available for graphing.

For head of line blocking you can’t tell from CPU alone. You need two signals together:

1. The pipeline is saturated - meaning throughput has plateaued and backpressure is building. You see this from:

   1. pipelines.utilization{component_type:sensitive_data_scanner} near 1.0 (SDS is the bottleneck)
   2. pipelines.source_buffer_utilization_mean rising (events are queuing at the source)
   3. Adding more ingest load doesn’t increase throughput (throughput is flat)

2. But CPU is well below allocated - pipelines.cpu_usage_seconds_total as rate shows 2.82 when the pod has 4 vCPU allocated

That combination is the HOL blocking signature: the pipeline can’t process more data (saturated), but the CPU has idle capacity (threads are waiting, not working). If the bottleneck were purely CPU, you’d see CPU at ~4.0 of 4.0. The gap between “pipeline is full” and “CPU has headroom” is the waste from HOL blocking.

Before and after: Heavy Processing + SDS at 4 vCPU

** This test used targeted SDS (same as the 1.97 TB/day/vCPU [1 vCPU test] configuration above in section “Scope SDS to the services that need it”), not blanket scanning (all events resulting in 0.79 TB/day). The lower per-vCPU efficiency at 4 vCPU (1.26 [tested at 4 vCPU] vs 1.97 [tested at 1 vCPU]) is the head-of-line blocking penalty: at higher vCPU, a single SDS processor with 40 rules creates long-running tasks at the head of the queue that block other threads from progressing, reducing per-core efficiency. Splitting recovers part of that loss by shortening per-event queue hold times.*

The unsplit configuration left 1.18 vCPU idle due to head-of-line blocking. Splitting broke through that ceiling, consuming 3.65 of 4 allocated vCPU. The memory overhead of an additional SDS processor instance is minimal.

Splitting vs horizontal scaling

Both splitting and horizontal scaling address the same underlying constraint, but through different mechanisms. Splitting reduces per-event queue time within a single pod. Horizontal scaling eliminates the shared queue entirely by running independent pods.

Horizontal scaling delivers the best total throughput because each pod has its own independent runtime with no shared queue. However, splitting is valuable when you cannot easily add pods (node constraints, licensing, or preference for fewer larger pods).

At production scale, splitting and horizontal scaling combine well. In our testing, 4 pods x 4 vCPU with split 2x20 rules achieved 21.6 TB/day total (250 MB/s), with near-linear scaling across pods.

When splitting helps and when it does not

• At 1 vCPU per pod: Splitting has no effect. Head-of-line blocking requires multiple threads to manifest.
• At 2 vCPU with 40 rules (Heavy tier): Splitting delivers negligible improvement (+0.3%). At this low vCPU count with high rule count, the per-event scan time dominates regardless of split configuration.
• At 2 vCPU with 10 rules (Medium tier): Splitting delivers +16%.
• At 3-4 vCPU with 40+ rules: Splitting is strongly recommended (+45%).
• Under moderate load (70-80% CPU utilization): The benefit is smaller. Splitting is most impactful when the pipeline is operating near capacity.

Customer validation

A quote from a customer who confirmed this optimization in production: “We split our single SDS into 6 with less than 20 rules each. We got significant improvement on the max pipelines utilization, about 50%.”

Place SDS after volume reduction processors

SDS pays the full scanning cost for every event it processes, including events that a downstream filter, sample, throttle, quota, or dedupe processor would discard. If your pipeline includes both SDS and volume reduction processors, place the reduction processors before SDS whenever pipeline ordering permits. This reduces the number of events SDS must scan without affecting your data quality.

Case study: SDS saturation and autoscaler failure

CPU-based autoscaling can actively harm SDS-heavy pipelines. When SDS saturates, CPU drops because workers are blocked waiting for the scanner to drain, not because the pipeline has spare capacity. The autoscaler interprets falling CPU as surplus capacity and scales down, exactly when it should scale up.

A customer running 187 SDS rules across 400 OPW pods experienced this failure mode during a deployment-induced log spike. The HPA scaled from 400 pods down to 283 because CPU dropped during SDS saturation. Total throughput dropped 60%, source buffers filled, and upstream log shippers began queuing retries.

The customer’s own observation: “CPU usage was down since the pipeline was blocked on SDS which resulted in the HPA actually scaling down the pipeline during this time. Probably the opposite of what we would have wanted.”

What happened: SDS was already operating at 99% utilization (pipelines.utilization{component_type:sensitive_data_scanner}) in steady state. This metric measures how busy a component is between a value of 0, idle, and 1, fully saturated and causing backpressure. When the log spike arrived, SDS could not absorb the additional load. Workers became blocked waiting for SDS to drain, and CPU utilization dropped. The Kubernetes HPA interpreted the falling CPU as surplus capacity and scaled the deployment from 400 pods down to 283, exactly the opposite of what was needed. Total throughput dropped 60%, source buffers filled, and upstream log shippers began queuing and issuing retries.

Root cause: CPU utilization is a misleading autoscaling signal for SDS-heavy pipelines. Under SDS backpressure, CPU drops because workers are blocked, not because the pipeline has spare capacity.

Recommendations:

• Audit rule necessity: 250+ rules is significantly more than most organizations need. Disabling unused rules directly reduces per-event processing time.
• Filter which events SDS scans: if only 20% of services handle sensitive data, SDS throughput effectively increases 5x.
• Limit which fields SDS scans: target only fields that could contain sensitive data (message, body, request attributes) instead of scanning every field.
• Split rules across multiple SDS processors: with 250 rules in a single processor, head-of-line blocking is severe. Splitting into 10-12 processors of approximately 20 rules each breaks the queue contention ceiling.
• Replace CPU-based HPA with pipeline-aware scaling: use the Datadog Pod Autoscaler or KEDA to scale on pipelines.source_buffer_utilization_mean instead of CPU. Note: this metric reports raw event counts (max = vCPU x 1,000), not a 0-1 ratio. Set KEDA thresholds as absolute values scaled to your pod size.

Case study: SDS saturation during log spike

A different customer running OPW on VMs behind an AWS Auto Scaling Group experienced SDS saturation during a burst of TRACE-level logs.

Setup: 4 vCPU EC2 instances, 50+ SDS rules scanning all events, processing approximately 120 million TRACE logs per hour (33,333 logs per second) sent via Datadog Agents.

What happened: SDS utilization on all active hosts reached 1.0 (100% saturated) simultaneously during the burst. At 33,000 logs per second with 50+ regex rules per event per field, the pipeline could not keep up. Source buffers filled from a baseline of approximately 20,000 events to over 37 million events. In this case the VMs CPU reached 100% during the event. Over 2.2 million timeout events were recorded on the two primary hosts.

Recommendations:

• Add a filter processor to drop TRACE/DEBUG logs before they reach SDS. This eliminated approximately 33,000 events per second of SDS workload.
• Add per-service quotas ahead of the workspace quota to isolate rogue services that produce log spikes.
• Lower ASG CPU scale-up thresholds to 50-60% so scale-out begins before existing instances saturate.
• Lower DD_LOGS_CONFIG_BATCH_WAIT from 5s to 2-3s on the DD Agent DaemonSet to reduce burst amplitude.

Need help designing your SDS strategy?

Every Observability Pipelines deployment is different. SDS requirements can affect both architecture and capacity. If you’d like help translating these optimization recommendations into a production-ready design for your environment, Datadog Services and Enablement can work with your team to plan, architect, and implement a deployment tailored to your requirements.

Authors

Chris Kelner - Senior Product Solutions Architect

References

• SDS product page
• Discovering sensitive data with SDS
• Secrets scanning
• SDS predefined scanning rules library
• Observability Pipelines: a guide to sizing, scaling, and performance