The volume of logs generated from modern environments can overwhelm teams, making it difficult to manage, process, and derive measurable value from them. As organizations seek to manage this influx of data with log management systems, SIEM providers, or storage solutions, they can inadvertently become locked into vendor ecosystems, face substantial network costs and processing fees, and run the risk of sensitive data leakage. Navigating the complex tradeoffs between flexibility, costs, and control requires organizations to test new tools and implement innovative workflows that prioritize their log volume efficiently, protect their data, and avoid vendor lock-in.
We are pleased to announce the latest version of Datadog Observability Pipelines. With new out-of-the-box templates and granular configuration options, Datadog Observability Pipelines enables organizations to quickly and easily take control of the volume and flow of their log data so that they can experiment with new tools and vendors, filter and route the most relevant logs, ensure data compliance, and more. Most importantly, Observability Pipelines helps you make value-based decisions on your logs before they leave your environment, striking a balance to maximize control, minimize operational costs, and maintain visibility.
In this post, we’ll discuss how Observability Pipelines allows you to:
- Design pipelines quickly with preconfigured templates
- Customize log processing with granular controls
- Monitor the health of pipeline components to optimize efficiency
Design pipelines quickly with preconfigured templates
Datadog uses the Observability Pipelines Worker, a software running in your infrastructure, to aggregate, process, and route logs. Each Observability Pipelines Worker instance operates independently, so you can scale quickly and easily with a simple load balancer. The Observability Pipelines UI acts as a centralized control plane where you can design and manage pipelines. Any changes that you implement with the centralized control plane can be deployed to the Observability Pipelines Worker with a single click.
To get you started quickly and easily, Observability Pipelines contains preconfigured templates that enable you to design pipelines for your most integral log use cases, including log volume control, dual shipping logs, archiving logs to Amazon S3, splitting logs, and sensitive data redaction.
These templates simplify the initial setup of your pipeline while allowing you to configure and fine-tune settings to fit your organization’s specific needs. For example, when you use the Log Volume Control template, you can enforce custom quotas that limit the amount of log volume sent to a specific destination. You can also edit log events to reduce log size—for example, by dropping certain fields—before they reach their destination. The Split Logs template allows you to transform and send logs to different destinations as you see fit, like sending system logs to one destination and networking logs to another.
Or let’s say you want to evaluate whether Datadog is the right fit for your log management needs. With the Dual Ship Logs template, Observability Pipelines will collect and process your logs and then send them to both Datadog and your current log management solution. Dual shipping allows you to evaluate the Datadog platform without impeding your existing workflows.
If you decide to migrate, the Archive Logs to S3 template helps you retain historical context by routing your logs to an Amazon S3 bucket alongside your current vendor. This ensures that your historical data is preserved for future analysis. And the Sensitive Data Redaction template can automatically redact sensitive data, like PII, before sending log data to its destination.
Customize log processing with granular controls
Datadog Observability Pipelines provides several different processors to transform your logs in multiple ways before sending them to their destinations. These processors include filtering, editing, sampling, quotas, and deduplication.
- Filtering logs enables you to isolate logs based on specific criteria, allowing you to process a narrow scope of logs or send different logs to multiple vendors. For example, send logs relevant to your DevOps team to Datadog and security logs to your SIEM provider.
- Editing gives you the flexibility to add, remove, or remap log fields. Editing enables you to transform log data for governance or based on the use case.
- Sampling logs allows you to exclude a percentage of noisy logs from being sent to a destination. You can create rules to identify specific logs and then define what percentage of them to retain.
- With quotas, you can enforce thresholds that help you control both volume and cost, allowing you to protect your downstream destinations from unexpected spikes.
- Deduplicating your logs ensures that identical logs are not processed or routed to your destinations unnecessarily, helping you reduce noise wherever your logs are sent.
By using the Observability Pipelines processors, you’ll gain control of your logs at a granular level. You can define how they are treated in your environment from source to destination, ensuring that you get the most value out of them.
Monitor pipeline components to optimize efficiency
Once you’ve created a pipeline, Datadog will display key metrics, including the pipeline’s CPU and memory utilization, event and byte throughput at both the source and destinations, and error rate. Use our out-of-the-box recommended monitors or create your own in order to get notified of spikes in dropped data, increasing error rates, high memory usage, and more. Observability Pipelines will automatically recommend monitors based on your pipeline components, processors, and activity. Recommended monitors can be enabled on the Observability Pipelines landing page or inside an individual pipeline’s Overview page.
Monitoring pipeline health enhances your ability to detect bottlenecks and ensure that logs are successfully transmitted through the pipeline without data loss or corruption.
Gain flexibility and control over your data with Observability Pipelines
Datadog Observability Pipelines makes it easy to control the volume of your logs and retain the ability to experiment with new tools and vendors while saving costs, minimizing disruption, and prioritizing compliance. For more information, visit our documentation. Or, if you’re not yet a Datadog customer, you can sign up for a 14-day free trial.