Route Logs to Third-Party Systems With Datadog Log Forwarding | Datadog

Route logs to third-party systems with Datadog Log Forwarding

Author Addie Beach
Author Avi Verma

Published: October 13, 2022

Large organizations often rely on multiple monitoring tools, security platforms, and auditing systems to meet the diverse needs of their observability, security, engineering, and compliance teams. Because these teams may use the same logs for many different use cases—including detecting potential threats or breaches, troubleshooting errors, and gauging the effectiveness of new features—it can be difficult to effectively standardize and route data. Additionally, organizations can struggle with tool sprawl in the absence of a strong central observability team, particularly as they acquire new teams or migrate to the cloud.

Datadog Log Pipelines offers a fully managed, centralized hub for your logs that is easy to set up. You can ingest logs from your entire stack, parse and enrich them with contextual information, add tags for usage attribution, generate metrics, and quickly identify log anomalies. You can also use Sensitive Data Scanner, standard attributes, and granular tagging to enforce organization-wide conventions and industry regulations. Distributing these processed logs to users across your enterprise can be a challenge, however, as some teams may prefer to use platforms or environments outside Datadog to accommodate certain workflows.

We are excited to announce that Log Pipelines now supports Log Forwarding, allowing you to send your logs from Datadog to Splunk, Elasticsearch, and HTTP endpoints. By using in-depth filtering options and dual shipping capabilities, you can provide standardized logs to your teams and easily manage a wide variety of logging use cases. With Log Forwarding, you can:

The Log Forwarding overview page in Datadog, showing Splunk, Elasticsearch, and HTTP endpoints.

Centralize log processing while accommodating flexible workflows

Datadog Log Pipelines allows you to ingest and transform your logs with features like grok parsing, remapping, and string extraction. Using Log Forwarding, you can take logs processed in Datadog pipelines and easily adapt them to the tools that work best for individual teams. This helps you centrally manage log processing while still providing enough autonomy to your teams that they can efficiently analyze logs according to their specific requirements.

You can create custom destinations for external forwarding, then choose which logs you forward and how you forward them to fit your best practices. Log Forwarding enables you to filter logs on an as-needed basis so that your teams receive only the data most relevant to them, which reduces the number of logs they need to store, cuts down on unnecessary noise, and helps prevent potential data leaks. In addition to simple configuration and integration with your teams’ HTTP, Splunk, or Elasticsearch endpoints, Log Forwarding also provides you with RBAC settings to manage who can create, edit, or remove these destinations.

The configuration window for a custom third-party destination, including RBAC authentication settings.

Let’s say your enterprise uses logs to monitor user authentication activity for troubleshooting system issues and detecting suspicious behavior. Your application and central observability teams have already adopted Datadog, but your security teams still follow established Security Information and Event Management (SIEM) workflows on a different observability tool. Using Log Forwarding, your central observability team can collect and process your logs in Log Pipelines, then easily forward the necessary logs to your security team’s external endpoint. This allows all of your enterprise users to work on their preferred platforms—your application teams can analyze their logs in Datadog using features like Application Performance Monitoring, and your security teams can still manage security threats using the application of their choice.

You can also use Log Forwarding to help your teams transition to new platforms. As you adopt Datadog Log Management, there may be certain teams that need to continue using existing solutions for contractual or business continuity reasons. You can easily ingest your logs in Datadog for centralized processing and parsing, then use Log Forwarding to route the logs to existing vendors. Each team can now migrate their workflows on their own schedule while still accessing standardized logs. Additionally, this dual shipping helps you ensure that downstream dependencies, such as data warehouses, continue to function as expected.

Duplicate logs across local offices, geographic regions, or external organizations

Using Log Forwarding, you can send your logs to dedicated storage locations anywhere in the world. Geographically distributed organizations with many satellite offices—including government agencies, financial institutions, or insurance companies—often have to maintain copies of their logs for compliance or regulatory reasons. For example, some laws or standards stipulate that organizations must store their logs in a neutral location for a specified period of time or maintain local backups via their own file servers. Log Forwarding can help your teams access a consistent view of your logs no matter where they are located. This centralizes log pipelines, streamlines collaboration, and reduces mistakes that can result from users having outdated or inconsistent logs.

Shipping logs between locations also helps you facilitate projects with external partners or consultants. By providing an easy-to-use interface for building these integrations, Log Forwarding frees your teams from having to create custom tools for importing and exporting data. This allows them to focus on project tasks and deliver faster turnaround times.

Send customized logs to customized destinations

With Datadog Log Forwarding, you get the best of both worlds—you can centrally process your logs according to industry regulations and your organization’s best practices while still giving your teams the autonomy they need to work effectively with their preferred tools. This flexibility enables you to easily support co-existence with other vendors, achieve compliance with industry regulations, ensure the accuracy of local backups, and streamline internal and external collaboration.

Log Forwarding is currently in Limited Availability. If you’re an existing customer, you can request access to Log Forwarding and then try it out with our documentation. Or, you can start using Datadog with a 14-day .