Add More Context to Your Logs With Enrichment Tables | Datadog

Add more context to your logs with Enrichment Tables

Author Mallory Mooney
Author Andrew Connelly

Published: October 19, 2020

Logs provide valuable information for troubleshooting application performance issues. But as your application scales and generates more logs, sifting through them becomes more difficult. Your logs may not provide enough context or human-readable data for understanding and resolving an issue, or you may need more information to help you interpret the IDs or error codes that application services log by default.

Datadog’s Enrichment Tables enable you to enrich logs with your own business-critical data, automatically providing more contextual information for quickly resolving application issues. Each table includes a primary key—an ID or code that appears as a field in your logs, such as an organization ID or a status code—and additional business data associated with that key. For example, the Enrichment Table below uses the merchant ID found in application logs as the primary key and maps each ID to a specific merchant name and point of contact.

Add context to your logs
Create a new Enrichment Table to map customer and merchant data to IDs in your logs

To create a new table in Datadog, you can upload a comma-separated value file (CSV) and select which column you want to use as the primary key. You can then use the table with a Lookup Processor to enrich your logs, as seen in the example processor below.

Use the Lookup Processor to apply Enrichment Tables to your logs

The Lookup Processor uses the Merchant_Details Enrichment Table to map each Merchant ID (i.e., the primary key) found in the shopist.webstore.merchant.display_id log attribute to a new merch_detail attribute. With this configuration, Datadog will automatically add merchant names and points of contact to incoming logs as new attributes, which you can use as facets to search and analyze your logs as well as build dashboards to get a better picture of log activity.

In this guide, we’ll show you how you can use Enrichment Tables to:

Connect application service logs to specific units

Many organizations group their application services by hierarchy (e.g., team, department, cost center). This allows stakeholders to focus on monitoring and managing specific parts of the application. You can use Enrichment Tables to automatically map service logs to specific groups within your organization and use that data to build dashboards for better visibility into critical, revenue-generating applications. For example, you can use a dashboard to monitor transactions broken down by application service or strategic business unit (SBU).

Create dashboards to monitor your critical data sets

You can also create a table to automatically connect business data with IDs in your service logs, such as merchant IDs, providing more context for faster troubleshooting. For example, you can quickly isolate transaction issues to a specific merchant portal or customer account. The example error log below provides contact information for a merchant portal that failed to process a transaction.

Troubleshoot logs with Enrichment Tables

Troubleshoot more efficiently with error code mapping

When an application generates an error, it logs a code to reflect the severity or type of error. But each application service may generate its own custom error codes, making it more difficult to pinpoint the issues that generated the errors. Teams need the ability to quickly assess an error without memorizing hundreds of codes. You can use Enrichment Tables to map error codes to descriptive error messages, making them more actionable for your team and key stakeholders who need better visibility into the state of their services. The example table below automatically maps standard Linux error codes to the appropriate error name and message.

Use Enrichment Tables to create custom tables for capturing issues

You can also add custom codes and error messages for your application services. Once you connect the table with a log pipeline, Datadog will automatically add the error name and message to all incoming logs that match the pipeline’s filter. Once your logs are enriched with descriptive error messages, you can spend more time investigating an issue with an application service, instead of researching error codes.

Enrich your logs, with more functionality coming soon

With Enrichment Tables, you can bring operational and transactional data together to streamline workflows and efficiently pinpoint the root cause of an application issue. And you’ll soon be able to automatically update your tables by linking to cloud buckets such as Amazon S3, Azure Storage, and Google Cloud Storage. This will allow for larger datasets that automatically update any time the underlying CSV file is modified.

Create an enrichment table from a cloud bucket

Contact support to sign up for the beta and start using Enrichment Tables today, or check out our documentation to learn more. If you don’t already have a Datadog account, you can sign up for a today.