Add More Context to Your Logs With Reference Tables | Datadog

Add more context to your logs with Reference Tables

Author Mallory Mooney
Author Andrew Connelly

Published: October 19, 2020

Logs provide valuable information for troubleshooting application performance issues. But as your application scales and generates more logs, sifting through them becomes more difficult. Your logs may not provide enough context or human-readable data for understanding and resolving an issue, or you may need more information to help you interpret the IDs or error codes that application services log by default.

Datadog’s Reference Tables enable you to enrich logs with your own business-critical data, automatically providing more contextual information for quickly resolving application issues. Each table includes a primary key—an ID or code that appears as a field in your logs, such as an organization ID or a status code—and additional business data associated with that key. For example, the Reference Table below uses the merchant ID found in application logs as the primary key and maps each ID to a specific merchant name and point of contact.

Add context to your logs
Create a new Reference Table to map customer and merchant data to IDs in your logs

To create a new table in Datadog, you can upload a comma-separated value file (CSV) and select which column you want to use as the primary key. You can then use the table with a Lookup Processor to enrich your logs, as seen in the example processor below.

Use the Lookup Processor to apply Reference Tables to your logs

The Lookup Processor uses the Merchant_Details Reference Table to map each Merchant ID (i.e., the primary key) found in the shopist.webstore.merchant.display_id log attribute to a new merch_detail attribute. With this configuration, Datadog will automatically add merchant names and points of contact to incoming logs as new attributes, which you can use as facets to search and analyze your logs as well as build dashboards to get a better picture of log activity.

In this guide, we’ll show you how you can use Reference Tables to:

Connect application service logs to specific units

Many organizations group their application services by hierarchy (e.g., team, department, cost center). This allows stakeholders to focus on monitoring and managing specific parts of the application. You can use Reference Tables to automatically map service logs to specific groups within your organization and use that data to build dashboards for better visibility into critical, revenue-generating applications. For example, you can use a dashboard to monitor transactions broken down by application service or strategic business unit (SBU).

Create dashboards to monitor your critical data sets

You can also create a table to automatically connect business data with IDs in your service logs, such as merchant IDs, providing more context for faster troubleshooting. For example, you can quickly isolate transaction issues to a specific merchant portal or customer account. The example error log below provides contact information for a merchant portal that failed to process a transaction.

Troubleshoot logs with Reference Tables

Troubleshoot more efficiently with error code mapping

When an application generates an error, it logs a code to reflect the severity or type of error. But each application service may generate its own custom error codes, making it more difficult to pinpoint the issues that generated the errors. Teams need the ability to quickly assess an error without memorizing hundreds of codes. You can use Reference Tables to map error codes to descriptive error messages, making them more actionable for your team and key stakeholders who need better visibility into the state of their services. The example table below automatically maps standard Linux error codes to the appropriate error name and message.

Use Reference Tables to create custom tables for capturing issues

You can also add custom codes and error messages for your application services. Once you connect the table with a log pipeline, Datadog will automatically add the error name and message to all incoming logs that match the pipeline’s filter. Once your logs are enriched with descriptive error messages, you can spend more time investigating an issue with an application service, instead of researching error codes.

Keep your tables up to date

With Reference Tables, you can bring operational and transactional data together to streamline workflows. To manage larger datasets, or reference data that changes over time, you can link your tables to cloud buckets such as Amazon S3 to automatically keep them up to date—support for Azure Storage and Google Cloud Storage is coming soon. This method supports tables of up to 200MB and ensures that they always have the latest data any time the underlying CSV file is modified.

Create an Reference Table from a cloud bucket

Contact support to sign up for the beta and start using Reference Tables today, or check out our documentation to learn more. If you don’t already have a Datadog account, you can sign up for a today.