The volume of logs that organizations collect from all over their systems is growing exponentially. Sources range from distributed infrastructure to data pipelines and APIs, and different types of logs demand different treatment. As a result, logs have become increasingly difficult to manage. Organizations must reconcile conflicting needs for long-term retention, rapid access, and cost-effective storage. Meanwhile, existing solutions tend to create their own hurdles by requiring you to maintain schemas, write complex queries, and perform time-consuming targeted re-indexing operations. All of this can lead to gaps in visibility and put key data out of reach when it is most urgently needed.
To address these problems, we’re pleased to introduce Flex Logs. With Flex Logs, Datadog now offers a solution for centralizing all of your logs within one platform, no matter the use case. Building on the flexibility offered by Logging Without Limits™, which decouples log ingest from storage—enabling Datadog customers to enrich, parse, and archive 100% of their logs while storing only what they choose to—Flex Logs decouples the costs of log storage from the costs of querying. It provides both short- and long-term log retention for a nominal monthly fee without sacrificing visibility, enabling streamlined correlation between all of your logs, metrics, and traces.
In this post, we’ll show you how Flex Logs enables you to:
- Consolidate all of your logs—no matter the use case—in one platform
- Rein in costs without sacrificing visibility
Consolidate all of your logs—no matter the use case—in one platform
The multifaceted utility of logs means that there is no one-size-fits-all solution to managing them. Organizations often face a dilemma: to index or to archive. Indexing makes your data rapidly searchable, so it’s essential for logs you query frequently and use for real-time monitoring and alerting, such as application logs. But indexing can be resource-intensive and costly, so indexing all of your logs is impractical at scale. Archiving tends to be a better solution for logs you need to keep long-term, for compliance and other purposes, but which you query infrequently and without much urgency. If and when you do need to search them, these types of logs can be retrieved using Datadog Log Rehydration™, for example.
But in many cases, neither indexing nor archiving is a perfect solution. Logs of network activity, security events, and business transactions, for example, are often generated in enormous volumes. Compared to application logs, these types of logs are seldom queried—often only months after they are generated. At that point, however, querying them is often a matter of urgency (such as in cases of potential security breaches or fraudulent activity).
By decoupling storage from compute costs, Flex Logs provides a solution for long-term retention of logs without impeding query performance. With Datadog Log Management, you can now:
- Use Standard Indexing for logs you tend to query frequently within a short time span (i.e., a few days), such as application logs
- Use Archiving for logs you need to retain in the long term (i.e., years), but which you query infrequently and without much urgency, such as audit and configuration logs
- Use Flex for logs you collect in high volumes, retain in the relatively long term (i.e., months or years), and sometimes urgently need to query, such as security, transaction, and network logs
Take, for example, a financial services organization. Visibility into the services this organization uses to execute millions of trades every day is indispensable. Engineering teams need the ability to rapidly query and alert on logs of those trades, generally for at least a few days at a time. Meanwhile, you also need to retain those logs longer-term—say, for a few months at a time—in case you need to investigate fraudulent activity. With Datadog Log Management, you can simply enable the Flex Tier within the configuration of any index to which you already route these logs, choosing retention periods that make sense for you in both the short and long term. As a result, logs of the trades would be kept “hot” for a few days for use in real-time investigations and alerting, and “warm” for any number of months after the fact in order to quickly facilitate ad hoc investigations.
Or, consider a video streaming platform that generates terabytes of network logs every day. Despite the volume of those logs, network analysts might need to query them only a few times a day. By enabling the Flex Tier on the index for the CDN logs (with retention set to the three-month minimum), you could make them instantly queryable without incurring the typical costs of indexing such massive volumes of data.
Rein in costs without sacrificing visibility
Flex Logs provides log retention at a commodity storage price point—$0.05 per million events per month—without impeding querying. It allows you to allocate a fixed level of compute capacity to individual teams within your organization according to their needs at a fixed monthly rate. With Flex Tier storage, you can retain logs for three, six, or fifteen months.
Flex Logs enables you to rapidly search any and all of your logs in one place without ballooning storage costs. The Log Explorer allows you to toggle between searching only standard-tier, fully indexed logs (e.g., frequently queried data such as application logs) or expanding your search to include Flex Logs. As a result, it allows you to avoid the logistical overhead of maintaining additional tools, writing complex query statements, or performing targeted re-indexing operations. It also allows for smooth correlation with your other data in Datadog, including metrics, traces, and more.
Maximize the value of your logs
Flex Logs provides organizations with a log management solution whose costs won’t multiply when their storage volumes do. It enables comprehensive observability at scale, giving organizations granular control as they manage fast-growing volumes of logs, while providing individual teams with quick, consistent access to the data they need.
Flex Logs is in Limited Availability. Datadog Log Management users can sign up with this form to get started. If you’re brand new to Datadog, you can sign up for a 14-day free trial.
