Store and search logs at petabyte scale in your own infrastructure with Datadog CloudPrem

As AI workloads and cloud-native applications expand, organizations are generating more log data than ever. Each service, container, and model inference produces continuous telemetry that must be stored, secured, and analyzed. As telemetry grows more complex, teams must balance full visibility with new retention and residency needs. Security teams often retain logs for a year or longer to investigate threats and meet stricter compliance requirements, while new regulations increasingly require sensitive data to stay within regional boundaries.

To manage these pressures, many teams turn to a mix of self-hosted log management solutions that are either custom-built or provided by specialized vendors. But this approach comes at the expense of manageability and visibility: Scaling is slow and costly, maintenance is challenging, and data is left fragmented across tools. Additionally, teams who host logs in their own environment normally lose the advantages of a SaaS platform—such as ease of infrastructure maintenance, ease of use, correlation across all telemetry data, AI-powered analysis, and centralized governance control.

Datadog CloudPrem removes the traditional drawbacks of hosting logs in your environment. A hybrid log management solution that runs in your own infrastructure, CloudPrem remains fully integrated with the Datadog SaaS platform. It gives you full control over your data, correlation across all telemetry, AI-powered agents and insights, and high performance based on the Quickwit engine. With CloudPrem, you can store and search logs at petabyte scale in any region while keeping the full Datadog experience that your teams already know.

In this post, we'll show how CloudPrem helps you:

• Easily scale a modern, cost-efficient infrastructure for self-hosted logs
• Centrally manage log storage
• Troubleshoot faster with unified correlation and AI assistance
• Cost-effectively normalize, consolidate, and store high-volume security logs
• Meet data residency and compliance requirements without sacrificing visibility

Easily scale a modern, cost-efficient infrastructure for logs

Self-hosted log management solutions are typically difficult to scale. Adding new nodes often means redistributing large amounts of data across disks, a time-consuming process that impacts performance. Managing local or network-attached storage adds even more overhead, requiring constant tuning and rebalancing to maintain stability.

Efficient scaling

CloudPrem removes this complexity with a modern architecture that is built to scale easily, flexibly, and cost-effectively. It separates compute from storage, using low-cost object storage as the primary, durable storage layer. This object storage can be located either in your own on-premises cluster or in a cloud storage solution such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. Because compute is truly decoupled from storage in CloudPrem, you can add or remove search nodes in seconds to match demand, scaling up during spikes and down to save costs, all without rebalancing data. Combined with advanced compression algorithms, this scalable architecture significantly reduces infrastructure and operational costs compared to legacy self-hosted systems.

High performance

The architecture is also designed to search data directly from object storage with high performance. As logs are ingested, indexers write optimized index pieces called splits directly to your object storage, while a central metastore tracks those splits and makes them available for querying almost instantly. Another advantage is that the search layer is entirely stateless. When you run a query, a coordinator node consults the metastore to locate relevant splits and distributes the request across peer search nodes. Each node fetches data directly from object storage in parallel, without maintaining local index data—making the clusters both resilient and easy to operate.

Centrally manage log storage

Managing multiple clusters is a significant challenge, especially across regions and cloud providers. CloudPrem solves this issue by bringing management into one place. From the Datadog UI, you can monitor cluster health and view ingestion rates, all within the same interface you already use for metrics and dashboards. Health monitoring shows where performance is dipping, so you can act before issues escalate.

You can also centrally manage multiple indexes for fine-grained data segmentation. With this capability, you can easily separate log data by value, application, or type—such as audit or security logs—and assign specific retention and quota policies to each index.

Finally, CloudPrem works together with Datadog Observability Pipelines in enabling you to store and manage logs from many different sources, all in one place. Observability Pipelines provides out-of-the-box integration with the tools, standards, and destinations you already use for logs—including open log formats and protocols, third-party observability platforms and SIEMs, and data lakes. This broad compatibility with both vendor-specific and vendor-neutral standards lets you process, enrich, and normalize logs of all types and then route them into CloudPrem for unified search and analysis.

Troubleshoot faster with unified correlation and AI assistance

As daily log volumes increase—coming from different teams and systems and in various formats—troubleshooting becomes increasingly difficult. The challenge becomes worse when engineers have to jump between multiple tools, each with its own UI, dashboards, and alerting workflows to understand issues.

With Datadog CloudPrem, teams can search and correlate their self-hosted logs in one central interface while still accessing the full breadth of Datadog's SaaS platform capabilities. You can investigate logs, metrics, and traces side by side without switching tools or losing context.

Let's say you're a developer overseeing a checkout API that suddenly starts timing out. You receive a change throughput alert, and Bits AI SRE immediately steps in as your co-pilot. The agent generates several hypotheses and begins reasoning over all available signals—automatically correlating logs stored in CloudPrem with metrics and traces collected in Datadog SaaS.

Following one of Bits' investigations, you ask it to show you recent web-store error logs. At the same time, you notice another hypothesis focused on authentication issues and decide to dig into those as well. Within Log Explorer, you use Natural Language Queries (NLQ) to type: “Show error logs for the auth service over the past 30 minutes.”

Datadog automatically translates this request into a structured query—understanding your context, services, tags, and time ranges—and correlates relevant traces and metrics. In the following screenshot, you can see how CloudPrem logs are correlated with APM traces in the same window, facilitating your troubleshooting investigations just like when your logs are stored in Datadog SaaS.

CloudPrem is also MCP-ready, supporting the Model Context Protocol for AI and LLM interoperability. This enables customers to use their own agents—such as those built on OpenAI, Anthropic, or Gemini frameworks—to discover datasets, generate structured queries, and retrieve insights directly from CloudPrem. With MCP, AI systems can reason natively over your observability data, enabling intelligent analytics and faster decision-making without any custom integration.

By correlating your self-hosted logs with metrics and traces in a single view, you can now investigate issues faster, reduce context switching, and make decisions with full visibility across your system.

Consolidate and preprocess security logs at scale in your infrastructure

Security teams are especially familiar with the pains of log growth. Every firewall connection, CDN request, and VPC flow produces chatty, high-volume data that's too important to discard and too expensive to store in a traditional SIEM. To control costs, many teams stitch together a SIEM for active alerts, a cheap storage tier for long-term retention, and custom scripts to bridge the gap. The result is fragmented data, high operational overhead, and visibility gaps that block investigations.

CloudPrem consolidates these security workloads into one place. You can retain petabytes of security telemetry from Akamai, Palo Alto Firewall, Zscaler, and more without worrying about SIEM ingestion limits or vendor pricing tiers.

For teams that want to preprocess or enrich data before indexing, logs can also flow through Observability Pipelines. Pipelines make it easy to normalize logs with OCSF; enrich logs with GeoIP, threat-intelligence feeds, and contextual metadata; or apply filters to control cost. Out-of-the-box Packs extend these capabilities with prebuilt rules for popular security sources, helping you reduce noisy traffic and standardize enrichment policies across environments. This helps ensure that incoming security data is structured and enriched with the context needed to spot patterns quickly.

Once ingested, CloudPrem automatically organizes this data for investigation. Using AI-based parsing, Datadog detects common security log formats and extracts key fields, while Datadog's correlation engine links related events across metrics, traces, and logs. Because it shares the same indexing and pipeline framework as Datadog SaaS, security teams get scalable retention and unified visibility without losing the Datadog experience they rely on.

Meet data residency and compliance requirements without sacrificing visibility

For many organizations, data location is dictated by policy or regulation. Countries such as India, Singapore, Brazil, and China are introducing stricter data-residency laws, requiring organizations to keep sensitive data within national or regional boundaries. Whether you're processing financial transactions in the European Union, managing patient data in healthcare, or securing government systems, some logs simply can't leave your network.

CloudPrem helps you meet these data residency, privacy, and security requirements without giving up unified observability and security. Your logs stay where they must—but everything else, from correlation to AI-assisted analysis, works exactly as it normally does in Datadog.

CloudPrem also includes the same enterprise governance features that power Datadog's SaaS platform:

• Role-Based Access Control (RBAC): Fine-grained permissions for sensitive data.
• Audit Trail: Track configuration changes and user activity across teams.
• Sensitive Data Scanner: Automatically detect and redact personally identifiable information (PII) before indexing.
• Flexible ingestion: Route logs using the Datadog Agent, Observability Pipelines, Cribl, or Vector.

Get started with Datadog CloudPrem

CloudPrem enables you to retain control of logs in your own infrastructure while still providing full access to Datadog platform capabilities—combining scalability, ease of use, compliance, and AI-powered analysis in one consistent, integrated experience. With CloudPrem, you can manage petabytes of logs efficiently, support regional data-residency requirements, and maintain full visibility across hybrid environments.

CloudPrem is now available in preview. To learn more, visit the CloudPrem documentation or reach out to your Datadog representative.

If you're new to Datadog, sign up for a 14-day free trial.