Leo Cavaille
Over the past year, Datadog has completed an initiative to unify our Site Reliability Engineering (SRE) and Security teams into one cohesive group. This effort aimed to unify all aspects of our operational and security posture. In this post, we'll explore key lessons learned and real-world use cases, highlighting both successes and challenges. This article complements our DASH 2025 talk, "Security and SRE: Internal Trust & Core Observability - A Real Example of the Combined Approach at Datadog," presented by Tanner Prynn and Angela Beteta, where they dive deeper into these experiences and insights.
Why combine SRE and security?
Traditionally, security focuses on preventing external threats, while SRE's job is to ensure system reliability. Both ultimately share the goal of "keeping the company running," which makes their integration natural and mutually beneficial. Unifying these teams at Datadog significantly improved visibility across previously isolated functions and encouraged proactive risk management, aligning their shared objectives more effectively.
Tanner Prynn, Engineering Manager for Internal Trust, and Angela Beteta, Engineering Manager for Core Observability, led the teams that drove this transformation. The Internal Trust team is responsible for prioritizing the safety of customer data across the Datadog ecosystem, which includes strengthening the security of data access and propagation. The Core Observability team, chartered to maximize the value of using the Datadog platform within Datadog, also played a crucial role. Their focus on platform governance—identifying business needs, implementing guardrails to prevent misuse, and ensuring uniform policy application—was essential in balancing security with usability.
Although the rationale for combining these parts of the business was clear, putting this vision into practice required thoughtful preparation and structured planning. First, we began by acknowledging Datadog's own criticality to our operations as well as the importance of analyzing past incidents to pinpoint specific problems and vulnerabilities. Second, we collected detailed data on necessary security features and permissions, evaluating gaps and the frequency of sensitive actions. This step informed our decision on where to place effective guardrails. Finally, we wanted to establish a partnership between internal engineering and product teams, which would enable us to work together effectively towards common goals.
With alignment established, our next challenge was operationalizing governance of this combined approach in a way that was both effective and flexible.
Our iterative approach to governance
In order to develop efficient workflows for our combined organization, we iterated on our approach to monitoring, remediation, enforcement, and collaboration. We first implemented robust monitoring systems to detect risky behaviors in our environments early, taking advantage of Datadog Cloud SIEM and user feedback loops for context.
We prioritized a human-centric remediation strategy, emphasizing a blameless culture and collaborative solutions. We started by creating recommendations for resolving issues (instead of mandates), essentially providing Golden Paths as easy, encouraged solutions. Documenting exceptions, ensuring open communication, and continuously refining processes were key in this strategy. To prevent drift from our Golden Paths, we introduced monitoring, feedback loops, and targeted communications during multi-phase rollouts. Throughout this process, we partnered closely with Datadog product teams, which enabled us to roadmap and prototype features that supported not only our internal goals but our customers as well.
Let's look at how this iterative governance strategy translated into tangible improvements within Datadog.
How we implemented our combined approach
Iterating on processes critical to our combined SRE and Security organization enabled us to refine and scale permissions, improve audit logging, and enhance organization governance.
Building scalable permissions
Initially, organization-wide admin access made sense for Datadog's rapid growth. However, as the company scaled, these overly broad permissions caused frequent, accidental misconfigurations. We implemented several measures to address this issue. First, we implemented an auto-approved roles system to efficiently grant engineers necessary permissions, balancing security with minimal friction. The goal with this system was to build confidence in the organization's ability to grant necessary permissions without adding friction to an engineer's day-to-day workflows. This step made it easier to gradually reduce the number of admin-level users.
Next, we restricted critical permissions that could easily create risk in our environment. One example was the user-access-manage permission, which enables a user to directly change the permissions granted to themselves or others. Using Terraform, we were able to efficiently define roles and create the necessary approval workflows for assigning them to users. This step eliminated the risk of assigning sensitive permissions to too many users.
Finally, we documented this process for restricting permissions clearly, so teams could manage them autonomously. To accomplish this, we mapped permissions to specific API actions, then analyzed usage patterns via audit logs to identify the most frequent users. This enabled us to proactively reach out and confirm legitimate needs before revoking unnecessary permissions.
Improving existing audit logging capabilities
We began improving audit logging by standardizing retention periods and log formats, significantly enhancing our ability to quickly detect and prioritize critical events. We applied these new standards across audit log sets via code changes, pipelines, standard attributes, and reference tables.
To reduce the risk of misconfigurations, we automated log configuration changes through Infrastructure as Code (IaC) and required reviews on all pull requests (PRs). Once Terraform was safely controlling those sensitive configurations, we were able to restrict access in the Datadog platform via role-based access control (RBAC), which safeguarded log indexes, pipelines, and Sensitive Data Scanner from unnecessary changes.
These steps provided stronger guarantees for log integrity and retention, which significantly enhanced our auditing and detection capabilities.
Developing a comprehensive guide for organization governance
A key unanswered question in this process was whether our remediation efforts covered all relevant organizations. To address this gap, we developed a comprehensive internal organization typology detailing every Datadog organization's purpose and ownership. We then applied specific governance configurations across organizations, using Cloud SIEM to monitor for any deviations.
Keys to our success
The integration of SRE and Security teams at Datadog succeeded because we recognized shared opportunities clearly. We also fostered mutual understanding and agreement on how to address high-priority pain points throughout this transition. Lastly, we clearly defined roles and responsibilities for both individual and joint efforts, which was critical to our overall success. This unified strategy not only addressed existing reliability challenges but also laid a solid foundation for continuous improvements in security and reliability at Datadog. Since then, other Security and SRE teams across Datadog have adopted and evolved this combined approach to redefine how they work and collaborate.
Check out our documentation to learn more about Datadog's security and incident management offerings. If you don't already have a Datadog account, you can sign up for a free 14-day trial.
