Bianca Lankford
Defining success looks different for security organizations than it does for product, infrastructure, and other engineering teams. The latter group can often point to tangible outcomes, such as newly shipped features or performance improvements. Security orgs succeed when risks are lowered and the company’s posture improves over time, which are results that aren’t as easy to recognize but still valuable.
This gap in expectations becomes harder to ignore as the company grows. Both new technologies and shifting priorities, such as building edge environments and agentic AI systems, naturally expand a company’s attack surface. Without clear direction, a security org will struggle to support the larger organization effectively. When we combined Datadog’s SRE and security groups, we had to rethink what adapting to growth looks like. We learned that it comes down to three priorities:
- Creating reliable systems that scale with the company
- Ensuring that specialized teams stay connected and effective
- Preparing for risks and emerging technologies such as AI
A resilient security organization requires developing leaders and systems that can tackle these priorities, especially considering the fast pace of AI adoption and the rate at which cloud environments are growing more complex. When their leaders work together, security orgs can respond to new risks and adapt to industry trends without interfering with the goals of the larger organization.
Creating reliable systems that scale with the company
A primary focus for security organizations is building strong identity, data, and infrastructure guardrails that can grow with the company. This makes it possible to respond to risks, adopt new technologies, and scale safely without slowing down important work. One of the primary challenges with scaling efficient systems is continually keeping them up-to-date. This requires security leaders to translate long-term goals into repeatable workflows.
How Datadog’s security leaders create secure-by-default systems
At Datadog, each level of security leadership plays a distinct role in scaling reliable, secure systems. Identities in particular are a critical part of building these systems as they shape the perimeter of cloud environments. Additionally, data perimeters are quickly becoming the mechanism for ensuring that identities, resources, and networks interact safely.
We rely on the following leaders to help us accomplish our goals in these growing areas of focus:
- Engineering managers (EMs) define team priorities and implement security guardrails for services
- Directors translate organization-wide goals into policies that teams can use to deploy guardrails consistently
- Vice presidents (VPs) anticipate bottlenecks by funding tooling and system improvements before gaps appear
EMs play an important part in implementing strong perimeters for their services while ensuring their teams still have autonomy. For example, EMs in Datadog’s Internal Trust and Core Observability teams collaborated on building an auto-approved roles system to grant engineers necessary permissions while reducing the number of admin-level users in our systems. This workflow enabled us to easily apply security best practices consistently throughout our platform.
Beyond building systems, EMs also work with directors and VPs to define and track meaningful success metrics, such as mean time to detect and the number of incidents and threats affecting their services. For example, tracking the false positive rate of our detection signals enables us to determine their accuracy. A consistently low percentage of false positives indicates that we have high-fidelity detection signals. If that rate increases at any point, EMs will work with their teams to determine the root cause, such as outdated signals that don’t account for changing indicators of compromise.
Ensuring that specialized teams stay connected and effective
A growing security organization naturally develops specialized domains—including application, platform, cloud security, detection and response, and customer trust—in parallel. Without bridging these domains, teams risk duplicating effort and creating inconsistent standards. This means that in order to maintain trust and consistency as the organization continues to grow, security leadership needs to actively guide investment in these domains. This enables security orgs to build the capabilities that the larger organization needs for faster, safer development. It also makes a compelling case for security’s role and overall involvement in product design and engineering.
How Datadog’s security leaders keep teams connected
We’ve found that ensuring our teams move in the same direction requires clear ownership, shared frameworks, and leadership transparency. Each leadership level builds on the other to reduce the risk of creating siloes:
- EMs define clear team boundaries within their domains so teams can collaborate without duplicating work or introducing inconsistent standards
- Directors maintain shared policies and metrics that help each domain leader remain up-to-date on priorities, standards, and progress
- VPs ensure the priorities for each security domain match and support other company goals
Directors play an important role in this investment. They own and define the shared tools and policies, such as compliance baselines, for their teams. These resources enable them to connect business priorities to security goals and ensure that all domains within the org understand how their work reinforces the company’s area of focus.
Success depends on leaders who communicate clearly and set appropriate expectations for their teams. Deciding when to favor progress over perfection speaks to how security leaders need to balance speed, safety, and focus when company priorities constantly change. For example, as supply chain risks become a bigger concern for companies, security orgs need to consider how to update their existing policies and tooling to meet demand. Enforcing provenance, such as signing and vetting all supply chain artifacts, is one way we’ve helped our teams move quickly and safely as they adopt new technologies.
Anticipating what’s next
Growing a security organization requires scaling existing systems in addition to preparing for what’s next. This means understanding how to balance immediate needs, such as emerging security trends, with preparing for the future. There will always be new trends or areas of concern for a security organization to respond to, so their ability to adapt should be second nature. The challenge is in using new technologies to enhance security without introducing unmanaged risk.
How Datadog’s security leaders prepare for the future
We’ve found that our leadership is most successful when they proactively invest in new technologies. Adopting AI technology is one example of a growing priority for companies. At Datadog, the following leadership levels help shape the way we use AI responsibly:
- EMs embed AI security and privacy controls into existing CI/CD pipelines so new technologies are safely integrated from the start
- Directors develop model governance guidelines so LLMs and their training data are thoroughly vetted
- VPs define risk tolerance, investment priorities, and organization-wide policies for responsible AI adoption
By embedding AI natively, security orgs can protect training data, models, and their output while addressing risks early on. This gives teams the confidence to use AI responsibly and keep up with business demands. For example, a VP might champion the integration of AI-powered anomaly detection into existing monitoring pipelines—a direction our teams have taken through LLM-assisted triage for detection signals, which allows us to significantly reduce manual reviews.
Integrating early, governing clearly, and designing for security by default applies to any new capability or tool a company adopts. At Datadog, we measure success for this in part by routinely reviewing our detection rules and security baselines to ensure they account for new vulnerabilities and threats. We do this by working closely with our security researchers, who are always working to stay on top of emerging security risks, to ensure our detection coverage and playbooks are complete.
Building a security organization that scales efficiently
Security organizations will always need to adapt to evolving threats, technologies, and company goals. Success requires scaling systems to meet demands, ensuring specialized teams remain effective, and addressing existing security risks while preparing for the future. To make this possible, we established leadership roles with clear ownership, which enables teams to operate independently without drifting away from the org’s goals.
To read more about how Datadog’s security organization operates, you can check out these posts about combining our SRE and security groups and our cloud infrastructure security playbook.
If you don’t already have a Datadog account, you can sign up for a free 14-day trial.
