Jaclyn Verga
Addie Beach
Proactively addressing risks in technical environments is a constant challenge. Many teams wait until it’s too late and key application functionality is disrupted or sensitive data is exposed. However, understanding risk severity in context can be difficult, especially in distributed systems where related issues and impacts may not be immediately obvious.
To manage risks effectively, teams need all risk-related data from across their systems in a single place combined with clear, actionable processes for mitigation. In this post, we’ll explore how our Risk team uses Datadog Case Management to collect this data into a central risk register and automate key remediation activities.
Creating a single source of truth for risk data
As the Datadog Risk team, we’re responsible for raising awareness of areas within our environments that could expose Datadog to harm. This role involves detecting, triaging, and prioritizing risks, as well as driving enforcement on risk remediation and severity reduction. We help teams throughout Datadog better understand the risks in their processes and technologies, enabling them to either eliminate these risks or reduce their severity.
With data spread across multiple internal organizations, it’s often hard to understand the severity of risks across the company. What may appear to be a minor, isolated risk can quickly be revealed as a meaningful business risk when we cross-reference it with related risks, vulnerabilities, or security incidents across other orgs.
For example, an over-permissioned cloud role in a lower environment may be categorized as low risk when no active threats are observed. Yet, if related indicators emerge—such as abnormal API usage or exploitation of similar roles in peer orgs—the potential blast radius expands to the point where the risk demands immediate attention. As a result, creating a centralized view of data from all of our orgs was essential to assessing Datadog’s overall risk posture.
We turned to Datadog Case Management to build this repository. Using Case Management’s ticketing functionality, we organized risks into a single, sortable list and enhanced them with cross-org findings.
Next, we used the Datadog API to enhance this list through a custom AI tool that evaluates the incoming risks and correlates them with data from our orgs.
The tool automatically populates each case with:
- A calculated risk score and case priority based on potential impact, likelihood of exploitation, control effectiveness, and factors that could increase the risk exposure.
- A mitigation plan detailing containment steps, timelines, dependencies, blockers, and residual risk.
- Correlated context including related risk cases, incidents, and OKRs, as well as vulnerabilities identified in Datadog Code Security.
To ensure the risk information compiled by our AI tool is both complete and accurate, we use Datadog LLM Observability to evaluate and monitor the outputs. LLM Observability gives us end-to-end tracing for our AI agents, with visibility into inputs, outputs, latency, token usage, and errors at each step. Additionally, we use LLM Observability to create structured experiments that test new prompts and validate output quality before integrating updates into our risk register tool. We also use these experiments to estimate potential cost changes our updates may introduce.
Once our tool has enhanced the risk data, we run workflows created with Workflow Automation to automate manual redundancies throughout the risk lifecycle. For example, we’ve established a workflow that automatically assigns risks to specific team members based on case attributes. Other workflows notify the owner, update the case status to in-progress, and set the associated team field.
From here, we use Case Management’s custom statuses to accurately represent the stage of each case. This helps differentiate cases that are actively being worked on versus ones that are blocked while we await new information. As our mitigation efforts reduce exposure, we can also adjust the risk severity rating downward to reflect an improved posture. Alternatively, if evidence emerges that the risk is being exploited, we can escalate the risk and treat it as an active incident within Datadog.
Remediating risks faster and more effectively
Once we organized our data, Case Management helped us improve our remediation and analysis processes. Some risks can be reduced quickly through targeted controls, while others require sustained mitigation efforts and ongoing reassessment. Adapting remediation processes and tracking mitigation activities for risks with varying severities and timelines is challenging but critical for maintaining a consistently strong risk posture.
To standardize our response processes across risks, we created additional workflows that simplify our remediation tasks. Within Case Management, we can access and run these workflows without ever leaving a case. For example, if the team member assigned to a risk case identifies that it relates to a customer access request, they can run a workflow that compiles recent request data and supports further assessment.
Additionally, using our consolidated risk register data, we created role-specific risk dashboards that give stakeholders tailored overviews of our current risk posture and visibility into our mitigation efforts. To do this, we created a custom metric that collected data on our risk cases. This metric enhances the default Case Management analytics data.
By filtering this metric on different facets, we generated widgets that provide a quick snapshot of risk management activity, such as:
- A breakdown of open risks by control domain and criticality
- A summary of risks in each status
- A timeline of risk severity history from the past 30 days
- A list of risks with severity changes over time
Start managing risk across your system with Datadog
Rapidly identifying and mitigating risks is critical to preventing them from escalating into active security incidents. However, the breadth and variety of modern threat vectors make consistent prioritization and management challenging. Additionally, risk exposure is rarely isolated. Adversaries often exploit multiple weaknesses together, making cross-organization risk correlation essential to understanding impact.
Using Datadog Case Management, our Risk team detects and identifies systemic risk patterns, gains visibility into exposure, and tracks mitigation efforts to strengthen overall risk posture. Datadog Case Management is free for all users. You can learn more by visiting our Case Management documentation. Or, if you’re new to Datadog, you can sign up for a 14-day free trial.
