Automate Remediation of Threats Detected by Datadog Security Monitoring | Datadog

Automate remediation of threats detected by Datadog Security Monitoring

Author Justin Massey
Author Jonathan Epstein

Published: June 16, 2021

When it comes to security threats, a few minutes additional response time can make the difference between a minor nuisance and a major problem. Datadog Security Monitoring enables you to easily triage and alert on threats as they occur. In this post, we’ll look at how you can use Datadog’s webhooks integration to automate responses to common threats Datadog might detect across your environments. This way, Datadog can act as a centralized threat remediation platform that connects to and initiates behavior in your other services, shaving valuable time off your threat-response workflow.

Use Datadog and webhooks to automate security responses

Webhooks are HTTP callbacks (usually written in JSON) that send messages to applications whenever a certain condition is met. Those applications can then trigger further actions based on the data contained in the webhook payload. Webhooks act as script-based connectors that link Datadog to your other tools. By setting up webhooks that respond to your Datadog security notifications, you can create simple, automated remediation workflows that neutralize threats in real-time.

Datadog’s webhook integration makes it easy to set up webhook messages that deliver their payloads to the services you want to automate whenever a Detection Rule is broken. Every webhook payload contains information on the triggering event and a custom message that can be used to initiate services downstream. You can automate commands for any service that has a webhook URL. The following is a simple webhook that will send a payload with the relevant data from the Security Signal that triggered it.

{
  "SECURITY_RULE_NAME": "$SECURITY_RULE_NAME",
  "SECURITY_SIGNAL_ID": "$SECURITY_SIGNAL_ID",
  "SECURITY_SIGNAL_ATTRIBUTES": $SECURITY_SIGNAL_ATTRIBUTES
}

In the following sections, we’ll walk through some more examples of how to use Datadog with webhooks to automate security responses and take a look at how to properly configure webhook payloads to respond to your Datadog alerts.

Delete misconfigured security groups

In order to secure your cloud environment, it’s important to delete a misconfigured resource as soon as it is created. You can do this by sending a webhook to your cloud provider’s API management service. For instance, the following diagram illustrates an automation workflow that uses a webhook to connect Datadog’s AWS integration with Amazon API Gateway to perform account management tasks.

Using webhooks with Datadog Security Monitoring allows you to automate security procedures in your cloud environment.

If an AWS user creates a poorly configured resource (e.g., an overly permissive security group, user role, etc.) within your AWS environment, Datadog Log Management ingests the related log. That triggers a security group–based Detection Rule, which then sends the webhook’s JSON payload to the designated AWS API Gateway URL, which in turn activates a AWS Lambda function that automatically deletes the offending resource.

Ban a suspicious IP address

Your cloud resources might receive traffic from hundreds of IP addresses a day, most of which belong to known and trusted users. A sign-in from an unrecognized IP address, however, might represent an attacker manipulating a trusted user’s credentials, with which they can then access your data and gain persistence in your environment.

To spot these types of threats, you can use Datadog Security Monitoring’s new term detection method, which analyzes your account’s historical data over a chosen period of time and alerts on previously unseen values in your cloud logs. To respond to anomalous activity as soon as it occurs, you can set up a new term–based Detection Rule that, when triggered, sends a webhook payload to your cloud’s identity and access management (IAM) service to ban the unknown IP. This way, no time is wasted between threat detection and resolution. The following example illustrates what the relevant webhook payload might look like when triggered by a new term-based detection rule from Datadog. It includes the relevant Security Signal attributes that our downstream service needs, which in this case is the IP address that Datadog has detected. Datadog then sends the payload to your IAM service, which can ban the offending address.

{
  "SECURITY_RULE_NAME": "Request from unexpected IP address",
  "SECURITY_SIGNAL_ID": "abcd1234",
  "SECURITY_SIGNAL_ATTRIBUTES": {
    "network": {
      "client": {
        "ip": [
          "1.2.3.4"
        ]
      }
    }
  }
}

Auto-remediate application abuse and fraud

Just as Datadog can help you spot anomalous activity in your logs, you can also use it to uncover patterns of abuse or fraud across your application. For example, you can set up a Detection Rule that gets triggered when a user repeatedly attempts to purchase something in your application with invalid credit card details. Datadog will then generate a Security Signal that details the offense as well as the suspicious user’s particulars, such as their IP address and user ID, and sends a webhook payload with remediation instructions to a service that disables the user. The following sample contains this identifying information from the Security Signal:

{
  "SECURITY_RULE_NAME": "Fraudulent Credit Card Authorizations",
  "SECURITY_SIGNAL_ID": "efgh5678",
  "SECURITY_SIGNAL_ATTRIBUTES": {
    "usr": {
      "id": "john.doe@your_domain.com"
    },
    "evt": {
      "name": "credit_card_authorization",
      "outcome": "fail"
    },
    "network": {
      "client": {
        "ip": [
          "1.2.3.4"
        ]
      }
    }
  }
}

Webhook, line, and sinker

By using webhooks to integrate Datadog Security Monitoring with other services across your stack, you can easily create Detection Rules that trigger auto-remediation workflows. Security Monitoring is part of the Datadog Cloud Security Platform, which protects an organization’s production environment with a full-stack offering providing threat detection and posture management, as well as workload security and application security. For more information on securing your environment with Datadog, check out our security documentation. And if you’re not already a Datadog customer, get started today with a .