--- title: "Collect Google Cloud Armor logs with Datadog" description: "Learn how you can use Datadog to monitor your environment for DDoS attacks by collecting Google Cloud Armor logs." author: "Sayma Khan, Mallory Mooney" date: 2023-11-14 tags: ["log management", "google cloud", "google cloud armor", "google cloud armor logging"] blog_type_id: the-monitor locale: en --- As the internet continues to evolve, cybersecurity threats—particularly Distributed Denial of Service (DDoS) attacks—are an increasingly significant concern for organizations. In this post, we'll look at how you can use Datadog to collect Google Cloud Armor (GCA) logs and detect and respond to potential DDoS attacks in real-time. But first, we'll briefly cover what DDoS attacks are and how they work. ## A primer on DDoS attacks A DDoS attack is a malicious attempt to overwhelm an application with more traffic than it can handle. To accomplish this, attackers will typically use a complicated network of compromised computers or [botnets](https://www.cloudflare.com/learning/ddos/what-is-a-ddos-botnet/) to flood an application's services, networks, or underlying resources with an excessive amount of HTTP requests. The ultimate goal for these kinds of attacks is to render a target temporarily or indefinitely unavailable to legitimate users, creating incidents that can be costly and challenging to mitigate. Though attackers may use various methods for executing a DDoS attack, the evidence of one in progress is fairly standard. As a general rule, these attacks look like unusual or unexpected surges in web traffic in combination with unresponsive application resources, such as web servers. Indicators of an unresponsive server can include significant spikes in CPU or memory usage. Traffic surges can also originate from a single IP address or a list of known-malicious ones, especially if an attacker is using a system of botnets. Knowing the signs of a DDoS attack can help you stop it before it becomes a serious issue for your application. Next, we'll talk about how tools like Google Cloud Armor automatically detect the presence of DDoS attacks in your environment. ## Shielding against DDoS attacks with Google Cloud Armor [Google Cloud Armor](https://cloud.google.com/armor/docs/cloud-armor-overview) is a cloud-native security service designed to protect your applications against a variety of threats, including DDoS attacks. The service is deployed via [security policies](https://cloud.google.com/armor/docs/security-policy-overview), which are attached to load balancers that sit in front of the application and services you need to protect. Each security policy includes a set of rules that you define to manage web traffic, such as allowlists and denylists for a range of IP addresses. Policies also include [preconfigured web application firewall (WAF) rules](https://cloud.google.com/armor/docs/waf-rules) that automatically stop activity from well-known attack types, such as SQL injections or cross-site scripting. ### Create a new security policy for your application and services This post assumes that you have a Google Cloud project and configured application load balancers. To get started, navigate to Google Cloud Console and select the appropriate project to work in. Next, search for and select the "Cloud Armor policies" option, which is a part of "Network Security"—you can also find this product under "Networking" in the navigation menu. This action will take you to your security policies dashboard, where you can review existing policies and create new ones. Click "Create Policy" near the top of the page to create a new security policy. #### Configure a new policy with a basic set of rules The first step of this process is to customize the policy's name, type, scope, and default behavior: ![Google Cloud Armor step 1](https://web-assets.dd-static.net/42588/1776292547-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-step1.png) For this post, we'll use the "Backend security policy" type. For more information about the different types of load balancers that security policies support (e.g., backend or edge), you can check out [Google's documentation](https://cloud.google.com/armor/docs/security-policy-overview). You can set the scope of the policy to either "Global" or "Regional" to determine which geographic location that Google Cloud Armor should be deployed. Finally, you can configure the policy's default behavior—allow or deny—and the appropriate response code. With the basic setup seen in the preceding screenshot, Google Cloud Armor will automatically allow all traffic to the configured target, which we'll look at in a later step. #### Optionally add more policy rules Clicking the "Next" button will take you to the "Add more rules" step, where you can fine-tune your policy. For example, in the following screenshot, we created an additional rule that customizes the [sensitivity level](https://cloud.google.com/armor/docs/rule-tuning) of Google Cloud Armor's preconfigured SQL injection rule. ![Google Cloud Armor step 2](https://web-assets.dd-static.net/42588/1776292551-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-step2.png) You can also create rules for a range of IP addresses, such as those that are known to be associated with scanning tools or other types of malicious activity, for greater control over application traffic. #### Apply a policy to a target Clicking "Next Step" again will give you the option to assign a target load balancer to your policy. Select the appropriate load balancer backend service from the dropdown menu, as seen in the following screenshot: ![Google Cloud Armor step 3](https://web-assets.dd-static.net/42588/1776292556-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-step3.png) Since we selected the "Backend security policy" type in the first step, our targets will include all available load balancers in our Google Cloud project. Click "Next Step" to enable Google Cloud Armor's [Adaption Protection](https://cloud.google.com/armor/docs/adaptive-protection-overview?) if needed. Once you have configured your policy with the appropriate rules and targets, you can click "Create Policy" to deploy Google Cloud Armor. With this step, Google Cloud Armor will begin monitoring traffic to your configured load balancers and automatically generate relevant activity logs. Next, we'll show you how you can configure Datadog to collect and analyze these logs for enhanced security monitoring. ## Enhance your DDoS protection strategy with Datadog Datadog enables you to monitor traffic patterns in real time and surface any anomalies that could indicate a DDoS attack. In this section, we'll show how you can monitor your Google Cloud Armor setup by: - [Collecting performance metrics with Datadog's Google Cloud integration](#configure-datadogs-google-cloud-integration-to-collect-telemetry-data) - [Forwarding Google Cloud Armor logs to Datadog](#forward-google-cloud-armor-logs-to-datadog) - [Using Datadog monitors and Cloud SIEM to easily surface issues that Google Cloud Armor logged](#automatically-detect-ddos-attacks-with-datadog-monitors-and-cloud-siem) ### Configure Datadog's Google Cloud integration to collect telemetry data A Datadog account is a prerequisite for following the steps in this section. If you don't already have an account, you can sign up for a today. Datadog provides an out-of-the-box integration with Google Cloud, enabling you to easily collect telemetry data from the application resources that can be vulnerable to a DDoS attack. Since DDoS attacks attempt to make application resources unavailable, it's important to be able to monitor performance metrics like CPU and memory usage. Correlating this data with Google Cloud Armor logs can help you determine if your application is experiencing a DDoS attack. Check out [our guide](https://docs.datadoghq.com/integrations/google_cloud_platform.md#installation) to learn how you can enable the integration and automatically discover your Google Cloud projects via service account impersonation. Once configured, Datadog will automatically start collecting metrics from Google Cloud resources. ### Forward Google Cloud Armor logs to Datadog Google Cloud Armor generates two types of logs: audit and request. Audit logs will log [admin-level activity](https://cloud.google.com/armor/docs/audit-logging) for your target load balancers and security policies, such as deleted or updated policies. Request logs are generated as part of your load balancers logging mechanism and [capture information about your security policies and how they respond to HTTP requests](https://cloud.google.com/armor/docs/request-logging). For example, if a security policy denies an incoming request, Google Cloud Armor will log that interaction. There are a couple of options for collecting Google Cloud Armor logs, but we recommend using Google Cloud Dataflow and [our dedicated template](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog). This approach requires a [Pub/Sub subscription](https://www.datadoghq.com/blog/stream-logs-datadog-dataflow-template.md), which enables you to stream data from Google Cloud to a configured destination like Datadog. We'll focus on forwarding request logs here, but you can check out [this post](https://www.datadoghq.com/blog/monitoring-gcp-audit-logs.md#shipping-your-audit-logs) to learn how to forward audit logs to Datadog. For example, you'll want to include the `backendServices` and `securityPolicies` resource types in your configuration. For request logs, [create a new sink](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink), which aggregates groups of logs to route to a Pub/Sub subscription. As part of the process, you can create a new Pub/Sub topic to serve as the subscription's destination for your Google Cloud Armor logs (if you don't already have one). The sink's inclusion filter should include specific criteria like the load balancer type and security policy name in order to fine-tune which logs you want to include, as seen in the following snippet: ```text resource.type:(http_load_balancer) AND jsonPayload.enforcedSecurityPolicy.name:(sample-security-policy) ``` Once you've configured your sink, you can [create a Dataflow job](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog#run-the-template ) to stream logs from your new Pub/Sub subscription to Datadog using our dedicated template. ![Google Cloud Armor log in Datadog](https://web-assets.dd-static.net/42588/1776292560-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-log-2.png) ### Automatically detect DDoS attacks with Datadog monitors and Cloud SIEM You can use Datadog monitors to automatically alert you to any anomalies in traffic behavior logged by Google Cloud Armor request logs, such as sudden spikes in HTTP requests or abnormal resource utilization. ![Google Cloud Armor monitor in Datadog](https://web-assets.dd-static.net/42588/1776292564-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-monitor.png) Datadog Cloud SIEM can take your monitoring further by analyzing Google Cloud Armor request and audit logs to automatically detect the signs of a DDoS attack. This visibility can help provide more context to any triggered monitors, which alert on the initial signs of an attack. ![Google Cloud Armor step 1](https://web-assets.dd-static.net/42588/1776292569-collect-google-cloud-armor-logs-with-datadog-google-cloud-armor-detection-rule2.png) For example, you can configure a Cloud SIEM detection rule to notify you when settings for a load balancer change, which is activity that's logged in Google Cloud Armor audit logs. This type of admin-level activity is routine, but it could also end up being the root cause of a successful DDoS attack if it introduces a misconfiguration that makes the load balancer more vulnerable. ## Continuously improve your Google Cloud Armor security policies Monitoring your Google Cloud Armor logs with Datadog can help you fine-tune your security policies. For example, if a security policy is generating too many false positives, you can [decrease its sensitivity level](https://cloud.google.com/armor/docs/rule-tuning). This adjustment can help you find the right balance in the types and volume of requests that Google Cloud Armor logs. Datadog can also help you define your protection and response strategies. For example, Datadog's anomaly algorithm can monitor historical trends in application traffic to discover DDoS attacks on new resources as well as new attack approaches, such as a new system of botnets. After an incident, you can create comprehensive runbooks to ensure your teams are equipped to handle future attacks. Together, these capabilities ensure that you can continue to stay on top of evolving DDoS attack vectors. ## Start collecting Google Cloud Armor logs today Defending your applications from DDoS attacks is a critical aspect of modern cybersecurity and ensuring the continued availability and reliability of your services. Datadog's ability to collect and analyze Google Cloud Armor logs equips you with the tools, threat intelligence, and context necessary to detect and respond to DDoS threats in real time. Check out [Google Cloud Armor's documentation](https://cloud.google.com/armor?hl=en) to learn more about their capabilities. For more information about Datadog's log collection and security monitoring capabilities, you can check out [our documentation](https://docs.datadoghq.com/). If you don't already have a Datadog account, sign up for a today.