---
title: "How to monitor Consul with Datadog"
description: "Monitor metrics and logs from all of your Consul hosts in a single platform with Datadog."
author: "Paul Gottschling"
date: 2019-07-10
tags: ["infrastructure monitoring", "hashicorp", "consul", "configuration management"]
blog_type_id: the-monitor
locale: en
---

In [Part 1](https://www.datadoghq.com/blog/consul-metrics.md), we walked you through key metrics for monitoring Consul, and in [Part 2](https://www.datadoghq.com/blog/consul-monitoring-tools.md), we showed you how to use Consul's built-in monitoring tools to get insights from your cluster. In this post, we'll show you how you can use Datadog to visualize your Consul metrics in custom dashboards, analyze your Consul logs, and set up alerts on your cluster's availability and performance. With more than 1,000 [integrations](https://docs.datadoghq.com/integrations.md), Datadog gives you deep visibility into Consul—and all of the applications that rely on Consul for service discovery and dynamic configuration—in a single platform. We'll explain how to:

1. Set up the [Consul integration](#get-started-with-datadog-and-consul)
1. Use [Datadog dashboards](#real-time-views-of-consul-cluster-health) to get visibility into multiple components of Consul—and the rest of your system
1. Collect, search, and analyze [Consul logs](#analyze-all-your-consul-logs-in-one-place)
1. Alert on unhealthy nodes and volatile leadership with [Consul monitors](#maintain-a-healthy-consul-cluster-with-datadog-alerts)

![The out-of-the-box dashboard for Consul.](https://web-assets.dd-static.net/42588/1776354737-consul-datadog-oob-dash.png)
*The out-of-the-box dashboard for Consul.*

## Get started with Datadog and Consul

The Datadog Agent is [open source software](https://github.com/DataDog/datadog-agent) that runs [on your hosts](https://docs.datadoghq.com/agent.md?tab=agentv6), reporting metrics, traces, and logs to Datadog. There are two methods the Datadog Agent uses to gather metrics from Consul, and you can configure one or both of them depending on the data you would like to collect.

First, you can configure Consul to send metrics directly to the Datadog Agent as one of the [telemetry sinks](https://www.datadoghq.com/blog/consul-monitoring-tools.md#configure-telemetry-sinks-for-comprehensive-monitoring) we explained in Part 2. These are the metrics that Consul uses to report its internal behavior, and includes those we introduced in [Part 1](https://www.datadoghq.com/blog/consul-metrics.md).

Second, the Agent queries Consul's [HTTP API](https://www.consul.io/api-docs) and uses the results to generate metrics about node and service status (i.e., whether they are `up`, `passing`, `warning`, or `critical`) as well as network latency (based on Consul's [network coordinates API](https://www.consul.io/api-docs/coordinate)). These metrics are useful for tracking the state of nodes and services within your Consul cluster.

To install the Agent on each of your Consul hosts, follow the instructions for your environment within your [Datadog account](https://app.datadoghq.com/account/settings/agent/latest?platform=overview).

### Push Consul-generated metrics to the Datadog Agent

The Datadog Agent ingests metrics with an aggregation service called [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd.md), and you will need to provide the address of this service to Consul. For each host you'd like to report metrics to Datadog, add the following to your Consul configuration (assuming DogStatsD is running locally on your Consul host):

*conf.json*
```json
"telemetry": {
  "dogstatsd_addr": "127.0.0.1:8125"
}
```

This instructs Consul to declare a [metrics sink for DogStatsD](https://godoc.org/github.com/armon/go-metrics/datadog) with the [Golang package](https://github.com/armon/go-metrics#sinks) it uses to expose metrics, meaning that any metrics Consul generates will go to DogStatsD in addition to the default in-memory sink and any other metric sinks you have enabled.

The DogStatsD sink uses the [DogStatsD client for Golang](https://github.com/DataDog/datadog-go/tree/master/statsd) to flush metrics [every 10 seconds](https://docs.datadoghq.com/developers/dogstatsd.md#how-it-works) to Datadog, where they are retained for [15 months](https://docs.datadoghq.com/developers/faq/data-collection-resolution-retention.md) (whereas Consul's default in-memory sink only has a [one-minute](https://github.com/hashicorp/consul/blob/6960afdfe3b3354fbd1d234273f4482002715d5a/lib/telemetry.go#L330-L332) retention period). Make sure to stop and restart Consul ([rather than running](https://www.consul.io/docs/agent/options.html#reloadable-configuration) `consul reload`) after changing your `telemetry` configuration, taking care that the [`-config-file` or `-config-dir`](https://www.consul.io/docs/agent/options#_config_file) options of the `consul agent` command include the path to the configuration file you just edited. Once it's running, Consul will start sending metrics to the Datadog Agent.

Consul's internal metrics are namespaced, and use dots to delimit hierarchies of concepts. For example, [`consul.http.GET.v1.kv._`](https://www.consul.io/docs/agent/telemetry#metrics-reference) tracks how long (in milliseconds) it takes the KV store's API endpoint to handle requests. The Datadog Agent can extract tags from these namespaces so you can group your Consul metrics more meaningfully. With the `consul.http` example above, you could compare response latencies across all of Consul's API endpoints to see if slow queries are due to your configuration—e.g., you've enabled [blocking queries](https://www.consul.io/api/features/blocking) for certain endpoints—or resource contention on your Consul hosts. Follow our [documentation](https://docs.datadoghq.com/integrations/consul.md#dogstatsd) to see how to configure the [DogStatsD Mapper](https://www.datadoghq.com/blog/dogstatsd-mapper.md) to start parsing your ingested Consul metrics.

### Pull API-based metrics from Consul

You can [configure](https://docs.datadoghq.com/integrations/consul.md#configuration) the Datadog Agent to query Consul's API by editing the **conf.yaml.example** [template file](https://github.com/DataDog/integrations-core/blob/master/consul/datadog_checks/consul/data/conf.yaml.example) in the **consul.d** subdirectory of your Datadog Agent (the location varies across platforms—refer to the [docs](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md?tab=agentv6#checks-configuration-files-for-agent-6)). Copy and rename this file as **conf.yaml**, and update it with the location of your Consul HTTP server.

You can edit **conf.yaml** to collect additional metrics and events from Consul using options such as the following:

1. `self_leader_check: true`: emit events when a [new leader is elected](#maintain-a-healthy-consul-cluster-with-datadog-alerts)
1. `network_latency_checks: true`: collect the [estimated](https://www.consul.io/docs/internals/coordinates.html) duration of requests from a given node to others
1. `catalog_checks: true`: query the Consul catalog for counts of nodes and services; metrics will be tagged by `mode:leader` or `mode:follower` (as applicable), meaning that you can use these metrics to alert on the loss of a leader

After making your changes, **conf.yaml** should resemble the following. You can find a full list of options within [**conf.yaml.example**](https://github.com/DataDog/integrations-core/blob/master/consul/datadog_checks/consul/data/conf.yaml.example).

*conf.yaml*
```yaml
init_config:

instances:
    - url: http://localhost:8500
      self_leader_check: true
      network_latency_checks: true
      catalog_checks: true
```

Datadog allows you to group and filter data using [tags](https://docs.datadoghq.com/tagging.md). The Consul integration tags all metrics with `consul_datacenter`, and we have shown how certain Consul metrics are tagged with `mode:leader` or `mode:follower`.

You can also configure the Datadog Agent to tag all of the metrics from its local host, helping you isolate host-level metrics from your Consul cluster when creating alerts and visualizations. To do this, add a custom key-value pair to the `tags` section of your Agent [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md?tab=agentv6#agent-main-configuration-file) (**datadog.yaml**). The Host Map shown [below](#real-time-views-of-consul-cluster-health) uses the host-level `consul_cluster` tag to visualize CPU utilization per host in the cluster.

*datadog.yaml*
```yaml
tags:
  - consul_cluster:consul-demo
```

[Restart the Datadog Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md?tab=agentv6#restart-the-agent) to apply these configuration changes.

### Enable log collection

Consul's logs provide insights into internal operations such as membership changes and the status of Raft elections (see [Part 2](https://www.datadoghq.com/blog/consul-monitoring-tools.md)), and aggregating and analyzing these logs with Datadog can help you diagnose issues. To [enable log collection](https://docs.datadoghq.com/integrations/consul.md#log-collection), revise your Datadog Agent [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md?tab=agentv6#agent-main-configuration-file) (**datadog.yaml**) to include the following:

*datadog.yaml*
```yaml
  logs_enabled: true
```

Then update the `logs` section of your Consul integration configuration file (e.g., **consul.d/conf.yaml**) to include the following:

*conf.yaml*
```yaml
instances:

# [...]

logs:
    - type: file
      path: <CONSUL_LOGS_DIRECTORY_PATH>/*
      source: consul
      service: <SERVICE_NAME>
```

This configuration instructs Datadog to tail one or more files at the specified path. Consul automatically adds a timestamp to the name of the log file and rotates it every 24 hours (this is [configurable](https://www.consul.io/docs/agent/options.html#_log_rotate_duration)). You can use a [wildcard](https://docs.datadoghq.com/logs/log_collection.md?tab=tailexistingfiles#tail-multiple-directories-or-whole-directories-by-using-wildcards) to configure Datadog to tail all log files (regardless of their names) within the chosen directory.

Specifying a `source` is mandatory, and activates the out-of-the-box processing pipeline for Consul logs. The `service` key (which is also mandatory) adds a `service:` tag that you can use to group and filter your logs. You can also use this tag to correlate your Consul logs with metrics as well as [distributed traces](https://docs.datadoghq.com/tracing.md) of requests to your applications. This can either be `consul` or a more specific categorization of your Consul deployment, such as  `consul-client`, `consul-server`, or `consul-demo`.

Once you've edited your configuration YAML, make sure the `dd-agent` user has [read access](https://docs.datadoghq.com/agent/faq/how-to-solve-permission-denied-errors.md) to your log files, then [restart the Datadog Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md?tab=agentv6#restart-the-agent) to apply the changes. You can verify that the Agent is reporting to Datadog by running the [`status` command](https://docs.datadoghq.com/agent/guide/agent-commands.md?tab=agentv6#agent-status-and-information) and looking for the `consul` section.

### Enable network monitoring

[Monitoring Consul networking](#network-performance-monitoring) allows you to visualize dependencies within your cluster, plus the latency and volume of traffic. To enable Network Performance Monitoring (NPM), see the [NPM setup documentation](https://docs.datadoghq.com/network_monitoring/performance/setup.md?tab=kubernetes#setup) for guidance on configuring your hosts, containers, Kubernetes clusters, or ECS clusters.

If you're using Consul with Kubernetes, NPM requires you to add the appropriate annotations to your pods and environment variables to your containers. The code snippet below shows a Deployment manifest that contains this information.

```yaml
apiVersion: apps/v1
kind: Deployment
[...]
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/connect-service-upstreams: 'web:9091'
        consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "9090"
        consul.hashicorp.com/transparent-proxy: "false"
        admission.datadoghq.com/enabled: "true"
        ad.datadoghq.com/envoy-sidecar.tags: '{"service": "envoy"}'
[...]
    spec:
      containers:
      - name: "<CONTAINER_NAME>"
        image: "<CONTAINER_IMAGE>/<TAG>"
        env:
          - name: TRACING_DATADOG_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
```

Additionally, you'll need to update your Consul config to include `extraConfig` and `extraLabel` items that contain the necessary values as shown in [this gist](https://gist.github.com/davidmlentz/255b68f66361fd18d78135a8955d2ee6). You'll also need to set your services' [ProxyDefaults](https://www.consul.io/docs/connect/config-entries/proxy-defaults) configuration entry as shown [here](https://gist.github.com/davidmlentz/9e5ac1db905d428657e1e7a0599066ec).

## Real-time views of Consul cluster health

Consul clusters are complex, and having an overview of all the components in a single view can help you identify patterns and isolate issues. Datadog provides a number of ways to visualize your entire cluster at once.

In this section, we'll show you how to use these Datadog features to gain visibility into the health of your Consul cluster:

- [The Host Map](#the-host-map)
- [Network Performance Monitoring](#network-performance-monitoring)
- [Dashboards](#dashboards)

### The Host Map 

Datadog's [Host Map](https://docs.datadoghq.com/graphing/infrastructure/hostmap.md) is a good starting point for visualizing resource utilization across your Consul cluster. You can group hosts by tag, making it clear when hosts belonging to a specific datacenter, service, or other part of your infrastructure require attention.

As shown below, you can filter the Host Map to display only those hosts that belong to a specific Consul cluster, using the custom `consul_cluster` tag. We can see how our hosts are performing in terms of a specific metric (in this case, CPU utilization). By identifying the scope of an issue—does it affect the whole cluster, or just a single host?—you can determine what to troubleshoot next.

![hostmap](https://web-assets.dd-static.net/42588/1776354745-consul-datadog-hostmap.png)

### Network Performance Monitoring

[Network Performance Monitoring](https://www.datadoghq.com/product/network-monitoring/) lets you visualize real-time traffic between the nodes in your Consul cluster. Just like the Host Map, the [Network Map](https://docs.datadoghq.com/network_monitoring/performance/network_map.md) supports tagging so you can filter and explore your network data on dimensions like service and cluster, as well as any custom tags you've applied. The Network Map—shown below—illustrates the volume of traffic, the number of connections, and the latency between nodes in your Consul network, making it easy to spot issues with the health of your cluster. If the Network Map shows unexpected traffic patterns—such as elevated latency or unusual request rates—you can click a node on the map to inspect its logs and processes. See our [blog post](https://www.datadoghq.com/blog/monitor-consul-with-datadog-npm.md) to learn more about using NPM to monitor the performance of your Consul network.

![The Network Map shows traffic between Kubernetes pods within a Consul cluster.](https://web-assets.dd-static.net/42588/1776301564-monitor-consul-with-datadog-npm-consul-kubernetes-network-map.png)

### Dashboards

For a more detailed view of Consul's performance, you can use Datadog's [out-of-the-box dashboard](https://app.datadoghq.com/screen/integration/83/consul---overview) to view metrics [key metrics from your Consul environment](https://www.datadoghq.com/blog/consul-metrics.md). The [template variables](https://docs.datadoghq.com/graphing/dashboards/template_variables.md) allow you to filter the entire dashboard to display metrics from a particular service or datacenter of your choice. You can clone and customize this dashboard to correlate Consul and system-level metrics with information from more than 1,000 [other technologies](https://docs.datadoghq.com/integrations.md).

![template-vars](https://web-assets.dd-static.net/42588/1776354752-consul-datadog-template-vars.png)

To bring more visibility into your own Consul environment, you can clone the dashboard to add other graph [widgets](https://docs.datadoghq.com/graphing/widgets.md), or track data from any Datadog integration. You can show, for example, metrics from [SNMP](https://docs.datadoghq.com/integrations/snmp.md) and [DNS](https://docs.datadoghq.com/integrations/dns_check.md), allowing you to get more visibility into the possible causes of underperforming network connections. And if you set `network_latency_checks` to `true` within the Consul integration configuration file, you can create graphs that show network latency between a given node and others in the cluster (all metrics beginning with `consul.net` listed in [our documentation](https://docs.datadoghq.com/integrations/consul.md#metrics)).

The dashboard below combines Consul metrics with system-level metrics from each host in your cluster to provide context for troubleshooting. In this example, you can see that a drastic decline in RPCs did not accompany a rise in failed RPCs, but rather a system-wide period of unavailability.

![consul-resource-dash](https://web-assets.dd-static.net/42588/1776354761-consul-datadog-consul-resource-dash.png)

If a particular Consul metric shows an unexpected change, you can click on the graph and navigate to logs from the same point in time. The next section will explain how to use Datadog log management to track and analyze all of your Consul logs.

![logs-transition](https://web-assets.dd-static.net/42588/1776354766-consul-datadog-logs-transition.png)

## Analyze all your Consul logs in one place

Consul agents include a lot of valuable information in their logs [out of the box](https://www.datadoghq.com/blog/consul-metrics.md#metrics-to-watch-consulclientrpc-consulclientrpcexceeded-consulclientrpcfailed-consulrpccross-dc). Since you'll have a whole cluster of agents reporting logs, collecting and analyzing them in a single platform is more efficient than SSHing into each node and reading log files locally when Consul encounters an issue. Using Datadog is also well-suited for the kind of dynamic cluster you would manage with Consul, as hosts continue to report logs even as they spin up, shut down, and change IPs, as long as they run a Datadog Agent configured to send logs to Datadog.

![log-explorer](https://web-assets.dd-static.net/42588/1776354770-consul-datadog-log-explorer.png)

You may recall from [Part 2](https://www.datadoghq.com/blog/consul-monitoring-tools.md) that Consul logs include several commonalities: a timestamp, the severity, the package that calls the logger, and a message, which can have various formats. Datadog's out-of-the-box processing pipeline automatically extracts attributes from your Consul logs, and enriches them with any tags specified in the `logs` section of your integration configuration file (**consul.d/conf.yaml**).

![Log collected by Datadog showing that a node hosting RabbitMQ has joined the cluster and synced its state.](https://web-assets.dd-static.net/42588/1776354777-consul-datadog-parsed-log.png)
*Log collected by Datadog showing that a node hosting RabbitMQ has joined the cluster and synced its state.*

Since Consul's log messages are written in plain English and sometimes include variable values at different locations depending on the log, they can be difficult to aggregate. Datadog's [Log Patterns view](https://www.datadoghq.com/blog/log-patterns.md) can automatically find commonalities between logs, helping you figure out which kinds of logs are most common and where to take action first.

For example, the Log Patterns view below indicates that logs carrying the message, `[WARN] manager: No servers available` are the most common over the chosen timeframe, meaning that we'll want to take further action to investigate why no server nodes have joined the cluster.

![log-patterns](https://web-assets.dd-static.net/42588/1776354785-consul-datadog-log-patterns.png)

## Maintain a healthy Consul cluster with Datadog alerts

While Consul already provides some automated monitoring in the form of health checks and watches, Datadog allows you to visualize all of your Consul data in context with the rest of your infrastructure, as well as to tailor your alerting strategy for your specific use case. Datadog can alert you of possible Consul incidents such as frequently failing health checks and unstable cluster leadership.

Rather than using [manual API and CLI commands](https://www.datadoghq.com/blog/consul-monitoring-tools.md) to track your Consul health checks, you can let Datadog notify you of their status, and see any related metrics, traces, or logs in the same platform as the rest of your Consul data. Since Datadog monitors Consul health checks [automatically](https://docs.datadoghq.com/integrations/consul.md#service-checks) as Datadog [service checks](https://www.datadoghq.com/blog/alerting-101-status-checks.md#service-checks), you can set an alert when a certain number of checks fail within a specific period of time by [creating an integration monitor](https://docs.datadoghq.com/monitors/monitor_types/integration.md).

The alert below reviews Consul's last three node-level health checks (`last(3)`), counting them by status ("Alert," "OK," or "Warn") and triggering after one "Alert." You can also configure a similar alert for tracking failed service-level health checks by using the `consul_service_id` tag (pulled automatically from Consul based on the [ID of the service](https://www.consul.io/docs/agent/services.html)). A custom message gives teammates access to helpful documents and dashboards. You can use a [tag variable](https://docs.datadoghq.com/monitors/notifications.md?tab=is_alertis_warning#tag-variables) (such as `{{host.name}}`) in the dashboard link, so the incident responder can easily view metrics collected from the host that triggered the alert.

![alert-status](https://web-assets.dd-static.net/42588/1776354793-consul-datadog-alert-status.png)

Frequent leadership transitions can compromise the availability of Consul's strongly consistent data (see [Part 1](https://www.datadoghq.com/blog/consul-metrics.md#leadership-transitions)), preventing cluster members from accessing network addresses and configuration details. Datadog [produces an event](https://docs.datadoghq.com/integrations/consul.md#events) whenever Consul elects a new leader (as long as `self_leader_check` is set to `true` in your Consul integration [configuration file](#get-started-with-datadog-and-consul)), making it possible to alert on a certain number of leadership transitions over a certain period of time. In the example below, we've used the `consul_datacenter` tag to trigger a separate alert for each Consul datacenter that breaches our threshold.

![leadership-alert](https://web-assets.dd-static.net/42588/1776354801-consul-datadog-leadership-alert.png)

You can also alert on the loss of a leader node by using any Consul metric that includes the `mode` tag, which, as we [explained earlier](#pull-api-based-metrics-from-consul), can have either the value `leader` or `follower` depending on a server's state. Most of these metrics are available after enabling catalog-based checks, though you can also use the metric `consul.peers` to count the number of servers in the cluster by leader or follower status.

## Start monitoring Consul with Datadog

In this post, we've shown you how to use Datadog to get comprehensive visibility into your Consul clusters, regardless of how many hosts you're managing with Consul and how often they change. Datadog allows you to collect the output of Consul's built-in monitoring capabilities—telemetry, logs, and health checks—visualize them in a single platform, and alert on them to discover issues. If you're not already using Datadog, you can sign up for a <!-- Sign-up trigger (free trial) omitted --> to start monitoring your Consul clusters—and all of the services that rely on them—in one platform.