--- title: "Introducing DogStatsD Mapper" description: "Use DogStatsD Mapper to generate tags for all of your StatsD-formatted metrics." author: "Paul Gottschling" date: 2020-02-18 tags: ["datadog agent", "statsd", "dogstatsd mapper", "tags"] blog_type_id: the-monitor locale: en --- Since StatsD was [introduced](https://codeascraft.com/2011/02/15/measure-anything-measure-everything/) by Etsy in 2011, it has become a mainstay of infrastructure monitoring. But as dev and ops teams rely [more and more](https://www.datadoghq.com/container-report-2020/) on containerized microservices, they have [pushed the limits](https://engineering.nanit.com/from-graphite-to-prometheus-things-ive-learned-e1d1e4b97fc) of StatsD's design. One major shortcoming is that StatsD has no built-in support for tagging your metrics with key-value pairs. If you want to [get context](https://www.datadoghq.com/blog/tagging-best-practices.md) into different slices of your infrastructure but are using software that can only generate metrics in StatsD format, you'll need some way to process your StatsD metrics into tags. To make this task quicker and easier, Datadog has introduced [DogStatsD Mapper](https://docs.datadoghq.com/developers/dogstatsd/dogstatsd_mapper.md), which automatically generates tags for StatsD metrics based on user-configured rules. By using DogStatsD Mapper, you can convert parts of your StatsD metric names into tags without having to re-instrument your code, and without having to run a separate server (like [statsd exporter](https://github.com/prometheus/statsd_exporter)) to transform your metrics. ## StatsD and tags In a typical StatsD setup, application code is [instrumented](https://www.datadoghq.com/blog/statsd.md) to send metrics as single-line [datagrams](https://thenewstack.io/collecting-metrics-using-statsd-a-standard-for-real-time-monitoring/#crayon-5e3866095f851449088971) to a StatsD server. StatsD then flushes these metrics periodically to a [storage backend](https://github.com/statsd/statsd/blob/d47bf4dee0c07e44bb4fff608e223dfc071ba462/docs/backend.md). The names of StatsD metrics are [designed](https://github.com/statsd/statsd/tree/d47bf4dee0c07e44bb4fff608e223dfc071ba462#key-concepts) to work with [Graphite](https://graphiteapp.org/)'s hierarchical [directory structure](https://graphiteapp.org/quick-start-guides/feeding-metrics.html), and take the form of [dot-delimited namespaces](https://github.com/statsd/statsd/blob/d47bf4dee0c07e44bb4fff608e223dfc071ba462/docs/namespacing.md). Since there are [no keys, only values](https://prometheus.io/docs/introduction/comparison/#data-model), the only way to group or filter metrics by different combinations of metadata is to use a dedicated tag processing server such as Graphite's [Carbon](https://graphite.readthedocs.io/en/latest/tags.html). ![statsd2](https://web-assets.dd-static.net/42588/1776299178-dogstatsd-mapper-statsd2.png) Some tools generate metrics only as StatsD datagrams. Apache Airflow, for example, [emits metrics](https://airflow.apache.org/docs/stable/metrics.html) using Python's [StatsD library](https://pypi.org/project/statsd/). If parts of your infrastructure are instrumented to emit metrics in StatsD format, and you want to group and filter your metrics by tag, you'll need a way to generate tags downstream from your StatsD client. The airflow-exporter plugin for Airflow attempts something like this, but exposes only [two Airflow metrics](https://github.com/epoch8/airflow-exporter). ## How DogStatsD Mapper works The Datadog Agent can ingest StatsD metrics as well as metrics in the DogStatsD format, which [extends StatsD](https://docs.datadoghq.com/developers/dogstatsd.md?tab=python) to include tags. DogStatsD Mapper runs as part of the Agent's built-in DogStatsD server. When the Agent receives an untagged metric in StatsD format, it [uses rules](https://docs.datadoghq.com/developers/dogstatsd/dogstatsd_mapper.md) to map segments of the metric's namespace to a list of tags. It then gives the metric a new name and [processes it](https://github.com/DataDog/datadog-agent/blob/71c7c478f1c55977e2102ccc69a38e79bc9a23f6/pkg/aggregator/README.md) like any other tagged metric. (If you're curious as to how this works [behind the scenes](https://github.com/DataDog/datadog-agent/blob/13e54f633dd9cddc5bbe83c63005fedc40d38f5c/pkg/dogstatsd/server.go#L323-L340), the Agent is open source.) ![mapper-rule2](https://web-assets.dd-static.net/42588/1776299183-dogstatsd-mapper-mapper-rule2.png) ## Tag any StatsD metric To create a rule, specify the namespace of a StatsD metric, replacing all or part of each namespace segment with wildcards or regular expressions. Then indicate the name of the metric as it will appear within Datadog and where in the metric's namespace DogStatsD Mapper will look for tags. If DogStatsD Mapper detects that the namespace of a StatsD metric matches a rule, it will extract tags and rename the metric. For example, you can generate tags for your Airflow StatsD metrics by copying rules we provide in our [documentation](https://docs.datadoghq.com/integrations/airflow.md) and pasting them into your Datadog Agent [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md?tab=agentv6v7): *datadog.yaml* ```yaml dogstatsd_mapper_profiles: - name: airflow prefix: "airflow." mappings: # [...] - match: "airflow.dag.*.*.duration" name: "airflow.dag.task.duration" tags: dag_id: "$1" task_id: "$2" # [...] ``` Using this configuration, DogStatsD Mapper will transform any StatsD metric matching the pattern `airflow.dag.*.*.duration` into a metric called `airflow.dag.task.duration` with the `dag_id` tag set to the name of the Airflow [DAG](https://airflow.apache.org/docs/stable/concepts.html#dags) and the `task_id` tag set to the name of the Airflow [task](https://airflow.apache.org/docs/stable/concepts.html#tasks). ![airflow2](https://web-assets.dd-static.net/42588/1776299187-dogstatsd-mapper-airflow2.png) ## Start mapping your StatsD metrics If you're a Datadog customer, you can start defining mapping rules right away to extract tags from your StatsD metrics and get insight into different slices of your environment. With DogStatsD Mapper built into the Datadog Agent (version 7.17+), you can enjoy the full power of Datadog tagging with your own StatsD-instrumented legacy code as well as with infrastructure tools written with StatsD in mind (e.g., Airflow). If you're new to Datadog, sign up for a 14-day .