New: Watchdog / Trace Search / Limitless Logs
Monitor AWS Fargate applications with Datadog

Monitor AWS Fargate applications with Datadog

/ / / /
Published: March 30, 2018

AWS Fargate is a new way of running applications in Amazon Elastic Container Service without having to manage the underlying infrastructure. With Fargate, you can define containerized tasks, specify the CPU and memory requirements, and launch your applications without spinning up EC2 instances or manually managing a cluster.

Datadog is proud to have been a launch partner for Fargate, and we have continued to collaborate with AWS on best practices for managing serverless container tasks. Now we are pleased to announce that monitoring your AWS Fargate tasks is even easier with the availability of a new metadata endpoint for Fargate. Datadog’s integration with version 1.1.0 of the Fargate platform enables you to collect real-time, high-resolution metrics from all your containerized tasks. It also enables Autodiscovery to detect containerized services running on Fargate and automatically configure Datadog Agent checks for those services. Additionally, the integration provides more than a dozen system metrics from Fargate that track the resource utilization (CPU, memory, I/O, etc.) of your tasks.

In this post, we’ll show a simple example of how you can deploy and monitor tasks on AWS Fargate with the Datadog Agent. Note that you will need version 6.1.1 or higher of the Datadog Agent to take full advantage of the Fargate integration.

Deploying the Datadog Agent on AWS Fargate

To start monitoring AWS Fargate tasks and services, you can deploy the containerized Datadog Agent on Fargate. In addition to passing the usual DD_API_KEY environment variable, you must set the ECS_FARGATE environment variable to true.

The example task definition below deploys the Datadog Agent to Fargate, along with a Redis container in the same task.

{
        "family": "redis-datadog",
        "networkMode": "awsvpc",
        "containerDefinitions": [
            {
                "name": "redis",
                "image": "redis:latest",
                "essential": true,
                "dockerLabels": {
                    "com.datadoghq.ad.instances": "[{\"host\": \"%%host%%\", \"port\": 6379}]",
                    "com.datadoghq.ad.check_names": "[\"redisdb\"]",
                    "com.datadoghq.ad.init_configs": "[{}]"
                }
            }, {
                "name": "datadog-agent",
                "image": "datadog/agent:latest",
                "essential": true,
                "environment": [
                    {
                        "name": "DD_API_KEY",
                        "value": "$YOUR_API_KEY"
                    },
                    {
                        "name": "ECS_FARGATE",
                        "value": "true"
                    }
                ]
           }
        ],
        "requiresCompatibilities": [
            "FARGATE"
        ],
        "cpu": "256",
        "memory": "512"
}

After saving that task definition locally as redis-datadog.json, you can register the task with ECS using the AWS CLI:

aws ecs register-task-definition --cli-input-json file://./redis-datadog.json

Once the task has been registered, you can run it in a Fargate cluster, making sure to provide one or more private subnets and security groups for your task. Note that Fargate version 1.1.0 or greater is required, so we specify the platform version in the command:

# create a cluster if you don’t already have one
aws ecs create-cluster --cluster-name "fargate-test"

aws ecs run-task --cluster fargate-test \
--network-configuration "awsvpcConfiguration={subnets=["$PRIVATE_SUBNET"],securityGroups=["$SECURITY_GROUP"]}" \
--task-definition arn:aws:ecs:us-east-1:$AWS_ACCOUNT_NUMBER:task-definition/redis-datadog:1 \
--region us-east-1 --launch-type FARGATE --platform-version 1.1.0

You can check the status of your task in the ECS console. In just a few moments, your task should be up and running, and your Fargate containers will appear on Datadog’s Containers page, where you can search, filter, and view high-resolution system metrics from all the running containers in your infrastructure.

Viewing high-resolution metrics from AWS Fargate containers in Datadog’s Live Container view.

Monitoring AWS Fargate containers with Autodiscovery

In the dockerLabels section of the task definition above, we applied three labels to the Redis container: this is the metadata that the Datadog Agent needs to auto-detect Redis and start collecting metrics from it. The labels specify the name of the Datadog Agent check to apply (redisdb), an empty list of init_configs, and the host and port where Redis will run. Note that the %%host%% template variable will dynamically configure the Agent to collect metrics from the Redis container, wherever it is running, but the %%port%% template variable is not supported in ECS, so we manually provide the default Redis port, 6379.

With Autodiscovery properly configured, our newly deployed Datadog Agent starts collecting metrics from the Redis container automatically.

Viewing Redis metrics from a container detected and monitored using Datadog Autodiscovery.

Container monitoring, minus the host

With the launch of AWS Fargate, the shift from focusing on hosts and other underlying infrastructure to focusing on services and applications takes another leap forward. Just as with AWS Lambda, Datadog provides full visibility into the health and performance of all your applications, wherever they run. If you’re already using Datadog to monitor your applications and infrastructure, you can start monitoring Fargate tasks today. And if you’re not yet using Datadog, you can sign up for a free trial .


Want to write articles like this one? Our team is hiring!