--- title: "Monitor AWS Lambda OpenTelemetry traces with Datadog" description: "Learn how you can use the AWS managed Lambda Layer for OpenTelemetry and Datadog's X-Ray integration to collect OpenTelemetry traces from your serverless applications." author: "Alex Cuoci, Kai Xin Tai" date: 2021-04-29 tags: ["serverless monitoring", "apm", "aws", "aws lambda", "aws x-ray", "opentelemetry", "lambda"] blog_type_id: the-monitor locale: en --- Datadog's support of [OpenTelemetry](https://opentelemetry.io/)—a vendor-agnostic, open source set of APIs and libraries for collecting system and application telemetry data—has helped thousands of organizations implement monitoring strategies that complement their existing workflows. Many of our customers leverage OpenTelemetry for their server- and container-based deployments, but also need visibility into the health and performance of their serverless applications running on AWS Lambda. Today, we're excited to partner with AWS for the launch of their [managed Lambda Layer for OpenTelemetry](https://aws.amazon.com/blogs/opensource/aws-distro-for-opentelemetry-adds-lambda-layers-for-more-languages-and-collector/), which extends OpenTelemetry tracing to Lambda. Developed as a Lambda Extension, the Layer contains the OpenTelemetry SDK (for [Python](https://github.com/open-telemetry/opentelemetry-python), [JavaScript](https://github.com/open-telemetry/opentelemetry-js), or [Java](https://github.com/open-telemetry/opentelemetry-java)) as well as the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector)—and lets you easily send traces to Datadog for monitoring and analysis. ## AWS managed Lambda Layer for OpenTelemetry as a Lambda Extension [Lambda Extensions](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-extensions-in-preview/), released as a preview late last year, offers a new and seamless way for monitoring and security tools to plug into the Lambda execution environment. Before Lambda Extensions, third-party tools could not integrate easily with Lambda without adding latency or cost. The [AWS managed Lambda Layer for OpenTelemetry](https://aws-otel.github.io/docs/getting-started/lambda) (built as a Lambda Extension) lets you collect OpenTelemetry traces from your Python, Node.js, and Java Lambda functions in the background, when they are not actively responding to events. The Layer contains the OpenTelemetry SDK, which generates traces, and the OpenTelemetry Collector, which sends those traces to [AWS X-Ray](https://aws.amazon.com/xray/) via the OpenTelemetry Collector X-Ray Exporter. AWS has carefully developed and tested the Layer so your teams don't have to worry about maintaining it or optimizing its performance—and can focus on building out your application. In the next section, we'll walk through how to instrument your Python functions with the managed Lambda Layer. If you're looking to instrument your Node.js or Java functions, see the [official AWS documentation](https://aws-otel.github.io/docs/getting-started/lambda#getting-started-with-aws-lambda-layers). ## Trace functions in Datadog with the AWS managed Lambda Layer for OpenTelemetry First, add the managed Lambda Layer for OpenTelemetry to your Python function (`arn:aws:lambda::901920570463:layer:aws-otel-python38-ver-1-1-0:1`) via the [AWS Management Console](https://console.aws.amazon.com/), the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html), or an infrastructure-as-code tool such as the [Serverless Framework](https://www.serverless.com/framework/docs/getting-started/) or [AWS Serverless Application Model (SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started.html). You can use this [sample function](https://github.com/aws-observability/aws-otel-lambda/tree/main/sample-apps/sam/python), if you don't already have one set up. Next, add the environment variable `AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument` to trace your Lambda function without any application code changes, and enable [X-Ray Active Tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) on the function. Then, install our [AWS 1-click integration](https://www.datadoghq.com/blog/aws-1-click-integration.md) to forward traces to Datadog. Once you've enabled our integration, you should begin to see OpenTelemetry traces from your Lambda function flowing into your Datadog account. X-Ray automatically stitches together each OpenTelemetry trace with X-Ray segments from the other AWS components that were involved in the request (e.g., Amazon API Gateway), so you can easily pinpoint bottlenecks. ![View OpenTelemetry traces of your Lambda applications in Datadog](https://web-assets.dd-static.net/42588/1776291739-aws-opentelemetry-lambda-layer-datadog-opentelemetry-lambda-trace.png) ## View traces in context with enhanced metrics and logs Traces are useful for troubleshooting an issue, but viewing them alongside Lambda metrics and logs gives you the context you need to identify the root cause as quickly as possible. To generate and forward enhanced Lambda metrics and logs to Datadog, you'll need to install and configure two additional Lambda Layers: - [Datadog Lambda Library](https://docs.datadoghq.com/serverless/datadog_lambda_library/python.md) (`arn:aws:lambda::464622532012:layer:Datadog-Python38:35`) - [Datadog Lambda Extension](https://docs.datadoghq.com/serverless/datadog_lambda_library/extension.md#setup) (`arn:aws:lambda::464622532012:layer:Datadog-Extension:6`) Next, add the following environment variables: - `DD_LOGS_ENABLED: true` - `DD_TRACE_ENABLED: false` - `DD_API_KEY: ` - `DD_LAMBDA_HANDLER: lambda_function.lambda_handler` You can find your Datadog API key [here](https://app.datadoghq.com/account/settings#api). If you need to create one, see our [documentation](https://docs.datadoghq.com/account_management/api-app-keys.md#add-an-api-key-or-client-token) for instructions. Lastly, modify your Lambda function handler to `datadog_lambda.handler.handler`. Once you've completed these steps, you'll be able to visualize key Lambda metrics (e.g., errors, timeouts, memory usage) together with OpenTelemetry traces and Lambda logs in our [Serverless View](https://app.datadoghq.com/functions). In addition, our enhanced metrics contain additional function metadata—and are collected at a higher granularity and lower latency than standard CloudWatch metrics—so you can investigate issues as soon as they crop up. ![Datadog’s Serverless view unifies your Lambda metrics, traces, and logs, giving you all the context you need to troubleshoot issues](https://web-assets.dd-static.net/42588/1776291743-aws-opentelemetry-lambda-layer-datadog-datadog-serverless-view.png) ## Native tracing with Datadog APM Datadog customers can choose to trace their Lambda functions with either OpenTelemetry and X-Ray or our native tracing libraries (`dd-trace`). Datadog’s tracing libraries automatically propagate trace context across service boundaries, so you get end-to-end visibility into every request, regardless of whether it flows through functions, containers, or other infrastructure components. These traces appear in your account in real time, and no Lambda function code changes are required to begin collecting them. If you'd like to learn more about Datadog Serverless monitoring and our native tracing capabilities, see our dedicated [blog post](https://www.datadoghq.com/blog/tracing-lambda-datadog-apm.md). ## Start monitoring AWS Lambda OpenTelemetry traces today At Datadog, we're constantly working to deliver comprehensive visibility into your serverless environment in a way that minimizes cost, operational complexity, and performance impact. The AWS managed Lambda Layer for OpenTelemetry, together with our AWS X-Ray integration, enables you to collect traces from your Lambda functions without any changes to your function code. To learn more, visit the AWS Distro for OpenTelemetry [developer portal](https://aws-otel.github.io/). And if you're new to Datadog, sign up for a 14-day today.