Mark Avery
Bowen Chen
Backstage is a popular open source framework for building internal developer portals (IDPs) used by organizations to aggregate service metadata and create a single source of truth for their software developers. However, data stored in the Backstage Software Catalog can quickly become siloed and inaccessible from monitoring tools such as Datadog. As a result, developers often spend excess time context switching between platforms and manually correlating telemetry signals with service metadata, which can be problematic during urgent situations such as incident response.
To help address these pain points and improve the developer experience, we now offer the Datadog Plugin for Backstage. Originally developed by engineers at Cvent to support their own usage of Datadog and Backstage, this plugin enables you to unify service metadata and other telemetry by making your Backstage Software Catalog data available within the Datadog Software Catalog, a component of Datadog Internal Developer Portal (IDP).
In this blog post, we’ll discuss how IDPs help create a single source of truth for service metadata, the importance of making this data available in your observability platform, and how you can use the Backstage plugin to enrich observability and enhance investigation workflows in Datadog.
How Cvent brought together their IDP and observability platform
Engineering organizations rely on a variety of tools for different tasks within the development life cycle. For example, you may use GitHub for CI/CD, PagerDuty for incident response, and Slack for communication. As a result, service information often becomes disjointed, as it’s spread across third-party SaaS providers, internal tools, YAML configurations, wikis, and other resources that your engineers rely on. For example, let’s say you get paged to respond to an incident, but upon investigating which service is affected, you discover that you were notified through a PagerDuty escalation policy, which covers a group of services. This can lead developers to waste time trying to find the correct communication channels, dashboards, and service knowledge.
To improve workflows and increase velocity, many organizations, including our friends at Cvent, implement an IDP as a single source of engineering truth designed to combine data related to authentication, authorization, teams, and services from various systems of record into a comprehensive catalog. After implementing a purpose-built, internal IDP, Cvent eventually adopted Backstage as an easier method for managing a growing number of internal functions their developers rely on. Still, they needed to be able to push this information to downstream consumers such as Datadog, which Cvent was using as their observability platform to maintain visibility into system health and performance.
To address this gap, Cvent engineers developed the Datadog Plugin for Backstage to keep entities stored in Backstage in sync with the Datadog Software Catalog. This enables developers to correlate telemetry signals with complete and up-to-date service metadata, speeding up incident response and allowing for more accurate monitoring of service health.
Enrich Datadog observability with Backstage metadata
To make these capabilities available to the broader community of Datadog users who also use Backstage as their IDP, Datadog now maintains the Backstage plugin Cvent recently open sourced. Once you install the Datadog Plugin in your Backstage backend, the plugin will begin mapping your Backstage entities into the definition schemas supported by Datadog Software Catalog entities.
The plugin gives you more granular control over your entities as opposed to reading and importing data directly from your catalog files. If you use related entities such as the owning team or the APIs the service provides, you’re able to collapse all of the related entity data from your Backstage catalog into a single Datadog Software Catalog entity. In cases where your service names in Datadog are inconsistent with Backstage or other systems of record, you can use the plugin to override the service name in Datadog using an annotation while still ensuring that your data is synced. You might also use different entity processors to update Backstage entities without having to make changes to your catalog files. Through the lifecycle of an entity, you can make sure that data such as PagerDuty Service IDs is consistent and accurate without having to manually manage it across your catalog files. The plugin helps you overcome this challenge, since it has access to all the final entities made available in Backstage after they are fully processed.
The goal of using a centralized software catalog is to establish a single source of truth to alleviate the concern of metadata duplication and the additional upkeep required to maintain it across multiple platforms. Following your initial configuration, our Backstage Plugin will automatically sync changes in your Backstage software catalog to reflect in Datadog Software Catalog. Your team can maintain existing workflows and continue to use Backstage as your single source of truth, and the plugin will ensure that this information stays up to date within the Datadog platform.
Once your Backstage data is integrated into Datadog IDP, Datadog will automatically detect live traffic flow between your system components, enabling you to visualize your Backstage services’ upstream and downstream dependencies alongside a hierarchy chart of declared relationships. When an engineer gets paged from a Datadog Metrics monitor that alerts on a core service, they can pivot directly from the triggered metric to the service page in Datadog Software Catalog. Using Backstage data made available in the service page, they can quickly identify key facets such as the owning team, links to the code repository, deployment pipeline, and software artifacts, as well as recent deployments and changes in code coverage.
Centralize observability and ownership with the Datadog IDP
The Datadog Plugin for Backstage enables you to correlate observability data with service metadata all within the Datadog platform. You can learn more about how the Datadog IDP unifies live telemetry, metadata, and self-service workflows to standardize and accelerate software delivery in our dedicated blog post. For help on getting started with the plugin, check out our source code repository, and read about other Datadog open source projects in our Open Source Hub. This project was largely made possible by Cvent—you can read more about how they use Datadog to support their high-throughput platform in our case study.
If you don’t already have a Datadog account, sign up for a free 14-day trial today.
