Engineers who support production environments are tasked with resolving new issues as quickly and efficiently as possible. But as they look to carry out these responsibilities, their remediation workflows tend to take on the following pattern:
- They discover and investigate the problem in an observability tool.
- They leave the observability tool to take whichever steps are needed to solve the problem.
- They return to the observability tool to continue to monitor the incident and ensure that it’s been resolved.
For example, someone on your team might discover in a log analysis tool that a user is flooding a key service by making an abnormal number of requests. To remediate the issue, they switch over to a terminal to connect to their user database or backend infrastructure and limit the user’s actions. Then, they have to go back to the log analysis tool to confirm that the request count from that user has dropped.
This kind of back-and-forth workflow can be both time-consuming and error prone. Time is lost as engineers leave their observability tool to perform remediation steps. And as they turn away from the source of monitoring data, they become more liable to make mistakes, such as accidentally blocking the wrong user ID or IP address.
With Datadog Apps, you can now bring external functionality from your other services directly into your Datadog dashboards, shortening these workflows and helping ensure that you can take remediation steps more quickly and accurately. You can create custom widgets for these apps—with side panels, modals, and other components—all with the goal of unifying your monitoring workflows and simplifying your application management.
In this post, we’ll walk through how to create a Datadog App that enables you to add several of these components to a dashboard. Then, we’ll look at how you can use this app to take action against potential bad actors abusing your web application.
Get started with the Stream application
Throughout this post, we will work with the flagship application of a fictional company, Stream. Stream is a basic Twitter-like webapp, offering a newsfeed and a way for users to post new content streams.
In this example scenario, you work for Stream as an administrator, and you’ve noticed a problem after Datadog alerted you to an anomalous spike in traffic on an API. Bad actors have been violating Stream’s terms of service (TOS) by scraping data and flooding the application’s APIs. Your engineering team has created a Datadog app, Stream Admin, that enables you to respond to this type of problem from directly within your Datadog dashboards.
We’ll look at how to deploy the Stream Admin app to start dealing with this potential issue directly from within Datadog. But first, you can spin up a containerized version of our example app so you can follow along. Note that you will need Docker and a Datadog account with an API key and an application key. If you aren’t already a Datadog customer, you can sign up for a free trial here.
You can begin the process of deploying the Stream Admin app by cloning this GitHub repository. After you complete this step, navigate to the examples/stream folder and follow the remaining instructions in the README file. Completing these instructions will spin up Docker containers running the services we need for our Stream app (the backend Stream API, Stream UI, and the Datadog app).
Launch the project using the command docker-compose up, and then open your browser and go to http://localhost:3000 to see the Stream application running. The API runs on port 3001, and the Datadog app on port 3002.
In addition to starting the three locally running services, the installation script also creates a dashboard in Datadog that you can use to monitor and administer your app. Next, log in to your Datadog account and open the Stream submissions dashboard.
This dashboard includes two timeseries graph widgets. The first tracks GET requests to your app’s API. You can see in the screenshot above that one user, lucious14, is creating a line notably higher than the others, suggesting this user is a bot set up to scrape the Stream service. The second timeseries widget visualizes incoming submissions. Here again, one user makes significantly more submissions than others: margaret.
Each widget therefore reveals an individual user who is violating Stream’s TOS. Allowing this type of activity could lead to Stream user data being mined or to the Stream API being flooded with requests from bad actors. To stop the bad behavior and block the offending users as a Stream admin, you previously needed to swivel-chair from Datadog to an internal tool.
The installation script we executed above added the Stream Admin app to the Datadog Developer Platform, as well as the context menus to the timeseries widgets in the dashboard. Next, you’ll get the Stream Admin widget ready to add to the dashboard so you can take further administrative action directly from the same dashboard where you’ve pinpointed and identified the issue.
Configuring your Datadog App
Before you can add your app to the dashboard, you need to configure it in Datadog. To review or configure your new app’s settings, begin by navigating to the Developer Platform on the menu bar under Integrations, and check the In Development checkbox. You should see the Stream Admin application.
Click on the menu button at the top right of the app tile, and then click Edit. This opens the Basic Information tab of the Edit App page, shown below:
The Basic Information tab includes general information related to your application, such as its name, description, and icon.
Navigating to the UI Extensions tab, shown below, you can configure Datadog to communicate with your app and define what UI elements your app can add to the dashboard.
Some of the most important settings on this tab are located in the App Settings section. Each app you develop will have its own settings, allowing you to specify:
- the URL and port where your Datadog app is hosted. In our example, our app is running locally, so we’ll point our root URL to
http://localhost:3002. - the Main controller path value, which sets the path where your app is hosted. In the case of the Stream Admin app, it is located in the root of the URL.
The Features section allows you to enable any elements that your app includes. In the case of the Stream Admin app, it includes a custom dashboard widget, a widget context menu, a modal, and a side panel. We’ll look at these in more detail later.
The right pane of the UI Extensions tab displays the app manifest, presented in a JSON format, which defines each of the elements associated with the app. By editing these values, you can configure settings such as a widget’s source (the URL corresponding to the widget), the dashboard widget’s name, the app icon used in the widget tray, and various other options.
Enhance your dashboard with custom widgets
Now that you’ve configured the app, you can begin to integrate its functionality into Datadog. For example, you can add Stream Admin as a custom widget to any of your dashboards, just as you would add a timeseries widget or a query value widget. These custom widgets can take action against your in-house APIs, or provide you with specialized visualizations that aren’t supported natively within Datadog today, such as a box-and-whisker chart, or a specialized map to interact with IoT devices.
Once you’ve added the Stream Admin app’s custom dashboard widget, you can perform the following tasks directly from your dashboard:
- Limit the traffic on the site by lowering the maximum rate of HTTP requests accepted from all users.
- Open a modal window to block or unblock a user.
For example, by implementing a rate limit, you can see an immediate result in our timeseries widget as the user lucious14 is no longer able to flood your API with GET requests.
You can find the widget code in the file stream/admin/src/widget/index.tsx. Note that it is a React app with TypeScript.
Display contextual information with side panels
In addition to the Stream Admin widget, the app also provides a custom side panel. Side panels slide from the right edge of the display window and can provide additional contextual information or more advanced user interactions and flows in your Datadog App.
In Stream Admin, the side panel allows you to check the submissions history for individual users, such as margaret, and determine if the submissions are spam. You can access it using a context menu item from either of the timeseries visualizations on the Stream submissions dashboard.
To see how this side panel was built, you can review the code in the file stream/admin/src/account-panel/index.tsx. To learn how to add a side panel to the context menu, you can review the code found in the source file stream/admin/src/controller/widget-cxt-menu.ts.
Use modals to perform complex actions
Now that you know that submissions from our user margaret are spam, you need to remediate the issue. Modals enable you to perform and confirm actions and make configuration changes based on specific input. In the Stream Admin app, for example, a modal is used to help you block and unblock users.
Once you add margaret@ecorp.com to the block list, you can see an immediate drop in incoming submissions.
The modal code can be found in the file stream/admin/src/blocklist-modal/index.tsx.
Discover sample applications and build your first Datadog app
In this post, we went over the main components that make up a Datadog App and how they can add powerful functionality to your dashboards, letting you remediate problems and monitor the results to confirm that your changes worked. If you’re already using Datadog, you can see apps developed by our partners, including Embrace, Fairwinds, Harness, Rookout, Shoreline, LaunchDarkly, and PagerDuty. Then, take a look at our GitHub repository to find more sample apps and learn more about the programming model. To get started building your app today, join the beta to access the Developer Platform. Or, if you’re not a customer, sign up for a 14-day free trial.
