Streamline Your CI Testing With Datadog Intelligent Test Runner | Datadog

Streamline your CI testing with Datadog Intelligent Test Runner

Author Bowen Chen

Published: October 19, 2022

Modern continuous integration (CI) practices enable development teams to quickly and efficiently build and deploy application code to a shared codebase. However, deploying new code is typically accompanied by tests, and as the codebase expands, this results in a proportionately larger test suite. When an entire test suite needs to run in order to validate each small code change, testing can feel more like a chore rather than an integral part of the development process, creating a cumbersome feedback cycle for developers.

Last year, Datadog introduced our Testing Visibility product as a part of Datadog CI Visibility. Testing Visibility gives you deep insights into each commit and automatically detects flaky tests compromising your build. Now, we’re excited to announce Datadog Intelligent Test Runner (ITR), a new feature for CI Visibility that helps you accelerate the testing process by identifying and running only the relevant tests for each commit. By enabling ITR for your test services, you can reduce both testing time and resources required, and minimize the risk of broken pipelines caused by irrelevant flaky tests.

In this post we’ll cover how to:

Create faster feedback cycles with intelligent test selection

Large codebases typically require a proportionately large test suite, which equates to lengthy testing wall times for each code change regardless of how big or small the change is. This can create a slow feedback cycle for developers, who spend valuable development time waiting since they usually need to monitor testing until the pipeline is complete.

Datadog’s Intelligent Test Runner (ITR) automatically selects and runs only the relevant tests needed to validate any given commit, enabling you to reduce testing downtime while maintaining test coverage. ITR provides test impact analysis by analyzing your test suite to determine the code each test covers, and then cross-referencing that coverage with the files impacted by a new code change. Datadog uses this information to run a selection of relevant, impacted tests, omitting the ones unaffected by the code change and reducing the overall testing duration.

By simply selecting the test services you’d like to enable ITR for in the CI Test Service Settings, Datadog will apply intelligent test selection to that service’s commits. You can also choose which branches you’d like to exclude from ITR within a test service. Running every test on your default branch is still recommended as a failsafe practice—as a part of this recommendation, your default branch is preset to be excluded from ITR, however, you can still configure it to be included.

Enable ITR for your test services.

Minimize disruptions due to flaky tests

By minimizing the number of tests run per commit, Datadog Intelligent Test Runner can help reduce the frequency of flaky tests disrupting your pipelines. Flaky tests are tests that may pass or fail at random given the same commit. This can be particularly frustrating when the test flaking is unrelated to the code change being tested. After enabling Intelligent Test Runner for your test services, you can limit each commit to its relevant tests to ensure that flaky tests unrelated to your code change don’t end up arbitrarily breaking your build. By monitoring your commits, Datadog CI Visibility will automatically identify any flaky tests affecting your CI/CD workflow.

Automatically detect flaky tests with Datadog CI Visibility

Get visibility into resource savings and slow tests

Datadog measures your Intelligent Test Runner performance against baseline testing wall times to calculate the time saved from unnecessary testing. ITR enables developers to more frequently integrate new application code by reducing the time spent testing. Once you enable ITR for a branch, you can begin visualizing these resource savings within the Tests View page or within a test service’s commit summary.

Visualize your ITR savings within the Tests view page

Since branches with ITR enabled will only run relevant tests, ITR can greatly reduce the total number of tests run per commit. By drilling down into a specific commit with ITR enabled, you gain a concentrated view into the tests impacted by your code change, enabling you to more quickly surface a relevant failed or slow test affecting your build.

Drill into slow and flaky tests with Datadog.

Start accelerating your software delivery process today

Using Datadog’s Intelligent Test Runner, you can begin to create faster developer feedback cycles and reduce the risk of your pipelines failing due to unrelated flaking tests. Since ITR is built into Datadog CI Visibility, you can reap its time-saving benefits while monitoring your pipelines and tests from the same platform. To learn more about troubleshooting your pipelines and tests with CI Visibility, read our blog post.

If you’re a Datadog customer, you can sign up for the limited-access beta today. Otherwise, sign for up for to begin monitoring your application today.