Modern continuous integration (CI) practices enable development teams to quickly and efficiently build and deploy application code to a shared codebase. However, deploying new code is typically accompanied by tests, and as the codebase expands, this results in a proportionately larger test suite. When an entire test suite needs to run in order to validate each small code change, testing can feel more like a chore rather than an integral part of the development process, creating a cumbersome feedback cycle for developers.
Datadog Testing Visibility gives you deep insights into each commit and automatically detects flaky tests that are compromising your build. Now, we’re excited to announce Datadog Intelligent Test Runner (ITR), a new feature for CI Visibility that helps you accelerate the testing process by identifying and running only the relevant tests for each commit. By enabling ITR for your test services, you can reduce both testing time and resources required, and minimize the risk of broken pipelines caused by irrelevant flaky tests.
In this post we’ll cover how to:
- Create faster testing feedback cycles with Datadog ITR
- Visualize resource savings with the ITR dashboard
Create faster feedback cycles with intelligent test selection
Large codebases typically require a proportionately large test suite, which equates to lengthy testing times for each code change regardless of how big or small the change is. This can create a slow feedback cycle for developers, who spend valuable development time waiting since they usually need to monitor testing until the pipeline is complete.
Datadog’s Intelligent Test Runner (ITR) automatically selects and runs only the relevant tests needed to validate any given commit, enabling you to reduce the amount of time spent testing while maintaining test coverage. ITR provides test impact analysis by analyzing your test suite to determine the code each test covers, and then cross-referencing that coverage with the files impacted by a new code change. Datadog uses this information to run a selection of relevant, impacted tests, omitting the ones unaffected by the code change and reducing the overall testing duration.
By simply selecting the test services you’d like to enable ITR for in the CI Test Service Settings, Datadog will apply intelligent test selection to that service’s commits. You can also choose which branches you’d like to exclude from ITR within a test service. Running every test on your default branch is still recommended as a failsafe practice—as a part of this recommendation, your default branch is preset to be excluded from ITR, however, you can still configure it to be included.
Minimize disruptions due to flaky tests
By minimizing the number of tests run per commit, Datadog Intelligent Test Runner can help reduce the frequency of flaky tests disrupting your pipelines. Flaky tests are tests that may pass or fail at random given the same commit. This can be particularly frustrating when the test flaking is unrelated to the code change being tested. After enabling Intelligent Test Runner for your test services, you can limit each commit to its relevant tests to ensure that flaky tests unrelated to your code change don’t end up arbitrarily breaking your build. By monitoring your commits, Datadog CI Visibility will automatically identify any flaky tests affecting your CI/CD workflow.
Get visibility into resource savings and slow tests
Datadog measures your Intelligent Test Runner performance against baseline testing times to calculate the time saved from unnecessary testing. ITR enables developers to more frequently integrate new application code by reducing the time spent testing. Once you enable ITR for your test services, you can begin visualizing these resource savings in the ITR dashboard. The dashboard breaks down the time saved across your services, repositories, commits, and authors so you can better understand where ITR is the most effective.
You can also view your time saved with ITR against your testing time within the Tests View page or within a test session’s details.
Since branches with ITR enabled will only run relevant tests, ITR can greatly reduce the total number of tests run per commit. By drilling down into a specific commit with ITR enabled, you gain a concentrated view into the tests impacted by your code change, enabling you to more quickly surface a relevant failed or slow test affecting your build.
Start accelerating your software delivery process today
Using Datadog’s Intelligent Test Runner, you can begin to create faster developer feedback cycles and reduce the risk of your pipelines failing due to unrelated flaking tests. Since ITR is built into Datadog CI Visibility, you can reap its time-saving benefits while monitoring your pipelines and tests from the same platform. ITR is now generally available to all Datadog customers—you can learn more in our documentation. For more information about troubleshooting your pipelines and tests with CI Visibility, check out our blog post.
If you aren’t already a Datadog customer, you can sign for up for free 14-day trial to begin monitoring your application today.