There are several different testing methods you can use as part of your development process to ensure you build high-quality applications. Shift-left testing is one approach that has become popular with agile teams because it enables them to move the testing phase to earlier stages of the development life cycle, which is a primary goal for agile development.
Shift-left testing has a few advantages over traditional methods:
- You can identify and resolve more bugs before they reach production and become too costly, time consuming, and risky to fix—bugs grow in complexity as features approach the end of their development cycle.
- You can automate more of your existing continuous integration and delivery (CI/CD) workflows and tests, which reduces the risk of human error and enables developers and testers to focus on building and testing new features.
- Your teams can share ownership of testing responsibilities. This encourages developers and testers to collaborate on a clear testing strategy for each release, and ensures that your testing infrastructure (e.g., test suites, test environments, servers) is capable of supporting new development.
These benefits enable you to limit the number of bugs introduced in production and make your development process more efficient overall, which can significantly improve the quality of your applications for end users and reduce the time to market for new features. But your teams may be hesitant to adopt a shift-left testing approach because it requires substantial changes to existing processes, such as reconfiguring CI/CD pipelines to run automated tests as part of the development process. These types of process changes require buy-in across teams, and knowing how to best implement them without adding overhead can be difficult.
There are a few best practices that can help your teams address some of the challenges of this approach and take advantage of testing in earlier stages of development. These practices include:
- enabling developers to test their code
- optimizing your test infrastructure
- creating builds that fail fast
- monitoring the performance of your CI/CD pipelines
In this post, we’ll look at these best practices in more detail and show a few ways Datadog supports each one, so you can efficiently implement shift-left testing into your development processes.
Enable developers to test their code
As you shift left, test automation becomes a more essential part of the development lifecycle because it enables you to reliably verify application functionality at any stage of development, such as a local build or a full-scale deployment to production.
There are some tests that should always run as part of your pipelines, making them better candidates for automation than others. These include tests for:
- individual modules of code (e.g., unit tests)
- the performance of API endpoints
- the performance of connected application services or modules
- business-critical or frequently used functionality, such as login workflows
- the cross-browser performance of key features
It’s important to ensure that the tests you decide to automate are easy to maintain, fast, and reliable enough to incorporate into CI/CD pipelines. Otherwise, your teams may be more hesitant to fully adopt the new process changes if they spend more time troubleshooting unstable pipelines caused by tests that take too long to run or that generate too many false positives.
Adding tests to CI/CD pipelines as part of a high-level strategy to test sooner and more often is only one aspect of shifting left. Another factor for shift-left testing is finding ways to merge testing with other workflows instead of treating it as an isolated activity. For example, with this approach, your developers can test the stability and performance of every module of code as they develop new features. This ensures that features are more thoroughly tested and more stable when you release them to customers.
There are a few ways you can add testing to development workflows, including leveraging unit tests—which test individual units of code—and static-code analysis tools to monitor code stability and performance. With these tools, you can catch code vulnerabilities before they become more serious security risks, ensure that you are creating code that is easy to maintain, and track changes to your code base over time. For example, you can monitor the cyclomatic complexity of each new version of code you deploy. This metric counts the number of independent paths a function creates via conditional statements.
Code with a lower complexity number is easier to test and maintain overall. If you see a sudden increase in this metric, you may need to refactor a recently deployed module to reduce complexity. You can also use this information with tools like deployment tracking to monitor performance data like request latency and the number of errors introduced with a new release. This enables you to see how the complexity of your code affects the performance and stability of your application at a high level.
Optimize test environments for efficient pipelines
Once you have identified which tests need to be added to your CI/CD pipelines, you can add them to the appropriate pipeline stages. For example, you may decide to use smaller, faster tests as smoke tests earlier in a pipeline to verify the stability of a new version of code, then run your larger test suites later in staging or other pre-production environments.
As you add more tests to additional stages of your deployment pipelines, you will need to ensure that the environments you test in are built to support rapid development. Many teams rely solely on long-running test environments, which—along with their underlying infrastructure, such as dedicated test servers—require additional maintenance to ensure they are always up to date with the latest versions of code and test data. Obsolete test data, for example, is a primary reason why permanent environments can quickly become unstable, which makes it more difficult to execute tests in your pipelines reliably.
You can mitigate these types of issues by using temporary environments to run certain tests in pipelines. This can improve the duration of your pipeline’s testing stages and make them more resilient in a fast-moving development cycle. For example, in the first test stage seen in the example diagram above, you can use Docker containers to package your smoke tests and the test environment together into a single container. Once the test has run, you can then destroy the container. Using temporary environments like these ensures that you are always testing in a consistent state where you need it the most, so you can spend less time troubleshooting issues caused by outdated or unstable environments.
Design your pipelines to fail fast and provide actionable feedback
As you run tests in more stages of the development lifecycle and potentially increase the frequency of finding application defects, you need the ability to immediately act when a pipeline job or test fails. Creating fail-fast mechanisms to stop a pipeline as soon as a build step or test encounters a failure gives you fast feedback for quickly resolving a problem before it affects other teams or downstream pipelines.
Some examples of fail-fast steps include immediately blocking a build or triggering automatic rollbacks for deployments by polling the results of Datadog API tests and reverting the associated commit that broke a build.
In addition to stopping a build or deployment as soon as an issue is found, these types of mechanisms need to also provide actionable feedback for addressing an issue. Without that, you may spend more time digging for relevant information about the build failure than fixing the problem. Ensuring that your pipeline’s CI outputs have enough details for resolving an issue, such as the results of API and browser tests, relevant build logs, and the build’s version and environment, allows you to resolve any failures as quickly as possible.
Monitor pipeline performance
Shift-left testing relies on automating key development and testing processes in order to help teams release high-quality services to customers. This means that it’s important to ensure that these processes are working optimally. Problems with your pipelines, such as consistently slow or failing jobs (the individual tasks executed within a pipeline) can disrupt development and risk your ability to release new features on time.
Monitoring your pipelines gives you better visibility into performance issues with your supporting infrastructure, so you can quickly find and resolve an issue at any stage of development. This also enables you to track the quality of your deployments over time and make gradual improvements to your processes where needed. Regardless of which CI/CD tools you use (e.g., Jenkins, GitLab, CircleCI) to deploy code, there are some key build metrics that you can monitor to better understand how your pipelines and individual jobs are performing, including:
- the slowest pipelines and jobs
- the rate of completed jobs per hour across your pipelines
- the success rate of individual pipelines and jobs
Visualizing this data in a dashboard, such as Datadog’s out-of-the-box dashboards for CI platforms like Jenkins and GitLab, enables you to detect unexpected changes in pipeline performance and quickly pinpoint issues.
For example, a significant change in job duration or the number of completed jobs could indicate an underlying issue with a CI server like Jenkins. You can compare this data with performance metrics from your Jenkins server, such as the memory utilization, to determine if you need to allocate more memory to the server or fine-tune a configuration setting.
Tracking the health of individual deployments can also provide better insight into pipeline performance and the efficiency of your development processes. You can monitor and compare overall performance across different deployments, which can help you gauge the effectiveness of shift-left testing for your teams. For example, you can monitor the stability of your deployments by comparing the total number of new errors created across several release versions.
If you see a significant increase in new errors for a release, you may need to audit your pipelines to determine if the problem was caused by issues such as gaps in test coverage for the new release, outdated test environments, or inefficient functions in your code.
Optimize development workflows by shifting left
In this post, we looked at some best practices for shifting testing to the left so you can improve the quality of your applications and release new features to your customers faster. Datadog supports shift-left testing with features that enable you to run tests directly from your CI/CD pipelines, monitor your CI pipelines and tests, and easily create good feedback channels across all of your teams. Check out our documentation to learn more or sign up for a free 14-day trial today.