State of DevSecOps | Datadog
State of DevSecOps

/ / /

Updated April 2025

This research builds on the previous edition of this article, which was published in April 2024. You can click here to download the graphs for each fact.

The practice of DevSecOps emphasizes the need to identify and respond to security risks at each point in the software development life cycle (SDLC), and has increasingly become the standard in the tech industry. In order to assess the types of risks defenders need to be aware of and what practices they can adopt to improve their security posture, we analyzed tens of thousands of applications and container images within thousands of cloud environments.

Our findings show that web applications face a wide range of risks, including known-exploitable vulnerabilities, supply chain attacks, and insecure identity configurations in CI/CD. However, applying context about the application in question, including what languages it uses and what environment it runs in, can reduce noise and help defenders prioritize risks. In addition, operational best practices like opting for smaller container images, utilizing infrastructure as code (IaC), and frequently deploying services tend to reduce applications’ risk exposure and provide a strong foundation for a secure SDLC.


Fact 1

Exploitable vulnerabilities are prevalent in web applications, particularly those that use Java

By analyzing our dataset of applications, we found that 15 percent of services are vulnerable to known-exploited vulnerabilities, affecting 30 percent of organizations. Vulnerabilities are particularly prevalent among Java services, 44 percent of which contain a known-exploited vulnerability. The next highest is Go at 5 percent, with the average for all non-Java services at 2 percent.

Almost half of Java applications are vulnerable to a known-exploited vulnerability

In fact, 14 percent of Java services still contain at least one vulnerability even when we look at only those that are known to be highly impactful, such as known remote code exploitation (RCE) vulnerabilities like Log4Shell, Spring4Shell, and other routinely exploited attack paths.

In addition to being more likely to contain high-impact vulnerabilities, Java applications are also patched more slowly than those from other programming ecosystems. We found that applications from the Java-based Apache Maven ecosystem had a median time to fix library vulnerabilities of 62 days, compared to 46 days for those in the .NET-based NuGet ecosystem and 19 days for applications built using npm packages, which are JavaScript-based.

Time to patch is longest for vulnerabilities in the Java-based Maven ecosystem

Leaving known vulnerabilities unpatched for prolonged periods is particularly risky because web applications are often exposed to the internet. Attackers use automated scanners to continuously scan the internet for high-impact, easy-to-exploit vulnerabilities, and recent research shows that vulnerabilities are often exploited just hours after they are initially disclosed.

Our research confirms this pattern of attacker behavior. We found that 88 percent of organizations received untargeted malicious HTTP requests, such as to /backup.sql, scanning for potentially exposed sensitive files or API routes. In addition, 65 percent of cases where a specific attacker attempted to exploit a specific URL were untargeted—i.e., the same attacker had tried to exploit the same URL in at least one other organization that Datadog monitors.

Given that attackers frequently operate not by targeting specific applications but by broadly scanning the internet for easily exploitable vulnerabilities, keeping library dependencies updated with the latest patches is critical, as any known vulnerability represents a serious risk. In addition, due to the high prevalence of known-exploitable RCE and other vulnerabilities in Java libraries—and the higher likelihood that these vulnerabilities remain unpatched—teams should pay particular attention to remediating vulnerabilities in Java applications.

Fact 2

Attackers continue to target the software supply chain

In addition to searching web applications for known vulnerabilities, attackers also commonly attempt to trick developers into downloading and deploying malicious packages from open source libraries. Throughout 2024, Datadog identified thousands of malicious PyPI and npm libraries in the wild. Some of these packages were malicious by nature and attempted to mimic a legitimate package (for instance, passports-js mimicking the legitimate passport library), a technique known as typosquatting. Others were active takeovers of popular, legitimate dependencies (such as Ultralytics, Solana web3.js, and lottie-player). These techniques are used both by state-sponsored actors and basic cybercriminals.

Both state-sponsored and basic cyber criminals are attacking the software supply chain

Attackers exploit the fact that vetting third-party dependencies is challenging for developers, especially if the package’s metadata and functionality appear to be legitimate.

To mitigate the growing prevalence of malicious packages in open source ecosystems, we hope package manager attestations—such as those now available for npm and PyPI—and increased maintainer account security will become more widespread and decrease the risk that legitimate packages will be taken over by bad actors. In addition, open source tools such as GuardDog and Supply-Chain Firewall can help users identify malicious code in software packages before they get installed.

Fact 3

Usage of long-lived credentials in CI/CD pipelines is still too high, but slowly decreasing

In addition to vulnerable code and malicious open source packages, the use of long-lived credentials that don’t expire is a common cause for cloud data breaches, and the practice is now widely frowned upon. Using long-lived credentials in CI/CD pipelines can be particularly risky, because their permissions are often highly privileged.

One scenario where long-lived credentials often make their way into CI/CD is when organizations use GitHub Actions to deploy applications to AWS, as these Actions can be configured with either short-lived credentials through OpenID Connect (OIDC) or AWS IAM user credentials, a type of long-term access key. We found that 37 percent of customers who send CloudTrail logs to Datadog are using IAM users in their GitHub Actions. We strongly recommend not relying on IAM users in this scenario but instead using OIDC.

While 63 percent of organizations using AWS and GitHub Actions leverage OIDC (up from 58 percent in 2024), 58 percent of organizations also leverage long-lived credentials from IAM users (down from 63 percent in 2024).

Though improved, most AWS organizations still use long-lived credentials in GitHub Actions pipelines

This shows that long-lived credentials, although a well-understood source of risk, remain a major source of technical debt, as they are typically challenging to migrate to more modern identity solutions.

Fact 4

Only a fraction of critical vulnerabilities are truly worth prioritizing

The high prevalence of library vulnerabilities, malicious open source packages, identity misconfigurations, and other issues creates a noisy environment for defenders. In this context, security teams need to understand how to prioritize issues and determine which to remediate first. These teams often make decisions based on the severity of a vulnerability—typically measured by its CVSS base score, which is published along with the metadata for a known vulnerability. Vulnerabilities considered high-severity by CVSS score are only increasing: We found that the average service has 13.5 vulnerabilities with a high or critical CVSS severity score, up from 11.9 in 2024.

However, in order to truly gauge a vulnerability’s severity, it’s essential to have the appropriate runtime context—for example, whether the vulnerability is running in a production environment, or if the application in which the vulnerability is found is exposed to the internet. CVSS does not take these factors into account, which leads to excessive noise for defenders.

To reduce this noise, we developed a prioritization algorithm that factors in runtime context. After applying this algorithm, we found the average number of high or critical vulnerabilities per application drops from 12.2 to 7.5.

The reduction in noise is even more significant when looking only at critical vulnerabilities—i.e., those that require the most urgent prioritization. Just 18 percent of vulnerabilities with a critical CVSS score—less than one in five—are still considered critical after applying our algorithm.

Runtime context shows that less than 1 in 5 critical vulnerabilities are worth prioritizing

This dramatic reduction in severe vulnerabilities represents hours saved each week, month, and quarter—and if defenders spend less time triaging issues, they can reduce their organizations’ attack surface all the faster. Focusing on easily exploitable vulnerabilities that are running in production environments for publicly exposed applications will yield the greatest real-world improvements in security posture.

“Without context, severity is just noise. True security comes not from patching everything, but from knowing what actually matters.”

Jean Burellier
Principal Software Engineer at Sanofi
Fact 5

Keeping libraries up to date is a major challenge for developers

In our research, we found that the median dependency is 215 days behind its latest major version. This reinforces the idea that engineers across DevSecOps functions are stretched and need to reduce noise, as teams struggle to keep library dependencies in their code up to date. Out-of-date libraries can increase the likelihood that a dependency contains unpatched, exploitable vulnerabilities.

The trend that dependencies tend to be out of date is true for all ecosystems, and stronger for some. For example, while the median JVM dependency is behind by 401 days, the median for Go is 168 days.

Across all languages, dependencies are months behind their latest major update

Among all services, those that are less frequently deployed are more likely to be using out-of-date libraries. We found that dependencies in services that are deployed less than once per month are 47 percent more outdated than those deployed daily, with a median of 217 days versus 148 days behind their latest version update.

Less frequently deployed services are more likely to have out-of-date dependencies

To complicate matters further, one in two services use libraries that are not actively maintained. Because updates are no longer being released at all for these libraries, any newly discovered vulnerabilities will remain unpatched.

To lower exposure to vulnerabilities, teams should identify and update outdated libraries on a consistent basis. This holds especially true for unmaintained libraries, which should be treated like an end-of-life operating system and immediately updated to an actively maintained version. In addition, developers should deploy their services frequently. Not only is doing so an operational best practice, but because each fresh deployment gives teams a chance to upgrade their dependencies, it also simplifies the task of maintaining a strong security posture.

Fact 6

Minimal container images improve security posture

In recent years, engineering teams have moved toward adopting minimal container images in order to improve the efficiency and cost-effectiveness of their systems. This trend began with minimal distributions such as Alpine Linux and has evolved into even smaller distroless and from-scratch images, all of which drastically reduce the size of a container image. For instance, the default Docker image for python:3.11 weighs over 1 GB, while the minimal version python:3.11-alpine weighs 58 MB, reducing its size by 95 percent.

This “less is more” approach has benefits for security as well. By analyzing thousands of container images, we found that on average, an image of less than 100 MB has three severe—i.e., with a high or critical CVSS score—vulnerabilities (median 0), while a container image of greater than 500 MB has on average 20 such vulnerabilities (median 7).

Using smaller container images leads to fewer vulnerabilities

In addition to containing fewer vulnerabilities in the first place, minimal container images also make it harder for attackers to “live off the land” by using system utilities like curl or wget to perform exploits (which is a very common behavior), since lightweight images typically don’t include this type of tooling. This reinforces the idea that teams should adopt lightweight container images wherever possible, both as an operational best practice and as a security measure.

“A smaller image means less to patch—and less to exploit. The more components you include, the larger the attack surface becomes. A distroless approach strips away all unnecessary packages and utilities, leaving only exactly what is needed for intended use; this minimizes vulnerabilities and ensures that attackers have far fewer footholds to work with. In our experience, minimal, hardened images not only cut down on CVEs but also help fast track compliance.”

Amber Bennoui
Principal Product Manager, Product Security, Chainguard
Fact 7

In AWS, infrastructure-as-code usage is high, but many teams still use ClickOps as well

Infrastructure as code (IaC) has become the dominant practice in cloud environments because it tends to lead to better operational outcomes, such as improved version control and traceability. In addition, IaC is a key component of security best practices, as it helps enforce peer review and makes infrastructure easier to scan for misconfigurations.

IaC adoption continues to grow more widespread. In AWS, 59 percent of organizations are using Terraform, and 57 percent use CloudFormation (slightly up from 2024). In all, 80 percent of organizations use at least one IaC tool.

In AWS, Terraform and CloudFormation are by far the leading IaC tools

At the same time, we also identified that at least 38 percent of organizations perform ClickOps deployments in production—i.e., having engineers manually log in to their AWS accounts to provision infrastructure through the AWS Console. ClickOps is generally less secure than IaC, as it introduces more opportunities for human error and results in a wider group of engineers having privileged access to the cloud environment.

Our research also shows that organizations using IaC are not exempt from ClickOps, as some are using both methods. This demonstrates that IaC, while an essential operational and security practice, is not enough—people are still accessing production environments manually. It remains critical to enforce IaC, limit privileged access, and continually scan cloud resources for vulnerabilities.

Subscribe for more Datadog security research, events, and updates.

Subscribe to the Datadog Security Digest

Methodology

Fact 1

For this fact, we analyzed vulnerabilities in third-party libraries of applications across various languages and runtimes (Java, .NET, PHP, Python, Ruby, Javascript, and Go) and that use Datadog Code Security’s Software Composition Analysis feature. We sourced known exploited vulnerabilities from the CISA KEV catalog, which we extracted on April 9, 2025.

For time to patch, we calculated the time it took for organizations to patch their vulnerabilities in each ecosystem.

We define “untargeted” as an instance in which the same attacker (identified by source IP) sent the same HTTP request to at least two Datadog customer orgs.

Fact 2

For this fact, we relied on the research and efforts of Datadog's security research team. Using tools like GuardDog, they identified and classified malicious packages across multiple software ecosystems since the publication of the 2024 version of this report.

Fact 3

To identify organizations that use GitHub Actions with OIDC authentication, we queried AWS CloudTrail logs using the following Datadog logs query:

@eventName:AssumeRoleWithWebIdentity 
@userIdentity.identityProvider:*token.actions.githubusercontent.com* -status:error

To identify organizations that use GitHub Actions with IAM users, we queried AWS CloudTrail logs using the following Datadog logs query: @userIdentity.accessKeyId:AKIA* @userIdentity.type:IAMUser -status:error. We also filtered the results on source IPs known to be used by GitHub Actions, as determined by GitHub’s API endpoint.

Fact 4

We analyzed vulnerabilities in third-party libraries called by applications that use Datadog Code Security’s Software Composition Analysis feature.

We considered vulnerabilities with a “critical” CVSSv3 base score and, based on the context available, used the following methodology to compute the “temporal” and “environmental” CVSSv3 metrics:

  • When the service was running in a non-production environment, we adjusted the “modified confidentiality, integrity, and availability impact” to “low.”
  • When the service was not publicly exposed on the internet and the exploit vector was “network,” we set the “modified attack vector” to “local.”
  • When the EPSS score was below 1 percent, we set the “modified attack complexity” to “high.”
  • When a public exploit was available, we set the “exploit code maturity” to “proof of concept” or to “unproven” otherwise.

We then computed the adjusted score based on the CVSS v3.1 methodology and considered the ratio of vulnerabilities whose adjusted score was still “critical.”

Fact 5

For this fact, we collected data on libraries that were active in March 2025. For each dependency, we calculated the number of days since the latest update to the current major version of a library was released. We then computed median lags across ecosystems (e.g., JVM, Go) and also compared them by deployment frequency.

We only looked at libraries that follow Semantic Versioning and excluded public libraries from our analysis.

For frequency of deployment, we obtained the data from the datadog.service.time_between_deployments metric, looking at deployments in the past six months. If a service was deployed 100 times or more in the last six months, we counted the service as being deployed daily, as there are 100 business days in six months. If a service was deployed 26 times or more in six months, we counted it as being deployed weekly. If the service was deployed six times or more, we counted it as being deployed monthly, and we categorized services deployed less than six times in six months as being deployed less than monthly.

Fact 6

We analyzed data from containers scanned through Datadog Cloud Security's Vulnerability Management feature and reviewed any identified OS-level vulnerabilities. This includes both publicly available images and images from private registries with at least one vulnerability. We excluded Datadog container images.

Fact 7

For this fact, we analyzed AWS CloudTrail Logs and determined which IaC technology was used based on the HTTP user agent.

Note: The data window for this fact is from the month of March 2025. If an organization did not use a known IaC technology during this period, we counted them as “not using IaC.”

To determine how many organizations were using ClickOps, we analyzed AWS CloudTrail logs. Specifically, we defined that an organization is “performing manual cloud deployments through the AWS Console” if we found at least one of the events from the list below in all the AWS accounts they monitor with Datadog. Because we assume that every organization monitors at least one production account with Datadog, we’re able to determine that organizations that meet this criteria use ClickOps in their production environment.

RunInstances
AuthorizeSecurityGroupIngress
CreateVpc
CreateCluster
CreateDBCluster
CreateDBInstance
CreateInstances
CreateKeyPair
RegisterTaskDefinition

We then filtered to identify when these events were performed manually from the AWS Console using the methodology described by Arkadiy Tetelman.

Licensing

Report: CC BY-ND 4.0

Images: CC BY-ND 4.0