Beyond container image signing: How we enforce software supply chain policies at Datadog

A malicious change introduced in source, a poisoned dependency, or a compromised build runner that swaps a base image can each produce an image with a valid signature. Container image signing can verify that the image has not been tampered with since it was signed, but it cannot detect those earlier compromises. The evidence needed to identify these risks, including pull request approvals and the identity of the build runner, is often scattered across independent systems and difficult to connect to the finished image deployed in production.

So how do we turn that evidence into a deployment decision we can verify, reuse, and audit without creating unnecessary friction for developers?

Our answer is an internal system we call Turnstile, in keeping with our transit-themed CI/CD tooling. Turnstile generates cryptographically signed, tamper-evident attestations across an artifact’s life cycle and evaluates them against environment-specific policies at deployment time. It builds on our existing container image signing and runtime verification system, extending the question from “Has this image changed since it was signed?” to “Does this artifact satisfy the security requirements for this environment?”

In this post, we’ll explain how we combined and adapted open source frameworks and standards, including the principles of source and build integrity documented in the SLSA threat model, to make these evaluations reliable in the deployment hot path at Datadog’s scale.

From image signing to artifact provenance

Container image signing gives organizations a strong primitive: a cryptographic link between an image digest and a trusted signing event. In our image-signing system, images are signed after they are built, and our custom containerd plugin verifies each signature when an image is pulled. This protects the final distribution step by verifying that the image has not been modified after signing and before it reaches the host on which the container runs.

But this still leaves a gap. Consider a malicious change that is introduced before the image is built. If the image is built normally, pushed normally, and signed normally, then signature verification will succeed. The image has not been altered since signing. The problem is what went into it. 

Similarly, consider an image that is built from a pull request but includes a dependency with a critical vulnerability. Or an image that was built from a branch that does not meet the security requirements for a regulated environment. Or an artifact that has a valid image signature but cannot be traced back to the code, commit, or pull request that introduced it.

In each case, the signature helps us answer two questions: “Is this the same artifact that was signed?” and “Was it signed by an identity we trust?” 

Turnstile is designed to answer a broader question: “Does this artifact satisfy the policies for this deployment?”

Turnstile was built around three core ideas:

• Signed evidence: Each meaningful event in the artifact life cycle can produce a cryptographically signed claim about an artifact.

• Scoped policy: Different environments can require different evidence, so policies can be stricter for production or regulated environments than for development workflows.

• Progressive enforcement at Datadog scale: Rolling out policy in a large, polyglot environment like Datadog’s can be challenging, so Turnstile lets us introduce policies progressively, with certain policies warning in some environments and blocking in others. These evaluations operate in the hot path at scale, across tens of thousands of deployments and images each day.

Close dialog

Modeling security checks as attestations

The foundation of Turnstile is the attestation, a cryptographically signed claim that includes metadata about an artifact. More specifically, an attestation is an identity-bound statement that something happened to or is true about a specific artifact at a specific point in its life cycle.

To model our attestations, we used a modified version of in-toto, an open source framework designed to make transparent what steps were performed, by whom, and in what order. In practice, each attestation needs to answer a few basic questions:

• What artifact is this about? The subject is usually an immutable artifact identifier, such as a container image digest or commit hash.

• What kind of claim is being made? The predicate field contains metadata describing the claim, while predicateType identifies what that claim means—for example, source provenance, build provenance, or a software bill of materials (SBOM).

• Who made the claim? The attestation is signed by a trusted producer, such as a CI job or platform service.

• Can the claim be verified later? The attestation is stored and keyed to the artifact identifier so that consumers can retrieve and verify it independently of the original workflow.

The key benefit is the binding between each step. Code goes through multiple steps on the way to production, transitioning from a commit to a pull request to a container image. As code moves toward production, source code management and build systems can drop or bury the references connecting these stages.

Using in-toto, we can connect these stages by keying each phase to the next and giving each generated attestation a queryable key that links it to the following phase. This produces a map of our software supply chain:

At Datadog, we use this format to generate attestations for every container image we build. The following is an example:

1
_type: https://in-toto.io/Statement/v1
2
subject:
3
  - name: git+https://github.com/acme/payments-api
4
    digest: { gitCommit: abc123 }
5
  - name: registry.acme.example/payments-api
6
    digest: { sha256: 789xyz }
7
predicateType: https://acme.example/supply-chain/v1
8
predicate:
9
  commitSafety:
10
    bewaireScan: { verdict: benign }
11
    signature: { verified: true, keyId: trusted-key }
12
  sourceProvenance: { repository: acme/payments-api, protections: enabled }
13
  pullRequest:
14
    number: 1842
15
    reviewed: true
16
    approvedBy: [bob, carol]
17
  buildProvenance: { pipeline: build-and-publish, job: build-image, imageSigned: true }
18
  sbom: [payments-api@1.18.4, go-chi@5.2.2, jwt@5.2.2, busybox@1.37]
19
  vulnerabilities: { scannedAtBuild: true, critical: 0, high: 0 }
20
  deployment: { cluster: prod-us-east-1, namespace: payments-production }
21

22
{
23
  "_type": "https://in-toto.io/Statement/v1",
24
  "subject": [
25
    {
26
      "name": "git+commit",
27
      "digest": { "sha1": "abc123" }
28
    }
29
  ],
30
  "predicateType": "https://ams.datadoghq.com/github-pull-request/v1",
31
  "predicate": {
32
    "repository": {
33
      "owner": "acme",
34
      "name": "payments-api"
35
    },
36
    "pr_info": {
37
      "number": 42,
38
      "merge_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0",
39
      "merged": true,
40
      "base_ref": "main",
41
      "head_ref": "feature/example-branch"
42
    },
43
    "review_summary": {
44
      "approval_count": 2,
45
      "approvers": ["reviewer-one", "reviewer-two"]
46
    },
47
    "is_base_ref_protected": true
48
  }
49
}

The structure of these attestations also provides auditable records. During an investigation, we can reconstruct the evidence associated with a deployed artifact: where it came from, how it was built, what security checks were performed, which policies were applied, and what decision the evaluation returned.

Keyless signing and identity-backed evidence

Once we understood the data structure we wanted, our next challenge was finding a simple way to generate attestations. Attestations are security-critical data used for governance and audit purposes and need to be handled sensitively. But there’s a risk: If every CI job, build service, scanner, approval workflow, and deployment workflow needs to manage long-lived signing keys, the platform and developer experience can degrade quickly.

To address this problem, Turnstile offers a client-side keyless-signing library called dd-attest. The library offers three core features:

• Sign an attestation for a subject such as a container image, file digest, URI, or named artifact, by using an in-toto predicate such as source provenance or an SBOM.

• Store and retrieve attestations from the Attestation Metadata Store (AMS), our DynamoDB backend for storing all attestations.

• Verify signatures and countersignatures to prove that an attestation was signed by a trusted identity-bound key.

Instead of distributing long-lived private keys to each producer, dd-attest signs each attestation with an ephemeral key that is bound to an existing workload identity, such as a CI job identity, Kubernetes service account, or service identity backed by OpenID Connect (OIDC). This shifts trust from “Who has possession of this long-lived key?” to “Which identity produced this attestation, and was that identity allowed to make this claim?” 

That distinction is important. In a large software delivery environment, the list of trusted attestation producers changes over time. CI jobs move, services are renamed, teams migrate build systems, and new security checks are added. Binding attestations to identity allows us to reason about trusted attestation producers in a way that maps more naturally to the systems we already operate. It also enables us to revoke trust in case of system compromise. If a CI job is compromised by a malicious actor, all attestations generated by that producer during the affected period can be invalidated. 

In Turnstile, policy evaluation can require that a specific type of attestation be signed by a trusted producer. For example, a policy might require that an SBOM attestation come from the approved image scanning pipeline or that source metadata come from the service responsible for observing source control state. If the claim is present but signed by an untrusted identity, it does not satisfy the policy.

Turning attestations into policies

Attestations answer the question, “What claims exist about this artifact, and who’s asserting them?” Policies answer the next question: “Are those claims sufficient to deploy this artifact to that environment?”

This is where Turnstile becomes more flexible than image signing alone. Using the attestations we generate for each artifact at Datadog, we can express complex supply chain policies such as:

• The artifact must have source provenance attestations that chain back to a known organization or repository.

• The artifact must have commits signed by employees.

• The artifact must have been signed by the approved image-signing service.

• The source commit must be associated with a merged, approved pull request on a protected branch.

• The pull request must have passed automated malicious-code review checks.

• The artifact must have an SBOM attestation from the approved scanner.

• The SBOM must not contain a package matching a known critical vulnerability rule.

• The policy evaluation itself must produce a signed summary that downstream consumers can verify.

We can also scope policies to different deployment contexts. Depending on an environment’s requirements, a policy might evaluate any combination of image-signing evidence, source provenance, pull request approval, SBOM availability, and the presence of disallowed dependencies.

We use Open Policy Agent (OPA) to evaluate policies written in Rego:

1
import rego.v1
2

3
sbom_predicate := "https://cyclonedx.org/bom"
4

5
default allow := false
6

7
allow if {
8
    input.predicateType == "https://slsa.dev/verification_summary/v1"
9
    input.predicate.evaluation_context.datacenter == "us3.datadoghq.com"
10

11
    some attestation in input.predicate.input_attestations
12
    attestation.name == sbom_predicate
13
    attestation.media_type == "application/vnd.in-toto+json"
14
    attestation.digest.sha256 != ""
15
    startswith(attestation.uri, "ams://")
16
    contains(attestation.uri, sprintf("predicateType=%s", [sbom_predicate]))
17
}
18

19
deny_reason contains "VSA does not reference a valid CycloneDX SBOM" if {
20
    not allow
21
}

We can introduce policies gradually by first running evaluations in audit mode to identify gaps in our pipeline and detect potential false positives before enforcement.

Evaluating policy at deployment time

To evaluate policies, Turnstile provides another platform component: policy-core, a library that can act as an enforcement point across environments. At deployment time, policy-core receives an image digest and evaluates it against the active policy set for the target environment.

The evaluation flow looks like this:

1. A deployment workflow reaches a Turnstile policy enforcement point.

2. Turnstile identifies the artifact being deployed, typically by container digest or commit SHA.

3. Policy-core retrieves all attestations associated with that artifact from the AMS.

4. It verifies that the attestations are valid and signed by trusted producers.

5. It evaluates the relevant policies for the deployment scope.

6. It records the result as a signed policy evaluation summary and writes it back to the AMS.

7. It emits a decision the deployment workflow can use to audit, warn, or block.

We model the policy evaluation result with our in-house modification of SLSA’s Verification Summary Attestation, or VSA. In SLSA, a VSA communicates an artifact’s verified SLSA level and other details about the verification. Our internal version includes additional verification details, allowing consumers to make decisions without independently evaluating every input attestation.

For Turnstile, the VSA becomes the durable record of a policy evaluation. It can include the artifact that was evaluated, the policy version, the input attestations considered, the evaluation time, and the pass-or-fail result. This gives us a compact record of the artifact’s policy state that other systems can consume without rerunning the entire evaluation.

1
_type: https://in-toto.io/Statement/v1
2
predicateType: https://slsa.dev/verification_summary/v1
3
subject:
4
  - name: pkg:oci/acme/security/artifact-observer?repository_url=registry.example.com
5
    digest: { sha256: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa }
6
predicate:
7
  caller: deployment-controller
8
  channel: stable
9
  evaluation_context:
10
    environment: prod
11
    datacenter: us-east-1.prod.example
12
    cluster: prod-cluster-a
13
    namespace: security-platform
14
    service: artifact-observer
15
  policy:
16
    name: pci-compliance
17
    uri: policy://pci-compliance
18
    digest: { sha256: dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd }
19
  input_attestations:
20
    - name: https://example.com/attestations/image-signature/v1
21
      digest: { sha256: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb }
22
    - name: https://example.com/attestations/pull-request-review/v1
23
      digest: { sha256: cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc }
24
  policy_summary: '{"commit":"eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee","image_valid":true,"pr_valid":true}'
25
  verification_result: PASSED
26
  time_verified: 2026-08-04T12:35:30Z
27
  verifier: { id: https://example.com/turnstile/opa-verifier, version: { opa: v1.15.2 } }

That pattern is especially useful as more systems begin to depend on artifact integrity data. A deployment UI can display policy status. A compliance workflow can report policy coverage. An incident response workflow can determine whether a deployed artifact passed the expected checks when it was introduced. Each of those systems can consume the policy result instead of reimplementing policy evaluation. Using VSAs to delegate policy decisions also enables us to verify an artifact closer to its time of use—at runtime—without requiring each consumer to rerun the full evaluation.

Distributing policy securely using TUF 

Once we had the full end-to-end engine working, we realized we needed another layer of security: secure distribution of the policies themselves.

Policies that govern where and under which conditions software can run are themselves security-sensitive targets. A compromised policy could allow artifacts that should fail. An outdated policy could miss a newly introduced requirement. A maliciously modified policy could disable an important check. An accidental policy change could block valid deployments across many services. 

This means that the policies themselves require their own root of trust.

Turnstile uses The Update Framework (TUF) to distribute and verify policy bundles and our dd-attest and policy-core libraries. We use tuf-on-ci to manage root operations. TUF was designed to secure software update systems and minimize the impact of key compromises by using signed metadata, role separation, and threshold signatures. At a high level, the model looks like this:

• Offline root keys establish the top-level trust anchor.

• Metadata describing policy bundles is signed before publication.

• Clients fetch policy material through the normal distribution path.

• Before use, clients verify the signed metadata against the trusted root and confirm that the policy material matches that metadata.

• If verification fails, the policy material is rejected.

In Turnstile, this matters for two reasons. First, policy consumers need to verify that the policy they are about to evaluate is authentic. Even if an attacker can tamper with storage or transit, the policy client should reject modified content that does not match metadata authenticated by the trusted root. Second, policy updates need to be operationally safe. We want security teams to be able to update policies quickly without bypassing review, signing, or distribution controls. 

TUF gives us a structured way to separate high-sensitivity root keys from more frequently updated policy metadata. Policies and their TUF metadata are packaged and distributed as Open Container Initiative (OCI) artifacts and pulled from the same registry as container images and signatures, avoiding another runtime dependency.

The result is a system that behaves less like a hard-coded deployment check and more like a distributed control plane for artifact security.

Scaling policy evaluations with DynamoDB and caching 

The next question was how to scale Turnstile for Datadog. Security is critical, but it cannot come at the cost of reliability. A security system that cannot scale with the growth and complexity of our company will ultimately fail operationally. 

Because policy evaluation would operate in the hot path, we needed a data store that could deliver low latency at scale, provide reliability across multiple availability zones, and scale horizontally without the operational overhead of managing database sharding. We also knew what we didn’t need: Policy evaluations primarily involve predictable key-value lookups rather than relational queries, so a relational data model wasn’t immediately relevant to our goals.

We selected DynamoDB to back AMS, which policy-core accesses during policy evaluation. Because artifacts are often deployed multiple times to the same service, we added a per-process least recently used (LRU) cache for VSAs to avoid repeated evaluations. The cache key includes details such as artifact digest, policy ID, the binary versions of the software and control plane included, environment, and other relevant input labels.

On a cache miss, Turnstile performs the following steps:

1. Loads the policy.

2. Fetches attestations from AMS.

3. Verifies the attestations.

4. Executes the Rego policy for the environment.

5. Builds a VSA statement.

6. Stores that statement under the fingerprint.

Close dialog

Across approximately 1 million evaluation traces during August 2026, Turnstile achieved:

• A 92.8% policy cache hit rate

• A 75% cache hit rate for per-artifact evaluations

• Individual evaluation latency:

  • P50: 36 ms

  • P99: 481 ms

Building a verifiable software supply chain

Turnstile evaluates every container image deployed at Datadog, generating millions of attestations and tens of thousands of policy evaluations each day. Building on our foundation of container image signing, Turnstile enables us to answer a broader set of questions about our supply chain—not only whether an image was signed, but who authored it, what it contains, where it was built, and whether it satisfies the policies for a given environment. It treats our entire software delivery life cycle as untrusted and requires each artifact to provide verifiable evidence that it meets the requirements for deployment to our production environments, giving us stronger security enforcement and auditability. 

As software delivery becomes more automated and more code is produced by agents, extending implicit trust to every system that produces or modifies an artifact becomes less tenable. Turnstile gives us a reusable model for governance in that environment: Cryptographically signed, tamper-evident records connect evidence across the artifact life cycle, while distributed enforcement points let us verify that evidence at deployment time. This helps us protect our systems and customers as the software supply chain continues to evolve.

If you’re interested in working on security challenges like these at Datadog, explore our open positions.

Acknowledgments

We would like to thank Beth Bayer, Mark Bestavros, James Carnegie, Harmon Herring, Mike Lococo, Claire Thibodeaux, and Fabio Zöllner for their contributions to this work.