Missing container-layer metadata: Why it happens and what you can do

Container image layers provide valuable insight into what goes into a container, including which packages were installed, what commands were run, and where vulnerabilities might live. The metadata associated with these image layers is essential for debugging, optimizing image size, and managing security risks. However, key container-layer metadata fields such as digest, size, and created_by are sometimes missing, which can disrupt important tasks. Each of these metadata fields plays an important role in understanding the health of your system:

• Without digest, it's difficult to trace child images back to their base images.
• Without size, it's harder to detect bloated layers or identify optimization opportunities.
• Without created_by, security teams can’t correlate vulnerabilities with specific commands or package additions.

In this post, we’ll explore the root causes of these missing fields and provide strategies to recover or work around them.

What metadata might be missing

The Open Container Initiative (OCI) Spec defines how container images are structured for compatibility across tools. Some fields are required for runtime correctness, while others—though optional—are highly useful for visibility and analysis. Optional fields may not always be populated by container runtimes, and their absence will be reflected in your observability solutions, including Datadog Container Monitoring’s container images view.

Optional fields under history like created, author, created_by, comment, and empty_layer are commonly absent, but they are helpful for reconstructing build timelines, debugging, understanding how a layer was created, and maintaining compliance. Beyond these, two important metadata fields defined in the OCI descriptor object—digest and size—are also frequently missing, and we’ll explore why below.

Root causes of missing fields

There are several reasons why container image metadata fields like digest, size, and created_by might be missing. Some of these reasons are expected and tied to how container layers work, while others stem from the limitations or behaviors of specific build tools. In this section, we’ll walk through the most common scenarios that lead to missing fields, explain whether they’re expected or problematic, and offer guidance on what you can do about them.

Temporary intermediate layers

Temporary intermediate or "empty" layers (where empty_layer=true) represent operations that don’t change the image file system, such as setting environment variables or setting arguments. While these layers are important for caching, they don’t generate file system changes, so they don’t receive digest or size values.

This behavior is expected and nothing to worry about—in this case, digest and size are intentionally omitted because the layer doesn’t introduce any changes to the image's file system. Since there’s no new content to hash or measure, these fields are simply not applicable. In the UI example below, layers using commands like ARG lack digest and size, while those modifying the file system (e.g., WORKDIR) include both.

Close dialog

Missing digest

In some cases, the digest field is marked <missing> as a default display in the Datadog UI when there is missing data for Docker images. This is confusing, as -- also represents missing data and suggests tooling bugs or incorrect image construction.

Close dialog

If you run into this issue, your first strategy should be to update the datadog-agent version to ≥ 7.64, which should repopulate most of the <missing> digest fields with non-missing values. Alternatively, you can re-export the image by using another runtime, such as containerd, which will replace <missing> with -- as well as potentially recover missing digests.

Third-party tooling

There are several types of third-party container image tools that can result in missing image-layer metadata.

Jib-built images

Jib is an open source tool developed by Google that builds container images for Java applications without requiring a Docker daemon or writing a Dockerfile. It's designed to work seamlessly with Maven and Gradle, enabling Java developers to containerize applications as part of their regular build process. Some Jib versions omit metadata fields like created or created_by and set timestamps to the Unix epoch by default, which results in missing metadata. The solution is to upgrade to Jib ≥ 0.9.10 to incorporate layer history and explicitly set creationTime in your Maven or Gradle config, as shown in the examples below:

Maven:

1
<configuration>
2
  <container>
3
    <creationTime>2019-07-15T10:15:30+09:00</creationTime>
4
  </container>
5
</configuration>

Gradle:

1
jib.container.creationTime = '2019-07-15T10:15:30+09:00'

Rules-oci

Rules-oci is a Bazel ruleset for building, manipulating, and publishing OCI-compliant container images. The images are built using Bazel's build system instead of through a daemon. This means the history must be manually added during implementation or as part of the build process. Because rules_oci v1 and earlier versions of rules_oci v2 builds omit history fields, you should update your rules_oci package to one that includes this change (any version ≥ v2.0.0).

Crane flatten

Crane is an open source command-line tool built on top of Docker commands that lets you inspect and modify containers. The crane flatten command merges multiple image layers into a single one, which can simplify or optimize images. However, the resulting layer often lacks a created_time. The omission is due to the tool not explicitly setting this field in the source code. To solve this issue, you can wait for this open source change to the source code maintained by Google, or fork and create the changes yourself—if you want more flexibility—and insert a created field, as shown in the example below.

1
img, err = mutate.Append(img, mutate.Addendum{
2
    Layer: layer,
3
    History: v1.History{
4
      CreatedBy: fmt.Sprintf("%s flatten %s", use, digest),
5
      Comment:   string(oldHistory),
6
      // We can see that there is no created time here
7
      // Add "Created" field from the History type and necessary changes to propagate value from unflattened images
8
    },
9
  })

Challenges with base images

When the base image is missing metadata in its layers, any child image built from it will inherit that missing information.

Close dialog

In the example above, there is no visible association of child layers to base image layers, which means you can't easily tell which layers came from the base image. Although the FROM command sets the base image, this command isn’t recorded in the layer history. Instead, the base image's layers are stacked with the new ones added in your build script without explicitly associating whether each layer belongs to a base image.

Unless you compare layer digests manually or refer to the original build script, it’s hard to tell which layer came from where. To figure out which layers came from the base image, you’d have to manually compare the layers in your image with those in other images—or look at the original Dockerfile or build scripts. Since there is a one-to-one association between base and child images’ layers, missing metadata in base image layers are inherited by the child image. For base images maintained by a third party, you can try contacting the image maintainer to request inclusion of the missing metadata when they construct their images. Otherwise, if you own the base image yourself, you’ll need to build the base image to recreate the image with correct metadata by using the methods we outlined above.

Summary

This table summarizes the expected reasons for each missing field as observed from the common patterns we’ve discussed.

Understand and troubleshoot missing container metadata

In this post, we’ve walked through why container image metadata like digest, size, and created_by may be missing, as well as practical solutions and tools to regain visibility. If you want to see how we help you understand what’s inside your containers, check out the container images view in Datadog Container Monitoring. Or, if you’re new to Datadog, sign up for a free trial to get started.