[NOT READY FOR REVIEW][TEST] Introduce a sentinel layer variant for zstd:chunked pulls by giuseppe · Pull Request #681 · containers/container-libs

giuseppe · 2026-03-05T16:43:30Z

Introduce a sentinel layer variant to safely skip the full-digest mitigation for zstd:chunked pulls.

This PR eliminates the duality by creating a separate manifest variant with a non-tar sentinel layer prepended. Unaware clients and scanners cannot process the sentinel variant so the image no longer has two possible interpretations.

Aware clients recognize the sentinel, skip it, and pull without the expensive full-digest recomputation safely because no unaware tool can be tricked into validating the wrong content. Data layers are reused by digest so there is no duplication.

I've pushed an image with the new format here: quay.io/giuseppe/zstd-chunked:sentinel

@mtrmac @actionmancan this is a PoC for the idea we have discussed on how to relax the mitigation

packit-as-a-service · 2026-03-05T16:44:32Z

Packit jobs failed. @containers/packit-build please check.

mtrmac

A first look…

mtrmac · 2026-03-05T16:58:25Z

image/copy/multiple.go

 	}

+	// Create zstd:chunked sentinel variants for instances where any layer uses zstd:chunked.
+	if cannotModifyManifestListReason == "" {


Initially, I would expect this decision to happen in prepareInstanceCopies (future prepareInstanceOps), and unit-tested there.

OTOH this seems to be happening later because we collect the pushed manifests… that might be a good reason.

mtrmac · 2026-03-05T16:59:00Z

image/copy/multiple.go

+
+// hasZstdChunkedLayers returns true if any non-empty layer in the manifest has
+// zstd:chunked TOC annotations.
+func hasZstdChunkedLayers(ociMan *manifest.OCI1) bool {


I wonder about correct layering / abstractions, but that can be decided a bit later.

mtrmac · 2026-03-05T17:04:03Z

image/copy/multiple.go

+	sentinelManifestDigest := digest.FromBytes(sentinelManifestBlob)
+
+	// Push sentinel manifest.
+	if err := c.dest.PutManifest(ctx, sentinelManifestBlob, &sentinelManifestDigest); err != nil {


The rules say: push each blob ~in order, then do PutManifest. Pushing only the extra blob works for docker:// but not for other transports.

I think this should probably be an “almost ordinary” copy from the one image on the destination to another, using copySingleImage with an “add a layer” flag. That would naturally trigger the API-required TryReusingBlob calls, and we don’t need a special code path creating an updated manifest here.

… or maybe, actually, “remove a layer” flag? Do we start with slow-pulling zstd-chunked and TOC annotations, and add the "fast pull" marker only, or do we start with "fast pull zstd-chunked with TOC", and turn that into "~ordinary zstd with no TOC annotations at all"?

Yet another aspect to this: When decompressing or re-compressing (to gzip), we need to strip the sentinel.

So, in this sense, the “copy single instance” code needs to be compression-sentinel aware whatever we do. It might then be elegant for the “create a zstd:chunked” code path to add the sentinel as part of an “ordinary compression”, and we would then clone that “fast pull” image into a compatible one, per the “remove a layer” option in the comment above.

image/copy/multiple.go

mtrmac · 2026-03-05T17:31:49Z

storage/pkg/chunked/toc/toc.go

+// ZstdChunkedSentinelContent is the well-known content of the sentinel layer
+// used to signal that a manifest's zstd:chunked layers can be used without
+// the full-digest mitigation.
+const ZstdChunkedSentinelContent = "zstd:chunked"


Make very sure this is not valid tar. I guess that means including enough for a tar header, but with a non-tar magic and invalid v7 checksum?

Ensure there is a at least one binary 0 close to the start to mark it as a non-text format

“This image must be consumed using the TOC digests and NEVER as a tar archive", perhaps link to some future documentation.

storage/pkg/chunked/storage_linux.go

mtrmac · 2026-03-05T17:38:01Z

storage/pkg/chunked/storage_unsupported.go

 )

+// SkipMitigation is a no-op on unsupported platforms.
+func SkipMitigation(d graphdriver.Differ) {}


Ultimately it would be cleaner for NewDiffer to return a struct that implements graphdriver.Differ; we could just add a method. (But now that would be an API break…)

mtrmac · 2026-03-05T17:47:36Z

image/storage/storage_dest.go

+// that should be treated as empty. For c/storage, this detects the zstd:chunked
+// sentinel layer and records that the full-digest mitigation can be skipped.
+func (s *storageImageDestination) FilterLayers(layerInfos []manifest.LayerInfo) []int {
+	if len(layerInfos) > 0 && layerInfos[0].Digest == toc.ZstdChunkedSentinelDigest {


⚠️ If we see this, we must refuse to use the tar consumption code in PutBlob (for layer with a TOC?, or for all?), and this almost certainly also affects TryReusingBlob. Perhaps more code around trustedLayerIdentityData also needs updating (hopefully that is already set up to correctly handle insecureAllowUnpredictableImageContents)

[Layer copies happen in parallel, and we need to enforce this for the very first (well, second, after the sentinel) layer, so this really needs to be either a new private.ImageDestination… operation, or an extension of the existing NoteOriginalOCIConfig to also provide a manifest or layer data.)

Another aspect to this: We need to disable the makeConvertFromRawDiffer fallback.

Add sentinel layer constants (ZstdChunkedSentinelContent and ZstdChunkedSentinelDigest) to the toc package and introduce SkipMitigation() to disable full-digest verification when a zstd:chunked sentinel layer is detected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

…election Add annotation constants for identifying zstd:chunked sentinel layers in OCI manifests. Extend instance candidate selection to prefer manifests with sentinel annotations when available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

…ion during pull When pulling an image, the storage destination's FilterLayers method detects the zstd:chunked sentinel layer, marks it as empty, and internally records that SkipMitigation() should be called to bypass unnecessary full-digest verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

When pushing images, generate zstd:chunked sentinel variant manifests. For manifest lists, create sentinel variants for each instance after copying. For single images, wrap the result in an OCI index with a sentinel variant when the destination supports it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the zstd-chunked-no-mitigation branch from 62a4a6a to ed03b46 Compare March 5, 2026 16:44

github-actions bot added storage Related to "storage" package image Related to "image" package labels Mar 5, 2026

mtrmac reviewed Mar 5, 2026

View reviewed changes

giuseppe force-pushed the zstd-chunked-no-mitigation branch 2 times, most recently from 536bf3d to d7973eb Compare March 6, 2026 15:58

giuseppe and others added 4 commits March 11, 2026 11:43

giuseppe force-pushed the zstd-chunked-no-mitigation branch from d7973eb to 8cfcfca Compare March 11, 2026 11:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NOT READY FOR REVIEW][TEST] Introduce a sentinel layer variant for zstd:chunked pulls#681

[NOT READY FOR REVIEW][TEST] Introduce a sentinel layer variant for zstd:chunked pulls#681
giuseppe wants to merge 4 commits intocontainers:mainfrom
giuseppe:zstd-chunked-no-mitigation

giuseppe commented Mar 5, 2026

Uh oh!

packit-as-a-service bot commented Mar 5, 2026

Uh oh!

mtrmac left a comment

Uh oh!

mtrmac Mar 5, 2026

Uh oh!

mtrmac Mar 5, 2026

Uh oh!

mtrmac Mar 5, 2026 •

edited

Loading

Uh oh!

mtrmac Mar 10, 2026

Uh oh!

Uh oh!

Uh oh!

mtrmac Mar 5, 2026

Uh oh!

Uh oh!

Uh oh!

mtrmac Mar 5, 2026

Uh oh!

mtrmac Mar 5, 2026

Uh oh!

mtrmac Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

giuseppe commented Mar 5, 2026

Uh oh!

packit-as-a-service bot commented Mar 5, 2026

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mtrmac Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mtrmac Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mtrmac Mar 5, 2026 •

edited

Loading