copy: add option to strip sparse manifest lists#655
copy: add option to strip sparse manifest lists#655aguidirh wants to merge 23 commits intocontainers:mainfrom
Conversation
|
Packit jobs failed. @containers/packit-build please check. |
mtrmac
left a comment
There was a problem hiding this comment.
Thanks! Just a brief first look: generally the structure and choice of affected sub packages look correct.
This is impressively small, I was expecting a that a larger change to internal/manifest would be necessary.
copy: implement code review feedback for sparse manifest stripping Address feedback from mtrmac's review on PR containers#655: 1. Simplify deletion logic by combining EditInstances calls - Moved deletion tracking into prepareInstanceCopies function - Combined update and delete operations into single EditInstances call - Removed separate digest-based tracking and redundant loops 2. Use slices.Delete for manifest entry removal - Replaced manual slice manipulation with slices.Delete in both docker_schema2_list.go and oci_index.go for cleaner code 3. Integrate deletion tests into existing test methods - Removed standalone TestListDeleteInstances function - Added deletion test cases to TestSchema2ListEditInstances and TestOCI1EditInstances for better test organization 4. Move deletion logic into prepareInstanceCopies (most critical fix) - Modified prepareInstanceCopies to return ([]instanceCopy, []int, error) - Track indices to delete when SparseManifestListAction == StripSparseManifestList - Properly handles duplicate digests by using instance indices instead of digest-based lookup, fixing edge case where same digest appears multiple times in manifest list 5. Use slices.Backward for reverse iteration - Simplified deletion loop using slices.Backward iterator - Maintains high-to-low deletion order to avoid index shifting 6. Move SparseManifestListAction field for better organization - Relocated field from line 150 to line 116 in copy.go - Now positioned near related ImageListSelection and Instances fields - Improves logical grouping of manifest list options All tests pass. The implementation now correctly handles: - Duplicate digests in manifest lists - Single EditInstances call for efficiency - Cleaner separation of concerns - Better code organization Fixes feedback on containers#655 Signed-off-by: Alex Guidi <aguidi@redhat.com>
|
@mtrmac when using |
mtrmac
left a comment
There was a problem hiding this comment.
Thanks!
Just a very quick look for now.
Hum… it’s a thought. We currently ask the user to explicitly use Actually, c/image only cares about the signatures of the per-platform instances, it never validates the signature on the list (but it does create it). So, ideally, we should allow stripping the list’s signature but keep the per-instance signatures, so that users can create sparse mirrors but still validate image integrity. I think that would be a new option in E.g., purely hypothetically, it might make sense to hard-code this to on when mirroring the OpenShift product payload, because we know how signatures are validated there; and maybe not hard-code it for other images, because those might be validated using some other software like |
|
LGTM |
Add StripOnlyListSignatures option to strip manifest list signatures while preserving per-instance signatures when using CopySpecificImages with StripSparseManifestList. This implements "Option 2" from PR containers#655. Previously, users had to use RemoveSignatures which removed all signatures, including per-instance ones that would still be valid after stripping the manifest list. The option requires explicit validation: - Only valid with CopySpecificImages + StripSparseManifestList - Validates at multiple points in copy.Image() and copyMultipleImages() - When used with signed images, requires explicit signature handling Per-instance signatures are preserved because copySingleImage handles each instance independently. Signed-off-by: Alex Guidi <aguidi@redhat.com>
I added a validation which requires the user to use
I added an option
I'm not sure if I understood this last comment, could you please explain it to me? I just pushed a commit with the last changes that I mentioned above. |
Software that knows how the images is going to be used can hard-code option values so that users don’t have to care; generic software like |
copy: implement code review feedback for sparse manifest stripping Address feedback from mtrmac's review on PR containers#655: 1. Simplify deletion logic by combining EditInstances calls - Moved deletion tracking into prepareInstanceCopies function - Combined update and delete operations into single EditInstances call - Removed separate digest-based tracking and redundant loops 2. Use slices.Delete for manifest entry removal - Replaced manual slice manipulation with slices.Delete in both docker_schema2_list.go and oci_index.go for cleaner code 3. Integrate deletion tests into existing test methods - Removed standalone TestListDeleteInstances function - Added deletion test cases to TestSchema2ListEditInstances and TestOCI1EditInstances for better test organization 4. Move deletion logic into prepareInstanceCopies (most critical fix) - Modified prepareInstanceCopies to return ([]instanceCopy, []int, error) - Track indices to delete when SparseManifestListAction == StripSparseManifestList - Properly handles duplicate digests by using instance indices instead of digest-based lookup, fixing edge case where same digest appears multiple times in manifest list 5. Use slices.Backward for reverse iteration - Simplified deletion loop using slices.Backward iterator - Maintains high-to-low deletion order to avoid index shifting 6. Move SparseManifestListAction field for better organization - Relocated field from line 150 to line 116 in copy.go - Now positioned near related ImageListSelection and Instances fields - Improves logical grouping of manifest list options All tests pass. The implementation now correctly handles: - Duplicate digests in manifest lists - Single EditInstances call for efficiency - Cleaner separation of concerns - Better code organization Fixes feedback on containers#655 Signed-off-by: Alex Guidi <aguidi@redhat.com>
Add StripOnlyListSignatures option to strip manifest list signatures while preserving per-instance signatures when using CopySpecificImages with StripSparseManifestList. This implements "Option 2" from PR containers#655. Previously, users had to use RemoveSignatures which removed all signatures, including per-instance ones that would still be valid after stripping the manifest list. The option requires explicit validation: - Only valid with CopySpecificImages + StripSparseManifestList - Validates at multiple points in copy.Image() and copyMultipleImages() - When used with signed images, requires explicit signature handling Per-instance signatures are preserved because copySingleImage handles each instance independently. Signed-off-by: Alex Guidi <aguidi@redhat.com>
9889bbf to
586fc62
Compare
image/copy/multiple_test.go
Outdated
|
|
||
| // TestStripOnlyListSignaturesValidation tests the validation logic for StripOnlyListSignatures | ||
| // by actually calling copy.Image() with various option combinations. | ||
| func TestStripOnlyListSignaturesValidation(t *testing.T) { |
There was a problem hiding this comment.
I really didn’t mean to ask for the in-memory test harness here.
It was more of… “if there is no plausible unit test, it is acceptable to leave the code untested here instead of writing thousands of lines”.
I would actually like to ask for an added test — but not here. At least one integration test in the Skopeo repo, perhaps exercising the platform filtering and sparse creation together. (Integration tests in Skopeo are where we run the top-level skopeo copy commands, and where we have local registries, other infrastructure, and license to take more than a few hundred milliseconds.) That would, though, mean also wiring up the new features as Skopeo options, which might not be in scope — so, non-blocking as well.
| } | ||
| single, err := c.copySingleImage(ctx, singleInstance, nil, copySingleImageOptions{requireCompressionFormatMatch: requireCompressionFormatMatch}) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("copying system image from manifest list: %w", err) |
There was a problem hiding this comment.
Hum, we probably want this context (at least Skopeo defaults to CopySystemImage, and users might not realize that). That’s a bit awkward (singleInstanceErrorWrapping?), but on balance, consolidating the singleInstance code paths is probably still worth it.
There was a problem hiding this comment.
I’m sorry, I by “this context” I meant “system image from manifest list”.
586fc62 to
ab225b6
Compare
image/copy/multiple.go
Outdated
| cannotModifyManifestListReason = "Would invalidate signatures" | ||
| // If RemoveListSignatures is set, we allow modification of the manifest list | ||
| // by stripping only the list-level signature while preserving per-instance signatures. | ||
| if !c.options.RemoveListSignatures { |
There was a problem hiding this comment.
Now it looks like we might not set cannotModifyManifestListReason at all if RemoveListSignatures?!
OTOH len(sigs) > 0 implies this if being always true (because sourceSignatures would have filtered the signatures out otherwise).
I’d prefer removing this if and setting the …Reason always. That should be equivalent to the logic in this PR, simpler, and if something broke in the future and we ended up with some signatures, we should fail and not silently continue.
image/copy/multiple.go
Outdated
| // If RemoveListSignatures is set, we allow modification of the manifest list | ||
| // by stripping only the list-level signature while preserving per-instance signatures. |
There was a problem hiding this comment.
(This comment doesn’t quite need to exist here…)
image/copy/multiple.go
Outdated
| slices.Reverse(deleteOps) | ||
| copyLen := len(res) // Count copy/clone operations before appending deletes | ||
|
|
||
| // Validate that we're not deleting all entries without copying any |
image/copy/multiple.go
Outdated
|
|
||
| // Validate that we're not deleting all entries without copying any | ||
| if copyLen == 0 && len(deleteOps) == len(instanceDigests) { | ||
| return nil, -1, fmt.Errorf("cannot delete all instances from manifest list without copying any") |
There was a problem hiding this comment.
… well we can do that :)
something like “requested operation filtered out all platforms and would create an empty image”?
image/copy/copy.go
Outdated
| if multiImage && c.options.ImageListSelection == CopySystemImage { | ||
| return nil, fmt.Errorf("copying system image from manifest list: %w", err) | ||
| } | ||
| return nil, fmt.Errorf("copying single image: %w", err) |
There was a problem hiding this comment.
“copying single image” is not useful to users, in that case I think no wrapping is better.
I’d prefer to avoid the remote dependency on the “copying system image” message:
if !multiImage {
singleInstance = …
singleInstanceErrorWrapping = ""
} else if … SystemImage… {
singleInstance = …
singleInstanceErrorWrapping = "copying system image from manifest list"
}
// …
if …ErrorWrapping != "" { err = fmt.Errorf("%s: %w", …, …) }Yes that’s ugly, but it won’t get out of sync.
image/copy/multiple_test.go
Outdated
| } | ||
|
|
||
| instancesToCopy, err := prepareInstanceCopies(list, sourceInstances, &Options{}) | ||
| instancesToCopy, _, err := prepareInstanceOps(list, sourceInstances, &Options{}, "") |
There was a problem hiding this comment.
Please add assertions for the copy count, throughout.
Add the ability to strip non-copied instances from manifest lists when using CopySpecificImages. This implements the functionality originally proposed in containers/image#1707. When copying a subset of images from a multi-architecture manifest list using CopySpecificImages, the current behavior always produces a sparse manifest list - a manifest list that references platforms that weren't actually copied. While this maintains the original digest, many registries reject sparse manifests with "blob unknown to registry" errors. This commit adds a new SparseManifestListAction option that gives users control over this behavior: - KeepSparseManifestList (default): preserves existing behavior - StripSparseManifestList (new): removes non-copied instances The implementation includes: 1. New SparseManifestListAction enum in image/copy 2. New ListOpDelete operation in image/internal/manifest with support for both OCI1Index and Schema2List formats 3. Index-based deletion (not digest-based) to handle platform variants that share the same digest 4. Stripping logic in copyMultipleImages that activates when StripSparseManifestList is set 5. Comprehensive test coverage for deletion operations Based on original work by @bertbaron and @mtrmac in containers/image#1707, adapted for the container-libs monorepo structure. Relates to containers#227 Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove 'orthogonal' emphasis from RemoveListSignatures comment. API consumers should assume options don't interact unless documented. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove redundant comment about single vs multiple image copying. The code logic with singleInstance variable is self-explanatory. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Restore inline comment for ChooseInstanceByCompression. Clarifies that SourceCtx is used to pick matching instance. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove redundant 'Single-image copy path' comment. The condition and variable name are self-documenting. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove redundant 'Multi-image copy path' comment. The else block context makes this self-evident. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Add validation to reject RemoveListSignatures in single instance mode. RemoveListSignatures only applies when copying manifest lists. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Return -1 instead of 0 on error paths in prepareInstanceOps. Makes it more obvious if the count is accidentally used despite an error. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove redundant comment about delete operation. The condition and debug text are self-explanatory. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Refactor signature validation per mtrmac's suggestion. Move validation into prepareInstanceOps where delete operations occur. Simplify error handling by removing string matching in consumer code. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Add error context for single image copy. Helps users understand when copying single instance from manifest list. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove unused ociManifestFile constant from tests. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Fix gofumpt formatting by using explicit octal prefix (0o644 instead of 0644) for file permissions in os.WriteFile calls Signed-off-by: Alex Guidi <aguidi@redhat.com>
The comment stated 'This is handled below by setting sigs to empty' but the code that set sigs to empty has been refactored away. Signed-off-by: Alex Guidi <aguidi@redhat.com>
The condition checking for exact string 'Would invalidate signatures' is unreachable because the actual value set is 'Would invalidate signatures; consider removing them from the multi-platform list'. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Fail if we would delete all instances from a manifest list without copying any, as this would result in an invalid empty manifest list. Signed-off-by: Alex Guidi <aguidi@redhat.com>
When copying a system-specific image selected from a manifest list, the error message now says 'copying system image from manifest list' to help users understand what operation failed. This is especially important since Skopeo defaults to CopySystemImage. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Remove nested condition checking RemoveListSignatures. The outer check len(sigs) > 0 is sufficient since sourceSignatures filters out signatures when RemoveListSignatures is set. This simplification makes the logic clearer and ensures we fail loudly if something breaks in the future. Signed-off-by: Alex Guidi <aguidi@redhat.com>
The comment explaining why we add to compressionList is unnecessary as the code is self-explanatory in context. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Use a more user-friendly error message that explains the operation filtered out all platforms rather than the technical detail about deletion. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Set error wrapping string when determining single instance to keep it in sync. Remove unhelpful 'copying single image' wrapper and return the underlying error directly for plain single image copies. Signed-off-by: Alex Guidi <aguidi@redhat.com>
Signed-off-by: Alex Guidi <aguidi@redhat.com>
The comment is redundant as the code and error message are self-explanatory. Signed-off-by: Alex Guidi <aguidi@redhat.com>
4747944 to
6d9fd31
Compare
Add option to strip sparse manifest lists when copying specific images
This PR adds the ability to strip non-copied instances from manifest lists when using
CopySpecificImages, implementing the functionality proposed in containers/image#1707.Motivation
Closes #227
When copying a subset of images from a multi-architecture manifest list using
CopySpecificImages, the current behavior always produces a sparse manifest list - a manifest list that references platforms that weren't actually copied. While this maintains the original digest, many registries reject sparse manifests with "blob unknown to registry" errors.This PR adds a new
SparseManifestListActionoption that gives users control over this behavior:KeepSparseManifestList(default, existing behavior) - keeps all references even for non-copied imagesStripSparseManifestList(new) - removes non-copied instances and generates a manifest list with only copied instancesThis is particularly important for use cases like selective mirroring where only specific architectures need to be copied to registries that don't support sparse manifests.
Changes
1. New
SparseManifestListActionoption inimage/copySparseManifestListActionenum with two values:KeepSparseManifestList(default): preserves existing behavior - keeps all manifest referencesStripSparseManifestList(new): removes non-copied instances from the manifest listOptionsstruct to give users explicit control2. New
ListOpDeleteoperation inimage/internal/manifestListEditwithListOpDeleteoperation andDeleteIndexfield to support removing instancesOCI1IndexandSchema2List3. Sparse manifest stripping in
copyMultipleImages(image/copy/multiple.go)SparseManifestListAction == StripSparseManifestListEditInstancesAPI withListOpDelete4. Comprehensive test coverage
TestListDeleteInstancescovering both OCI and Docker manifest formatsTesting
All existing tests pass. New tests verify:
Credits
This implementation is based on the original work by @bertbaron and @mtrmac in containers/image#1707, adapted for the container-libs monorepo structure with the following changes:
manifestpackage tointernal/manifest(per container-libs directory structure)EditInstancesAPI pattern used throughout the codebase