Initial reimplementation of composefs-c#225
Draft
cgwalters wants to merge 9 commits intocomposefs:mainfrom
Draft
Initial reimplementation of composefs-c#225cgwalters wants to merge 9 commits intocomposefs:mainfrom
cgwalters wants to merge 9 commits intocomposefs:mainfrom
Conversation
Collaborator
Author
|
There's definitely some sub-tasks to this and pieces that we need to break out. One that I'm realizing is that the dumpfile format is hardcoded to sha256-12. I guess we can just auto-detect from length (like we're doing in other places) but the more I think about this the more I feel we need to formalize it (as is argued in #224 ) So how about a magic comment in the dumpfile like or so? |
cgwalters
force-pushed
the
composefs-c-compat
branch
2 times, most recently
from
4d43b61 to
1871128
Compare
Collaborator
Author
|
Let's make the format layout a choice to avoid breaking sealed UKIs as is today |
cgwalters
force-pushed
the
composefs-c-compat
branch
3 times, most recently
from
8a5c48d to
9cb1923
Compare
This was referenced Mar 11, 2026
cgwalters
force-pushed
the
composefs-c-compat
branch
4 times, most recently
from
6eda766 to
dc1fed7
Compare
This was referenced Mar 17, 2026
The composefs-dump(5) spec leaves several fields unspecified or explicitly ignored. Canonicalize them at parse time so that parsed entries have a single canonical representation regardless of which implementation produced them: - **Directory sizes**: "This is ignored for directories." Drop the size field from Item::Directory, always emit 0. - **Hardlink metadata**: "We ignore all the fields except the payload." Zero uid/gid/mode/mtime and skip xattrs, matching the C parser which bails out early (mkcomposefs.c:477-491). - **Xattr ordering**: The spec doesn't define an order. Sort lexicographically so output is deterministic regardless of on-disk ordering. The parser still accepts any input values for backward compatibility. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
XFS limits symlink targets to 1024 bytes, and since generic Linux containers are commonly backed by XFS, enforce that limit in both the dumpfile parser and the EROFS reader rather than allowing up to PATH_MAX (4096). This also avoids exercising a known limitation in our EROFS reader where symlink data that spills into a non-inline data block (which can happen with long symlinks + xattrs) is not read back correctly. See composefs/composefs#342 for the corresponding C fix for that edge case. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
Increase alignment for dumpfile generation with the composefs C implementation - on general principle but also motivated by the goal of reimplementing it in Rust here. The C composefs implementation uses named escapes for backslash, newline, carriage return, and tab (\\ \n \r \t), while our writer was hex-escaping them uniformly (\x5c \x0a etc). Both forms parse correctly, but byte-identical output matters for cross-implementation comparison. Similarly, C only escapes '=' in xattr key/value fields (where it separates key from value). We were escaping it as \x3d in all fields including paths and content, where '=' is a normal graphic character. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
Add a FormatVersion enum (V1/V2) that controls the EROFS image format: V1 produces byte-identical output to C mkcomposefs. It sets composefs_version=0 in the superblock, uses compact inodes where possible, BFS inode ordering, C-compatible xattr sorting, and includes overlay whiteout character device entries in the root directory. The build_time is set to the minimum mtime across all inodes, matching the C implementation. V2 remains the default (composefs_version=2). It uses extended inodes, DFS ordering, and the composefs-rs native xattr layout. Key V1 writer differences from V2: - BFS (breadth-first) inode ordering vs DFS (depth-first) - Compact inodes when uid/gid fit in u16 and mtime == build_time - Xattr sorting by full key name for C compatibility - Overlay whiteout char devices (00-ff) added to root directory - trusted.overlay.opaque=y xattr on root directory Tests cover both format versions: insta snapshots, proptest round-trips, fsck validation, and byte-identical comparison against the C mkcomposefs tool. The fuzz corpus generator also produces both V1 and V2 seed images. Assisted-by: OpenCode (Claude Opus 4)
Make cfsctl a multi-call binary that dispatches based on argv[0]: when invoked as "mkcomposefs" or "composefs-info" (via symlink or hardlink), it runs the corresponding tool directly. This avoids separate binary crates while providing drop-in compatibility with the C composefs tools. mkcomposefs reads composefs-dump(5) format or directory trees and produces compatible EROFS images with v1.0/v1.1 format support. composefs-info inspects composefs/EROFS images: dumps filesystem trees, lists objects, and displays detailed inode/xattr info. The integration tests create a symlink from cfsctl to mkcomposefs for the multi-call dispatch rather than looking for a separate mkcomposefs binary. Assisted-by: OpenCode (Claude Opus 4)
The repository fsck tests only exercised V2 (Rust-native) EROFS images. Add tests that create V1 (C-compatible) images via mkfs_erofs_versioned and verify fsck handles them correctly — both for healthy images and for detecting missing referenced objects. Also add a V1 digest stability test alongside the existing V2 one, pinning the fsverity digests so any accidental change to V1 output (which must match C mkcomposefs) is caught immediately. Assisted-by: OpenCode (Claude Opus 4)
Document the current state of the C composefs reimplementation across the CLI tools, with specific TODO(compat) markers for each known gap. This makes it easy to grep for remaining work and understand what's implemented vs what's missing. Key gaps tracked: --use-epoch leaf mtimes, --threads, --digest-store path layout, --max-version auto-upgrade, mtime nanoseconds, and the C shared library (libcomposefs) which is the next major milestone. Also fixes an outdated comment that claimed compact inodes were not implemented (they are, and have been tested byte-for-byte against C mkcomposefs). Assisted-by: OpenCode (Claude Opus 4)
Generate random filesystem trees via proptest, write them as V1 and V2 EROFS images, feed the images to C composefs-info dump, and compare the output against our Rust reader's interpretation. Both V1 and V2 tests pass with 64 cases each. Comparison uses Entry::canonicalize() to normalize spec-permitted differences (hardlink metadata fields, xattr ordering) before comparing parsed entries. Also fix erofs_to_filesystem to skip overlay whiteout entries (chardev with rdev 0), matching the C reader behavior. These are internal composefs overlay machinery, not user-visible filesystem content. Assisted-by: OpenCode (Claude Opus 4)
Previously we accepted any composefs_version value, which means a future format change could be silently misinterpreted. Reject unknown versions and only accept the two known ones: - V1 (composefs_version=0): original C format - V2 (composefs_version=2): Rust-native format Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
composefs-c-compat
branch
from
dc1fed7 to
9a845fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically starting on composefs/composefs#423
3 key goals:
Assisted-by: OpenCode (Claude Sonnet 4)