copy: Add text-based progress output for non-TTY environments (skopeo#658) by oshe-gi · Pull Request #665 · containers/container-libs

oshe-gi · 2026-02-18T10:18:55Z

Problem

When copying images in non-TTY environments (CI/CD pipelines, redirected output, piped commands), the visual mpb progress bars are discarded via io.Discard, leaving users with no visibility into transfer progress. This makes it difficult to:

Detect stalled transfers in CI/CD pipelines
Monitor progress of long-running copies
Distinguish between a slow transfer and a hung process

Currently, only a single "Copying blob..." message is printed via printCopyInfo() when non-TTY, with no subsequent progress updates.

Solution

This change automatically enables text-based progress output for non-TTY environments by leveraging the existing Options.Progress channel mechanism.

Design decisions

Automatic for non-TTY: When output is not a TTY and the caller hasn't already provided a buffered Progress channel, we automatically set up text-based progress output. No opt-in flag needed.
Aggregate progress instead of per-blob: Rather than printing a line for each blob (which would be verbose for multi-layer images), we track total bytes across all blobs and print a single aggregate progress line. This keeps CI logs clean while still providing visibility.
Reuse existing Progress channel: The progressReader in blob.go already sends ProgressEventNewArtifact, ProgressEventRead, and ProgressEventDone events. We simply consume these events and format them as text output.
Buffered channel: We create a buffered channel to prevent blocking senders during parallel blob downloads. Callers who need custom consumption should provide a properly buffered channel.
Sensible interval defaults: If ProgressInterval is not set, we default to 500ms. If the caller sets a larger interval, we respect that. The interval is clamped to a minimum of 500ms to prevent log spam.

Output example

Progress: 13.1 MiB / 52.3 MiB
Progress: 26.2 MiB / 52.3 MiB
Progress: 52.3 MiB / 52.3 MiB

Files changed

copy/progress_nontty.go - New file with nonTTYProgressWriter implementation
copy/copy.go - Initialize text progress writer in non-TTY mode

Testing

Build and unit tests:

go build ./copy/...  # ✓
go test ./copy/...   # ✓ all existing tests pass

Manual testing with skopeo (pipe to cat to force non-TTY):

go run ./cmd/skopeo \
  --policy ./default-policy.json \
  --override-os linux \
  --override-arch amd64 \
  copy \
  --image-parallel-copies 8 \
  docker://docker.io/library/golang:1.24-bookworm \
  dir:/tmp/golang-copy | cat
Getting image source signatures
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:89edcaae7ec479668d9bf0891145726173a305c809a8c4165723ceaf15b5a05f
Copying blob sha256:bbceb003542957cee7df7b79249eaf0a71d21c5203d086969b565fb6dec85d86
Copying blob sha256:d1ae0a59862a60fde488eccaa7f764b5f5cb60746b7adf2335f9cc05ce1ed745
Copying blob sha256:6bc9f599b3efabc64230fd3b969d7654fcd6c6c98ad7cf7470093fe85274a7fc
Copying blob sha256:ee9e6246b78f3a784fefa655e89ccdf2271e396000b7624d6a785cb2c2580001
Copying blob sha256:f7bdfd728ac2ad72d43b82689890dc698260d3a1049845f48fb3fb942df6c581
Progress: 1.5MiB / 271.3MiB
Progress: 5.8MiB / 294.2MiB
Progress: 10.9MiB / 294.2MiB
Progress: 15.7MiB / 294.2MiB
Progress: 20.8MiB / 294.2MiB
Progress: 26.3MiB / 294.2MiB
Progress: 31.7MiB / 294.2MiB
Progress: 37.9MiB / 294.2MiB
Progress: 43.6MiB / 294.2MiB
Progress: 49.9MiB / 294.2MiB
Progress: 56.0MiB / 294.2MiB
Progress: 62.3MiB / 294.2MiB
Progress: 68.6MiB / 294.2MiB
Progress: 75.0MiB / 294.2MiB
Progress: 80.7MiB / 294.2MiB
Progress: 87.2MiB / 294.2MiB
Progress: 91.6MiB / 294.2MiB
Progress: 95.4MiB / 294.2MiB
Progress: 98.1MiB / 294.2MiB
Progress: 101.5MiB / 294.2MiB
Progress: 104.9MiB / 294.2MiB
Progress: 108.8MiB / 294.2MiB
Progress: 112.9MiB / 294.2MiB
Progress: 117.0MiB / 294.2MiB
Progress: 120.5MiB / 294.2MiB
Progress: 125.1MiB / 294.2MiB
Progress: 129.7MiB / 294.2MiB
Progress: 134.6MiB / 294.2MiB
Progress: 139.6MiB / 294.2MiB
Progress: 144.6MiB / 294.2MiB
Progress: 149.6MiB / 294.2MiB
Progress: 154.7MiB / 294.2MiB
Progress: 160.1MiB / 294.2MiB
Progress: 165.8MiB / 294.2MiB
Progress: 172.1MiB / 294.2MiB
Progress: 177.9MiB / 294.2MiB
Progress: 184.2MiB / 294.2MiB
Progress: 190.3MiB / 294.2MiB
Progress: 196.4MiB / 294.2MiB
Progress: 203.4MiB / 294.2MiB
Progress: 210.8MiB / 294.2MiB
Progress: 218.8MiB / 294.2MiB
Progress: 227.1MiB / 294.2MiB
Progress: 236.7MiB / 294.2MiB
Progress: 243.8MiB / 294.2MiB
Progress: 249.7MiB / 294.2MiB
Progress: 255.9MiB / 294.2MiB
Progress: 262.3MiB / 294.2MiB
Progress: 270.8MiB / 294.2MiB
Progress: 284.7MiB / 294.2MiB
Copying config sha256:e0cffc405270b9114fac7706d07c373727d1b42b0e47c525b9cd1ab1097779ff
Writing manifest to image destination

Output may be shorter, if ProgressInterval is set longer.
Signed-off-by: Oleksandr Shestopal ar.shestopal@gmail.com

TomSweeneyRedHat · 2026-02-25T21:29:03Z

image/copy/progress_nontty.go

+// setupNonTTYProgressWriter configures text-based progress output for non-TTY
+// environments unless the caller already provided a buffered Progress channel.
+// Returns a cleanup function that must be deferred by the caller.
+// It relies on I dea that options.Progress channel is ony used once, to track progress with progress bar


Suggested change

// It relies on I dea that options.Progress channel is ony used once, to track progress with progress bar

// It relies on the idea that options.Progress channel is only used once, to track progress with a progress bar

TomSweeneyRedHat · 2026-02-25T21:29:23Z

image/copy/progress_nontty.go

+// environments unless the caller already provided a buffered Progress channel.
+// Returns a cleanup function that must be deferred by the caller.
+// It relies on I dea that options.Progress channel is ony used once, to track progress with progress bar
+// Otherwize we must do some sort of fan-out


Suggested change

// Otherwize we must do some sort of fan-out

// Otherwise we must do some sort of fan-out

TomSweeneyRedHat · 2026-02-25T21:42:12Z

@oshe-gi, your test code needs some formatting to pass our lint tests.

  Error: /home/runner/work/container-libs/container-libs/image/copy/progress_nontty_test.go:9:1: File is not properly formatted (gofumpt)
  	"go.podman.io/image/v5/types"

try gofmt -s -w progress_nontty_test.go, then git add, commit, fetch, rebase, push.

oshe-gi · 2026-02-27T16:44:48Z

Thanks for comments @TomSweeneyRedHat , done.

TomSweeneyRedHat · 2026-02-27T18:48:20Z

@mtrmac the Skopeo tests is failing with:

go: github.com/containers/skopeo/cmd/skopeo imports
	go.podman.io/image/v5/pkg/cli/basetls/tlsdetails:

Do we need to vendor the latest c/image into Skopeo and create a release to make this test happy again?

TomSweeneyRedHat · 2026-02-27T18:49:45Z

Changes LGTM once we can make the tests happy. I don't think they're failing due to these changes.

mtrmac · 2026-02-27T18:50:29Z

the Skopeo tests is failing

Just rebasing this PR on top of current main should help. (I’m afraid I didn’t otherwise read the PR.)

When copying images in non-TTY environments (CI/CD pipelines, redirected output, piped commands), the visual mpb progress bars are discarded, leaving users with no visibility into transfer progress. This makes it difficult to detect stalled transfers or monitor long-running copies. This change adds a nonTTYProgressWriter that consumes progress events from the existing Progress channel and prints periodic aggregate progress lines suitable for log output: Progress: 13.1 MiB / 52.3 MiB Progress: 26.2 MiB / 52.3 MiB Progress: 52.3 MiB / 52.3 MiB The feature is enabled when, output is not a TTY, we check if option.Progress is set, otherwise create a new buffered channel for progress events. Note: Unbuffered channels are replaced with buffered ones to prevent blocking during parallel blob downloads. Callers who need custom consumption should provide a properly buffered channel. Relates-to: containers/skopeo#658 Signed-off-by: Oleksandr Shestopal <ar.shestopal-oshegithub@gmail.com>

oshe-gi · 2026-02-28T14:40:35Z

@mtrmac @TomSweeneyRedHat rebased on latest main, please approve validation CI

mtrmac

Thanks!

Just a fairly brief look now.

mtrmac · 2026-03-06T00:24:13Z

image/copy/copy.go

 	// If reportWriter is not a TTY (e.g., when piping to a file), do not
 	// print the progress bars to avoid long and hard to parse output.
-	// Instead use printCopyInfo() to print single line "Copying ..." messages.
+	// Instead use text-based aggregate progress via nonTTYProgressWriter.


None of the existing output to reportWriter was removed; that’s probably good but the comment is now misleading.

mtrmac · 2026-03-06T00:26:08Z

image/copy/progress_nontty.go

+
+// setupNonTTYProgressWriter configures text-based progress output for non-TTY
+// environments unless the caller already provided a buffered Progress channel.
+// Returns a cleanup function that must be deferred by the caller.


mtrmac · 2026-03-06T00:29:50Z

image/copy/progress_nontty.go

+		options.ProgressInterval = nonTTYProgressInterval
+	}
+
+	if options.Progress == nil || cap(options.Progress) == 0 {


What does cap have to do with anything?!

I suppose options.Progress to be buffered to prevent blocking.

If the caller of copy.Image does submit a progress channel but doesn’t make it buffered for whatever reason (confident in not blocking the copy, or just forgot), I don’t think that’s a reason to entirely ignore the caller’s progress channel and to use the text progress reporting instead.

mtrmac · 2026-03-06T00:31:12Z

image/copy/progress_nontty.go

+	// This allows users to slow down output while maintaining a sensible minimum.
+	interval := max(options.ProgressInterval, nonTTYProgressInterval)
+	if options.ProgressInterval <= 0 {
+		options.ProgressInterval = nonTTYProgressInterval


Why does this touch the option if it is not going to consume the channel?

And if it is going to consume the channel, why does it make sense to set the consumers’ and the producers’ intervals to different values? (Should there even be two?)

With interval user can change how often he want to see the update, if updates are too often, it will produce long output in non-TTY. If it is not provided or too small - set the default.

I did not find where we set ProgressInterval as well

c/image/copy.Image is a public API; it has primary callers in Podman and friends, but there are users outside of the GitHub.com/containers namespace.

So then I think we need to make sure we initialize ProgressInterval and Progress channel in skopeo if nonTTY and no --quiet option is set.

mtrmac · 2026-03-06T00:32:14Z

image/copy/progress_nontty.go

+// aggregate progress. Blocks until the channel is closed. Intended to
+// be called as a goroutine: go tw.Run(progressChan)
+func (w *nonTTYProgressWriter) Run() {
+	for props := range w.progressChannel {


This is not going to ever terminate, AFAICS

mtrmac · 2026-03-06T00:36:11Z

image/copy/progress_nontty.go

+		switch props.Event {
+		case types.ProgressEventNewArtifact:
+			// New blob starting - add its size to total
+			w.totalSize += props.Artifact.Size


That’s not a very useful concept of a “total progress” if the total is increasing over time. And sometimes we trigger an upload but then never consume all of the source; if this is not watching ProgressEventDone, it might never reach 100%.

How does this interact with multi-platform images?

(Working one platform at a time would be easier in that we usually (not always!) know the size of the input in advance.)

The Idea of total progress is to give at least rough estimate, and short output.
What we can do is to wait when all parallel copying have started, and then start computing the progress.

Do you want to track per-blob progress?
Currently It will stop when copying is done, but I can work on processing ProgressEventDone events as well.
I have tested with golang image, there is an example in PR description. Is it a valid test?

@mtrmac this is how I tested.

--policy ./default-policy.json \ --override-os linux \ --override-arch amd64 \ copy \ --image-parallel-copies 8 \ docker://docker.io/library/golang:1.24-bookworm \ dir:/tmp/golang-copy | cat

Could you provide better/more ways to test?

What we can do is to wait when all parallel copying have started, and then start computing the progress.

We don’t start all layer copies simultaneously. Try a 20-layer image (large enough, or on a network slow enough, to be able to observe the progress.). Compare maxParallelDownloads.

We read the manifest before starting copies, and that usually (not always!) contains all of the layers’ sizes for the per-platform instance.

docker://docker.io/library/golang:1.24-bookworm is a reasonable image, but try copy --all

…ProgressInterval Relates-to: containers/skopeo#658 Signed-off-by: Oleksandr Shestopal <ar.shestopal-oshegithub@gmail.com>

github-actions bot added the image Related to "image" package label Feb 18, 2026

oshe-gi force-pushed the 658-text-progress-for-non-tty branch from 4879e54 to 6f11c00 Compare February 18, 2026 10:38

oshe-gi mentioned this pull request Feb 18, 2026

Please provide a basic progress / transfer speed output ("CI compatible") on copy containers/skopeo#658

Open

oshe-gi force-pushed the 658-text-progress-for-non-tty branch from 6f11c00 to 7feabb7 Compare February 21, 2026 20:18

oshe-gi changed the title ~~copy: Add text-based progress output for non-TTY environments (skopeo#658)~~ WIP copy: Add text-based progress output for non-TTY environments (skopeo#658) Feb 22, 2026

oshe-gi force-pushed the 658-text-progress-for-non-tty branch from 3d21a59 to dc20b30 Compare February 22, 2026 08:59

oshe-gi changed the title ~~WIP copy: Add text-based progress output for non-TTY environments (skopeo#658)~~ copy: Add text-based progress output for non-TTY environments (skopeo#658) Feb 22, 2026

TomSweeneyRedHat reviewed Feb 25, 2026

View reviewed changes

oshe-gi force-pushed the 658-text-progress-for-non-tty branch 2 times, most recently from dde7dd7 to 6b03cf7 Compare February 27, 2026 17:19

oshe-gi force-pushed the 658-text-progress-for-non-tty branch from 6b03cf7 to e5f2e34 Compare February 28, 2026 13:15

mtrmac requested changes Mar 6, 2026

View reviewed changes

Fix comments and set default values for options.Progress and options.…

fbd6770

…ProgressInterval Relates-to: containers/skopeo#658 Signed-off-by: Oleksandr Shestopal <ar.shestopal-oshegithub@gmail.com>

	// It relies on I dea that options.Progress channel is ony used once, to track progress with progress bar
	// It relies on the idea that options.Progress channel is only used once, to track progress with a progress bar

	// Otherwize we must do some sort of fan-out
	// Otherwise we must do some sort of fan-out

Conversation

oshe-gi commented Feb 18, 2026

Problem

Solution

Design decisions

Output example

Files changed

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomSweeneyRedHat commented Feb 25, 2026

Uh oh!

oshe-gi commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomSweeneyRedHat commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomSweeneyRedHat commented Feb 27, 2026

Uh oh!

mtrmac commented Feb 27, 2026

Uh oh!

oshe-gi commented Feb 28, 2026

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oshe-gi Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

oshe-gi commented Feb 27, 2026 •

edited

Loading

TomSweeneyRedHat commented Feb 27, 2026 •

edited

Loading

oshe-gi Mar 8, 2026 •

edited

Loading

mtrmac Mar 9, 2026 •

edited

Loading