Why should we choose Zstd?

[Apoorv012]

Issue: #72

Compression Strategy Exploration

To reduce object store growth, multiple compression algorithms were evaluated instead of choosing one by convention. The goal was to find the best size vs speed tradeoff for storing full file snapshots.

Algorithms considered

• raw (no compression) — control
• gzip (DEFLATE / zlib)
• bzip2
• lzma (xz)
• zstandard (zstd)
• lz4

Benchmark Results

Command:

python -m src.cli hash .\diff.c

This is the diff.c from https://github.com/vim/vim/blob/master/src/diff.c

Result:

File hash: e86919b440bbff10d1f52d7e1de707e9d1e43d9ef7949fa9a304984b585d93c3
Original size: 129580 bytes

Algorithm | Time (ms) | Size (bytes) | Ratio
--------------------------------------------------
raw      |     0.00 |       129580 | 1.000
gzip     |     8.02 |        35836 | 0.277
bz2      |    16.00 |        29238 | 0.226
lzma     |    67.53 |        31344 | 0.242
zstd     |     5.26 |        34954 | 0.270
lz4      |     2.31 |        58674 | 0.453

Observations

• raw – baseline, no savings
• gzip – reasonable compression, slower, legacy choice
• bz2 – strong compression, too slow for interactive use
• lzma – very slow, high CPU cost
• lz4 – extremely fast, poor compression
• zstd – near-gzip compression with much better speed

Final Decision

Zstandard (zstd) offers the best overall tradeoff:

• Significant size reduction (~73%)
• Faster than gzip
• Designed as a modern replacement for gzip

Conclusion: zstd is the default compression algorithm for Ding’s object storage.

[OpenGitBot]

Hey @Apoorv012

Thanks for opening this PR 🚀. Mentor will review your pull request soon and till then, keep contributing and stay calm.

Thanks for contributing in OpenCode'25 ✨✨!

[Rational-Idiot]

Be sure to pull the main repo before making changes or you can get a detached head

[OpenGitBot]

Hey @Apoorv012

Your PR has been merged 🥳🥳 and you have earned 20 points.

Thanks for contributing in OpenCode'25✨✨