bottomless: add zstd compression option#468
Conversation
|
as an env var |
|
@MarinPostma downgraded to draft, because I see some spontaneous errors when testing xz compression on large random data (one that does not compress well). I'll evalutate other compression implementations as well, because xz support looks a little abandoned in the async_compression crate |
During stress tests, xz turned out to spontaneously fail to compress, same with bzip2. All compression algos are supported by separate crates, so these were simply ruled out. Zstd proved to be: - fast - correct - more than acceptable on compression ratio
psarna
changed the title
|
Rebranded to zstd. It's a modern one that also powers ScyllaDB, and it first and foremost passed all correctness tests on which xz and bzip2 failed. It's also way faster than any of the other ones, while still providing compression ratios of 12-15 where gzip gives mere 2. |
Gzip does not perform well on data in form of libSQL 4KiB pages, and zstd performed uniformly better in all test cases I covered locally (and not worse in case of random data with super high entropy).
2 participants
Transplanted from libsql/sqld#780