Life & fun · September 4, 2023
Rapidgzip – Truly Parallel Gzip Decompression with 10 GB/s
I have posted a much earlier version of this over a year ago [0]. Since then a lot has changed. Obviously, the name has changed. This happened for the paper publication [1]. I have also optimized the speed, integrated ISA-L for special cases, limited the compression-ratio-dependent maximum memory consumption, and finally added parallelized CRC32 computation, which adds ~5% overhead no matter the number of cores used. At this point, I am leaning towards calling it production-ready although there are still many ideas for improvements. Redoing the benchmarks of the older Show HN, would look…
In plain words
Rapidgzip is a parallel gzip decompression tool that achieves decompression speeds of approximately 10 GB/s, significantly faster than traditional single-threaded alternatives. It is designed for users who need to decompress large gzip files efficiently, particularly those working with large datasets or in performance-critical environments. The tool features optimized speed through ISA-L integration, memory consumption controls, and parallelized CRC32 computation, making it suitable for production use.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I have posted a much earlier version of this over a year ago [0]. Since then a lot has changed. Obviously, the name has changed. This happened for the paper publication [1]. I have also optimized the speed, integrated ISA-L for special cases, limited the compression-ratio-dependent maximum memory consumption, and finally added parallelized CRC32 computation, which adds ~5% overhead no matter the number of cores used. At this point, I am leaning towards calling it production-ready although there are still many ideas for improvements. Redoing the benchmarks of the older Show HN, would look like this: time pigz -d -c 4GiB-base64.gz | wc -c # real ~13.4 s -> ~320 MB/s time rapidgzip -d -c 4GiB-base64.gz | wc -c # real ~1.26 s -> ~3.4 GB/s However, at this point, the piping itself becomes a problem. Rapidgzip is actually slightly faster than cat when comparing the piped bandwidth! E.g., compare these additional benchmarks: time cat 4GiB-base64.gz | wc -c # real ~1.06 s -> ~3.1 GB/s time fcat 4GiB-base64.gz | wc -c # real ~0.41 s -> ~8.0 GB/s time rapidgzip -o /dev/null -d 4GiB-base64.gz # real ~0.68 s -> ~6.5 GB/s fcat is an alternative cat implementation that uses vmsplice to speed up piping. According to the ReadMe it currently is broken, but it works fine on my system and piping it to md5sum yields consistent results [2]. So, at this point, I/O and actually also allocations have become a limiting factor and if you want full speed, you would have to interface with the rapidgzip library interface directly (in C++ or via the Python bindings) and process the decompressed data in memory. The project ReadMe contains further benchmarks with Silesia and FASTQ data and scaling up to 128 cores, for which rapidgzip achieves 12 GB/s for Silesia and 24 GB/s when an index has been created with --export-index and is used with --import-index. It can also be tested with ratarmount 0.14.0, which now uses rapidgzip as a backend by default for .gz and .tar.gz files [3]. [0] https://news.ycombinator.com/item?id=32366959 [1] https://dl.acm.org/doi/10.1145/3588195.3592992 [2] https://github.com/mre/fcat [3] https://github.com/mxmlnkn/ratarmount
More life & fun this month
the category →- TL
Life & fun · 10d ago · louisabraham.github.io
Photosynthesis fires two of your iPhone
Life & fun · 28d ago · photosynthesis.camera
SoloUno▲310Take control of hair pulling, nail biting & skin picking
Life & fun · 28d ago · solouno.io
Scroll through all 43,252,003,274,489,856,000 reachable Rubik's Cube permutations.
Life & fun · 26d ago · everycube.alen.is
Hi HN, I built Eigendrum, a web tool that solves the 2D wave equation for arbitrary shapes so you can hear what they sound like as drums. How it works: * Solves -∇²u = λu using finite element analysis (Kφ = λMφ) on a triangle mesh. * Validated to <0.1% error against closed-form solutions for circles (Bessel zeros) and rectangles. * Sound model factors in strike location, Rayleigh damping, and mallet width. * Includes Kac drums I & II to demonstrate identical sound spectra from different geometries. * No frameworks, build steps, or dependencies. Repo and tests:…
Life & fun · 27d ago · baselashraf81.github.io
