Life & fun · alternatives · 2026
24 alternatives to Compressing a Histogram from 16GB to 2KB
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026 · its alternatives →
- 2
Cool trick to store 4 5-bit values in one 16-bit value. - jaamesd/BucketCompressionTrick
2018 · github.com · its alternatives →
- 3
Optimal Quantile Approximation in Streams. Contribute to axiomhq/quantiles development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 4
While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
2025 · github.com · its alternatives →
- 5
Properties of a Novel Binary Representation of Integers using Base-𝜙 - irradix/paper.md at main · DOSAYGO-STUDIO/irradix
2024 · github.com · its alternatives →
- 6
MB2kB▲108Compress image or PDF to any specific size from MB to kB
2023 · mb2kb.com · its alternatives →
- 7
The core question: how did HP's scientific calculators actually work at the gate level? That rabbit hole led to building one from scratch. The architectural decision everything else follows from: a decimal calculator should store numbers as BCD — one decimal digit per 4-bit nibble. A standard byte-oriented CPU (Z80, 6502) fights that layout constantly. So I designed a small custom CPU in Verilog where 4 bits is the natural data width and memory is nibble addressable. What the project covers: - Custom CPU: Harvard architecture, 12-bit ISA, 8-state execution FSM, hardware stack guard with a…
May 2026 · github.com · its alternatives →
- 8
Contribute to seiflotfy/pcsa development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 9
repo of my sizecoding entries. Contribute to ilmenit/sizecoding development by creating an account on GitHub.
2024 · github.com · its alternatives →
- 10
Probabilistic data structures for processing continuous, unbounded streams. - tylertreat/BoomFilters
2015 · github.com · its alternatives →
- 11
Long-simmering side project that is finally ready to see the light. HAMTs are a cool persistent data structure and implementing one has been a lot of fun. Beyond the code, there is likely some value in the extensive and largely complete implementation docs; basic benchmarks are linked in the README, too. Kind of aiming to be "the libavl for HAMTs". That is obviously a high and aspirational bar but a distinct possibility if it stirs up a little interest and/or contribution. Anyways, it's time for this to go out, collect feedback and maybe even some use outside of toy projects. Let me…
2023 · github.com · its alternatives →
- 12
MemStream: Memory-Based Streaming Anomaly Detection - Stream-AD/MemStream
2021 · github.com · its alternatives →
- 13
I’ve spent the last year developing a decomposition kernel that addresses the "Linear Bottleneck" in standard compression (Zstd, LZMA, Brotli). The Problem: Traditional algorithms operate on 1D streams. They are blind to the spatial, temporal, or hierarchical architecture of modern data (3D models, nested JSON, multi-modal blobs). When you force these structures into a 1D window, you leave significant entropy on the table. The Solution: PXG v3.0 maps binary data to a 13505x13505 grayscale grid. By treating data as a coordinate system rather than a line, our engine utilizes adaptive tiling to…
2025 · limitpixel.com · its alternatives →
- 14
Estimate video archive size for video surveilance systems
2022 · jvsg.com · its alternatives →
- 15
Bitwise operations and conversions to hexadecimals
2016 · flippybitandtheattackofthehexadecimalsfrombase16.com · its alternatives →
- 16
This paper has been accepted to KDD2025.
2025 · github.com · its alternatives →
- 17
A very fast scalar implementation for Frame Of Reference integer compression. - cruppstahl/libfor
2015 · github.com · its alternatives →
- 18
Embeddable minimal in-memory GZIP decompression library for embedded systems - wkoszek/mini_gzip
2015 · github.com · its alternatives →
- 19
This is an upgrade of a tool I created 15 years ago in Python to learn OOP and solve some inadequacies in the HDR stacking tools I could find at the time. The problem was, none of them were really "batch friendly". None of them properly preserved the metadata I wanted them to stuff into the output file. There were probably some other reasons also, I just can't remember them now. It got the job done, but was very slow. Python was what I knew at the time and even with NumPy, I was limited in the speed I could squeeze out of it. (I also made some very specific, conscious, architectural choices…
Jun 2026 · github.com · its alternatives →
- 20
I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library. The part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware.…
2023 · ashvardanian.com · its alternatives →
- 21
I'd like to share a little toy project of mine, a really simple image codec that can do lossy to full lossless image compression with complete scalability at a byte level granularity - you can compress an image just once, even fully losslessly if needed, and then get any lossy version possible by simply stopping decompression at any offset in the compressed data. This "encode onde, serve many" approach is especially interesting for providing downscaled low quality image previews (LQIP) in as tight a storage budget as possible, and then allowing seamless, transparent refinement as deemed…
2024 · github.com · its alternatives →
- 22
A simple re-implementation of Google's sparsehash as a learning excercise. - qpfiffer/Simple-Sparsehash
2015 · github.com · its alternatives →
- 23
Author: Danison Nuñez Project: Coderive Project Type: Programming Language Highlight: Formula-based Execution ## Coderive - Iterating Through 1 Quintillion in a Loop *Subtitle: How a phone interpreter achieves what supercomputers cannot* The Impossible Loop: ```java // In any other language, this would be computational suicide for i in [0 to 1Qi] { // 1,000,000,000,000,000,000 iterations arr[i] = i * i } ``` Traditional Reality: · Python: MemoryError at array creation · Java/C++: Theoretical 31 years (with 8 exabytes of RAM) · NumPy/TensorFlow: Immediate crash ·…
Dec 2025 · github.com · its alternatives →
- 24
Compression suite for data frames and tabular data files, csv, excel etc. Using LZHW algorithm. - MNoorFawi/lzhw
2024 · github.com · its alternatives →
Also compare
- Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB alternatives
- How to store a set of four 5-bit values in one 16-bit value alternatives
- A Go implementation of TensorFlow's streaming quantiles estimator alternatives
- I compressed 10k PDFs into a 1.4GB video for LLM memory alternatives
- Outperforming VByte for Large Integers Using Phi-Encoding alternatives
- MB2kB alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →