Compressing a Histogram from 16GB to 2KB
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Compressing a Histogram from 16GB to 2KB launched on January 15, 2026 with 5 votes, #3,400 of 14,584 launches that month and more than 75% of that year's launches. The site was not reachable when checked in September 2026.
Does a similar job
all alternatives →- OAOverfitted a 900KB Transformer to Compress a 100MB CSV into 7MBJun 2026 · ▲112
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
How to store a set of four 5-bit values in one 16-bit value2018 · github.com · ▲188Cool trick to store 4 5-bit values in one 16-bit value. - jaamesd/BucketCompressionTrick
A Go implementation of TensorFlow's streaming quantiles estimator2018 · github.com · ▲98Optimal Quantile Approximation in Streams. Contribute to axiomhq/quantiles development by creating an account on GitHub.
I compressed 10k PDFs into a 1.4GB video for LLM memory2025 · github.com · ▲61While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
Outperforming VByte for Large Integers Using Phi-Encoding2024 · github.com · ▲15Properties of a Novel Binary Representation of Integers using Base-𝜙 - irradix/paper.md at main · DOSAYGO-STUDIO/irradix
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 2d ago · permit.petercai.com
- IV
Life & fun · 14d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 15d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 9d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 13d ago · carrierexplode.com
Life & fun · 23d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, January 2026
the whole month →- IN
Hey HN! I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents. I didn't write a single line of code. Of course no-code doesn't mean no-engineering. This project took a lot more manual labor than I'd hoped! I wrote a deep dive on the workflow and some thoughts about the future of AI coding and creativity: http://cannoneyed.com/projects/isometric-nyc
AI · Jan 2026 · cannoneyed.com
Automatic AI-powered code reviews the moment you open a PR
Dev tools · Jan 2026 · kilo.ai
