Alternatives
Products that do what Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB does
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
- 1IC
While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
2025 · github.com
- 2AI
2017 · unwttng.com
- 3MA
I've spent the last few months working on this codec. It has the following characteristics: - SOTA decompression throughput in its ratio class - Decent ratios (comparable to LZ4 at high effort levels) - Slow compression Most of the gains can be attributed to reducing branches and making decompression very friendly to out-of-order cores, by using a smart format. Results on the tarred Silesia corpus on Intel x86-64 follow: codec decode ratio encode misa77 -0 5219 MB/s 42.64% 54.5 MB/s misa77 -1 4274 MB/s 39.65% 51.2 MB/s lz4 2505 MB/s 47.59% 371 MB/s lz4hc -12…
Jul 2026 · github.com
- 4AT
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
27d ago · mikeayles.com
- 5
- 6TA
2015 · ruarai.github.io
- 7I4
It's our new text-to-image model: a 9.3B single-stream diffusion transformer trained entirely from scratch. We focused heavily on controllability through structured JSON prompts, with strong text rendering, spatial awareness through bounding box guidance, and color palette control. It has the best text rendering of any open-weight model we've tested so far, and the NF4 quantized checkpoint runs on a single 24GB GPU. For more technical details and examples see our blog post: https://ideogram.ai/blog/ideogram-4.0/ We will be happy to answer any questions :)
Jun 2026 · github.com
- 8CA
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Jan 2026 · alexkranias.com
- 9NI
2020 · colab.research.google.com
- 10IM
Jan 2026 · github.com
- 11FI
2015 · github.com
- 12CG
I built CSV GB+ by Data.olllo, a local data tool that lets you open, clean, and export gigabyte-sized CSVs (even billions of rows) without writing code. Most spreadsheet apps choke on big files. Coding in pandas or Polars works—but not everyone wants to write scripts just to filter or merge CSVs. CSV GB+ gives you a fast, point-and-click interface built on dual backends (memory-optimized or disk-backed) so you can process huge datasets offline. Key Features: Handles massive CSVs with ease — merge, split, dedup, filter, batch export Smart engine switch: disk-based "V Core" or RAM-based "P…
2025 · apps.microsoft.com
- 13XC
2019 · github.com
- 14CR
Oct 2025 · github.com
- 15AB
2019 · github.com
- 16IC
This is an upgrade of a tool I created 15 years ago in Python to learn OOP and solve some inadequacies in the HDR stacking tools I could find at the time. The problem was, none of them were really "batch friendly". None of them properly preserved the metadata I wanted them to stuff into the output file. There were probably some other reasons also, I just can't remember them now. It got the job done, but was very slow. Python was what I knew at the time and even with NumPy, I was limited in the speed I could squeeze out of it. (I also made some very specific, conscious, architectural choices…
Jun 2026 · github.com
- 17DR
2018 · dropbox.github.io
- 18MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
2024 · github.com
- 19RG
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev
- 20TE
2021 · github.com
- 21TB
2015 · github.com
- 22SL
I'd like to share a little toy project of mine, a really simple image codec that can do lossy to full lossless image compression with complete scalability at a byte level granularity - you can compress an image just once, even fully losslessly if needed, and then get any lossy version possible by simply stopping decompression at any offset in the compressed data. This "encode onde, serve many" approach is especially interesting for providing downscaled low quality image previews (LQIP) in as tight a storage budget as possible, and then allowing seamless, transparent refinement as deemed…
2024 · github.com
- 23
- 24
Ranked by how close each launch is in meaning, then by votes. Refine with a description →