Dev tools · alternatives · 2026
24 alternatives to Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
This experiment demonstrates a novel compression technique that trains a small 900KB transformer model to memorize individual files rather than generalize across data. The transformer predicts each successive byte… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
2025 · github.com · its alternatives →
- 2
- 3
Paritok▲258Spend up to 85% less and run 3× longer coding agent sessions
Aug 2026 · paritok.com · its alternatives →
- 4WS
I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…
2022 · xethub.com · its alternatives →
- 5
We combine Generative Adversarial Networks with learned compression to obtain a state-of-the-art generative lossy compression system.
2020 · hific.github.io · its alternatives →
- 6
CompressImage.io▲261Reduce image size up to 90% in seconds, free & works offline
2022 · compressimage.io · its alternatives →
- 7
I've spent the last few months working on this codec. It has the following characteristics: - SOTA decompression throughput in its ratio class - Decent ratios (comparable to LZ4 at high effort levels) - Slow compression Most of the gains can be attributed to reducing branches and making decompression very friendly to out-of-order cores, by using a smart format. Results on the tarred Silesia corpus on Intel x86-64 follow: codec decode ratio encode misa77 -0 5219 MB/s 42.64% 54.5 MB/s misa77 -1 4274 MB/s 39.65% 51.2 MB/s lz4 2505 MB/s 47.59% 371 MB/s lz4hc -12…
Jul 2026 · github.com · its alternatives →
- 8AT
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
Aug 2026 · mikeayles.com · its alternatives →
- 9
- 10
It's our new text-to-image model: a 9.3B single-stream diffusion transformer trained entirely from scratch. We focused heavily on controllability through structured JSON prompts, with strong text rendering, spatial awareness through bounding box guidance, and color palette control. It has the best text rendering of any open-weight model we've tested so far, and the NF4 quantized checkpoint runs on a single 24GB GPU. For more technical details and examples see our blog post: https://ideogram.ai/blog/ideogram-4.0/ We will be happy to answer any questions :)
Jun 2026 · github.com · its alternatives →
- 11CA
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Jan 2026 · alexkranias.com · its alternatives →
- 12
2020 · colab.research.google.com · its alternatives →
- 13
I have posted a much earlier version of this over a year ago [0]. Since then a lot has changed. Obviously, the name has changed. This happened for the paper publication [1]. I have also optimized the speed, integrated ISA-L for special cases, limited the compression-ratio-dependent maximum memory consumption, and finally added parallelized CRC32 computation, which adds ~5% overhead no matter the number of cores used. At this point, I am leaning towards calling it production-ready although there are still many ideas for improvements. Redoing the benchmarks of the older Show HN, would look…
2023 · github.com · its alternatives →
- 14
Contribute to seiflotfy/pcsa development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 15
Experiments that involve encoding arbitrary data into video files that survive Youtube compression. - ianling/steg-experiments
2025 · github.com · its alternatives →
- 16
Compress & resize your large video files to save disk space
2020 · play.google.com · its alternatives →
- 17
This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is. It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one…
2023 · github.com · its alternatives →
- 18
Properties of a Novel Binary Representation of Integers using Base-𝜙 - irradix/paper.md at main · DOSAYGO-STUDIO/irradix
2024 · github.com · its alternatives →
- 19KC
If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a single notebook—covering tokenization, embeddings, causal self-attention, training, and autoregressive text generation without relying on high-level abstractions. The focus was on mechanistic clarity rather than scale or performance, and the notebook is structured to read more like a technical walkthrough than an experiment log. Feedback from people who’ve built or studied transformers would be very welcome.
May 2026 · pythongiant.github.io · its alternatives →
- 20
Embeddable minimal in-memory GZIP decompression library for embedded systems - wkoszek/mini_gzip
2015 · github.com · its alternatives →
- 21
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
2024 · github.com · its alternatives →
- 22
Compress.io▲10Compresses images up to 90% file size reduction
2017 · compressor.io · its alternatives →
- 23
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev · its alternatives →
- 24
Compress 3D files in seconds. No quality loss.
2025 · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →