Alternatives
Products that do what misa77 - a codec that decodes 2x faster than LZ4 (at better ratios) does
I've spent the last few months working on this codec. It has the following characteristics: - SOTA decompression throughput in its ratio class - Decent ratios (comparable to LZ4 at high effort levels) - Slow compression Most of the gains can be attributed to reducing branches and making decompression very friendly to out-of-order cores, by using a smart format. Results on the tarred Silesia corpus on Intel x86-64 follow: codec decode ratio encode misa77 -0 5219 MB/s 42.64% 54.5 MB/s misa77 -1 4274 MB/s 39.65% 51.2 MB/s lz4 2505 MB/s 47.59% 371 MB/s lz4hc -12…
- 1OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026
- 2SL
I'd like to share a little toy project of mine, a really simple image codec that can do lossy to full lossless image compression with complete scalability at a byte level granularity - you can compress an image just once, even fully losslessly if needed, and then get any lossy version possible by simply stopping decompression at any offset in the compressed data. This "encode onde, serve many" approach is especially interesting for providing downscaled low quality image previews (LQIP) in as tight a storage budget as possible, and then allowing seamless, transparent refinement as deemed…
2024 · github.com
- 3AI
2017 · unwttng.com
- 4
- 5
- 6FI
2015 · github.com
- 7SV
I wrote libcorrect in C in 2016 and wanted to revisit it in Rust. Instead of doing just a direct conversion, I went down the rabbit hole of making Rust's std::simd work for me. I ended up with a templated, generic Viterbi decoder for convolutional codes that dispatches the decode at runtime depending on which instruction sets are available. For small rates and orders, the entire decode lives in registers. Larger codes work through memory but take advantage of some acceleration structures. I also spent some time building a tool to find optimal (max d_free) conv codes for a given rate and…
Aug 2026 · github.com
- 8RG
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev
- 9TF
Mar 2026 · github.com
- 10OS
I’ve been working on a compression algorithm for fast random access to individual strings in large collections. The problem came up when working with large in-memory database columns (emails, URLs, product titles, etc.), where low-latency point queries are essential. With short strings, LZ77-based compressors don’t perform well. Block compression helps, but block size forces a trade-off between ratio and access speed. Some existing options: - BPE: good ratios, but slow and memory-heavy - FSST (discussed here: https://news.ycombinator.com/item?id=41489047): very fast, but…
2025 · github.com
- 11
This project is a little performance-optimization experiment focused on reducing the size of Go binary patches, which can be useful for quick incremental updates of live services. When a Go program grows by a few bytes through an incremental change, a bunch of related references and offsets throughout the compiled binary end up changing that makes general-purpose binary-diff encoders inefficient: `zstd --patch-from` produces a 530kb patch for a one-line change in a 30MB binary. bsdiff improves upon this (150KB patch) thanks to a delta compression algorithm that can handle relative offsets.…
10d ago · github.com
- 12
- 13AB
2019 · github.com
- 14FC
Mar 2026 · ory.com
- 15
25% faster while cutting token costs by 30%-67%
Jul 2026 · forsale.dynadot.com
- 16TE
2021 · github.com
- 17
The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 3
20d ago · github.com
- 18
I built a specialized package of DeepSeek V4 Flash 0731 (originally 284B total parameters, 13B active), preserving reasoning, tool calling and coding capabilities: https://huggingface.co/steadfastgaze/DeepSeek-V4-Flash-0731-... I let it write a minimal C compiler targeting ARM64, then test the result with Fibonacci and FizzBuzz programs, and it succeeded in less than 1 hour, with the full recording at: https://youtu.be/XiwSilmV8B0 You can run it on Silicon Macs with my engine https://github.com/steadfastgaze/MoEspresso, while one of the…
21d ago · huggingface.co
- 19AW
I've spent the past couple of weeks building a Wasm-to-Go translator. It supports a subset of Wasm useful enough to translate SQLite into 600k LoC (~20 MiB) of Go code. It already passes all of my Go SQLite driver's tests across the 20 platforms I support. Performance compared to https://wazero.io/ is a bit of a mixed bag: code that frequently crosses the Go-Wasm boundary improves, but code that spends most of its time in "Wasm land" doesn't. There's probably room for improvement (I'd love to hear your ideas), but this is also a testament to how good the wazero AOT compiler…
Feb 2026 · github.com
- 20
- 21SC
2018 · github.com
- 22CA
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Jan 2026 · alexkranias.com
- 23FN
"I wanted to see if I could optimize the dequantization bottleneck during 4-bit LLM inference. By writing a custom kernel in Triton to optimize memory access patterns, I managed to get up to a 1.41x speedup over the standard bitsandbytes implementation. Check out the source code and benchmarks, feedback is highly appreciated!"
Jul 2026 · github.com
- 24
Hi everyone, I was recently much frustrated with speed of k9s that I use regularly and decided to create a rust alternative that is much faster. Architecture comes from me, and (shocking) Fable coded most of the things. My whole company switched to it and experience has been great - much faster, integrated tightly with Flux/helm/externalsecrets and develops much faster. I know that there is an overall sentiment on HN with using LLMs, so YMMV. In terms of speed, you can check the benchmarks:…
18d ago · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →