TurboQuant-MoE:8.5x KV-Cache Compression
8.5x KV-cache compression for LLM inference
TurboQuant-MoE:8.5x KV-Cache Compression launched on March 29, 2026.
What it does
Production KV-cache compression for Mixture-of-Experts language models. LLM inference costs explode because: • KV-cache grows with sequence length (16k tokens = 256MB per token) • MoE models waste GPU storing inactive experts • Memory becomes the bottleneck, not compute 📊 REAL BENCHMARKS (Mixtral 8x7B) • KV Memory: 256MB → 30MB (8.53x smaller) • Quality: 100% preserved (zero degradation) • Speed: 8.48x faster in production • Expert Cache Hit: 96.75% • GPU Memory Saved: 6.42 GB per layer
TurboQuant: KV-cache compression for faster and cheaper LLM inference. - RemizovDenis/turboquant
TurboQuant-MoE is a KV-cache compression and dynamic MoE expert management engine for LLMfrom github.com
Does a similar job
all alternatives →
KVSplit – Run 2-3x longer contexts on Apple Silicon2025 · github.com · ▲272I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
Speeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
Cuts Long Horizon Inference Costs by 50% via external KV Cache OffloadJul 2026 · github.com · ▲22Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory. A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed. Once the KV cache is offloaded,…
More dev tools this month
the category →
Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.
Dev tools · 2d ago · github.com
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 30d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 28d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 25d ago · openbot.run
Talanoa▲346Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 22d ago · talanoa.email
Source: Product Hunt launch ↗
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
Littlebird▲709Littlebird remembers everything across your meetings, messages, docs, and apps, then takes action so you can recall any detail and never lose context.
AI · Mar 2026 · littlebird.ai
