KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT
If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a single notebook—covering tokenization, embeddings, causal self-attention, training, and autoregressive text generation without relying on high-level abstractions. The focus was on mechanistic clarity rather than scale or performance, and the notebook is structured to read more like a technical walkthrough than an experiment log. Feedback from people who’ve built or studied transformers would be very welcome.
KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT launched on May 22, 2026 with 20 votes, #911 of 20,538 launches that month and more than 90% of that year's launches.
In plain words
KVBoost is a tool that speeds up the initial response time of language models by reusing key-value cache data at the chunk level. It integrates with HuggingFace and can deliver 5 to 48 times faster time-to-first-token performance. The tool is designed for developers and researchers working with large language models who need quicker inference responses.
written from the facts on this page · September 2026
Does a similar job
all alternatives →
HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try itJul 2026 · ctgt.ai · ▲170We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's…
Timber – Ollama for classical ML models, 336x faster than PythonMar 2026 · github.com · ▲207Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference. - kossisoroyce/timber
KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill2025 · github.com · ▲14Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗
Launched alongside, May 2026
the whole month →
Parallel agents, diff reviewer, and multi-model comparisons
Dev tools · May 2026 · kilo.ai
Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices. We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we conducted investigations that led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly (match query to tool name, extract argument values, emit JSON), not reasoning. Cross-attention…
Life & fun · May 2026 · github.com
🌱 Private, quiet space for thinking. Simple app for .md files. - zakirullin/files.md
Dev tools · May 2026 · github.com