KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill
Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill launched on February 10, 2025 with 14 votes, #1,075 of 5,999 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
KTransformers is a Python framework for optimizing large language model inference on consumer hardware. It enables users to run massive models like DeepSeek-R1 (671B parameters) and handle extended contexts (up to 1M tokens) locally at competitive speeds through optimized kernels. The framework integrates seamlessly with HuggingFace Transformers and provides multiple interfaces including RESTful APIs compatible with OpenAI and Ollama, plus a web chat interface. Developers can add performance optimizations with minimal code changes while maintaining compatibility with existing tools and plugins.
written from the facts on this page · September 2026
From the sources
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations - kvcache-ai/ktransformers
In the maker’s words, at launch
Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to…
Does a similar job
all alternatives →
KTransformers–236B Model and 1M Context LLM Inference on Local Machines2024 · github.com · ▲20Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try itJul 2026 · ctgt.ai · ▲170We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's…
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
- ATA tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)Aug 2026 · mikeayles.com · ▲79
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, February 2025
the whole month →
Screen Studio 3.0▲1,832Beautiful screen recordings with instant shareable links
Growth · 2025 · screen.studio
- IG
I was at FB/Meta from late 2013 to early 2023, mostly working in the compiler/runtime spaces. I got hit in the spring 2023 layoff wave. I immediately started making games in my newfound free time (a lifelong interest, and I even worked in AA(A?) back ca. ~2000), and in October 2023 I stumbled upon the idea of a roguelike pachinko/plinko game inspired by Luck Be A Landlord. Things snowballed quickly, I started talking to publishers, then worked like crazy through all of 2024, almost the hardest I've ever worked in my career, and launched the game in December 2024. It's sold…
Work · 2025
i wanted to change the habit of reaching for my phone in the morning and doomscrolling away an hour so i built an app to help me. now i have to literally touch grass before accessing my most distracting apps the app is built in swiftui, uses the screen time apis provided by apple and google vision to recognise grass or not i'd love to get your thoughts on the concept.
Life & fun · 2025 · touchgrass.now