KTransformers–236B Model and 1M Context LLM Inference on Local Machines
Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
In plain words
KTransformers is an open-source Python framework that optimizes large language model inference on local machines. It enables users to run massive 236B parameter models or process 1 million token contexts efficiently by integrating optimized kernels into HuggingFace Transformers with minimal code changes. The framework provides a Transformers-compatible interface, OpenAI and Ollama-compliant REST APIs, and a web chat UI, making it suitable for developers seeking high-performance LLM deployment without cloud infrastructure.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a Transformers-compatible interface, RESTful APIs compliant with OpenAI and Ollama, and even a simplified ChatGPT-like web UI. For example, it allows you to integrate with all your familiar frontends, such as the VS Code plugin backed by Tabby. To demonstrate its capability, we present two showcase demos: - GPT-4-level Local VSCode Copilot: It runs the huge 236B DeepSeek-Coder-V2's Q4_K_M variant using just 11GB VRAM and 136GB DRAM on a local machine, which matches the score of GPT4-0613 in BigCodeBench with a promising 126 tokens/s for prompt prefill and 13.6 tokens/s for generation. - 1M Context Local Inference:Achieves 15 tokens/s with nearly 100% accuracy on the "Needle In a Haystack" test via the InternLM2.5-7B-Chat-1M model, utilizing 24GB VRAM and 150GB DRAM, and is several times faster than llama.cpp. Check it out on GitHub: https://github.com/kvcache-ai/ktransformers
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, August 2024
the whole month →
- IY
Life & fun · 2024 · ytch.xyz
- IA
Hey there HN! We’re Joe and Stopa, and today we’re open sourcing InstantDB, a client-side database that makes it easy to build real-time and collaborative apps like Notion and Figma. Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can…
Dev tools · 2024 · github.com