Speeding up LLM inference 2x times (possibly)
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
Speeding up LLM inference 2x times (possibly) launched on April 17, 2024 with 419 votes, #84 of 4,347 launches that month and more than 99% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
This project presents an algorithm for accelerating large language model inference on Apple Silicon chips. It reduces the number of weight multiplications needed during inference to 20-25% of the total while maintaining output quality, allowing users to adjust speed and accuracy in real time. The implementation achieves approximately 2x speedup in matrix multiplication operations and runs comparably to Llama.cpp, with potential for further optimization through improved development.
written from the facts on this page · September 2026
From the sources
A quick preview of the Effort algorithm - more details on https://kolinko.github.io/effort/
In the maker’s words, at launch
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them…
Does a similar job
all alternatives →
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
KVSplit – Run 2-3x longer contexts on Apple Silicon2025 · github.com · ▲272I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
We made our own inference engine for Apple Silicon2025 · github.com · ▲186We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
Faster LLM evaluation with Bayesian optimization2024 · github.com · ▲131Recently I've been working on making LLM evaluations fast by using bayesian optimization to select a sensible subset. Bayesian optimization is used because it’s good for exploration / exploitation of expensive black box (paraphrase, LLM). I would love to hear your thoughts and suggestions on this!
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 28d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, April 2024
the whole month →
Supabase▲2,325The Postgres developer platform is now generally available
Dev tools · 2024 · supabase.com
Cal.com Platform▲1,601Build your pixel-perfect booking experience with Atoms
Dev tools · 2024 · cal.com
deco.cx 2.0▲1,078Build web apps 10x faster with Deno, JSX, TS & Tailwind
Dev tools · 2024 · decocms.com
