Dev tools · alternatives · 2026

24 alternatives to RunNburn – Run a 295B Moe from a 98GB GGUF on a 64GB RAM Desktop

runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU caches are sized from detected free/total VRAM — never from device-name presets. There is no conversion step, no sidecar cache files, no silent requantization. The GGUF on disk is the single source of truth. The result that made me want to post this: Tencent's Hy3 (295B total / 21B active sparse MoE, a single 97.8 GiB Q2_K GGUF) runs on my desktop…

RunNburn is a Rust-based inference engine for running quantized GGUF models larger than available system memory. It keeps model weights file-backed using memory mapping while enforcing an explicit RAM budget, and sizes… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Prices are shown for the 1 we have checked on their own sites, 1 of them free or with a free tier.

1
Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac▲919
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com · its alternatives →
2WM
We made glhf.chat – run almost any open-source LLM, including 405B▲161
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat · its alternatives →
3
Getting GLM 5.2 running on my slow computer▲937
A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me. But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility. I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context.…
Jul 2026 · github.com · its alternatives →
4
General Compute▲315
AI models that run on an inference cloud optimized for speed
May 2026 · generalcompute.com · its alternatives →
5
I've built a locally running Perplexity clonefree▲669
The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
2024 · github.com · its alternatives →
6
LocalGPT – A local-first AI assistant in Rust with persistent memory▲331
I built LocalGPT over 4 nights as a Rust reimagining of the OpenClaw assistant pattern (markdown-based persistent memory, autonomous heartbeat tasks, skills system). It compiles to a single ~27MB binary — no Node.js, Docker, or Python required. Key features: - Persistent memory via markdown files (MEMORY, HEARTBEAT, SOUL markdown files) — compatible with OpenClaw's format - Full-text search (SQLite FTS5) + semantic search (local embeddings, no API key needed) - Autonomous heartbeat runner that checks tasks on a configurable interval - CLI + web interface + desktop GUI - Multi-provider:…
Feb 2026 · github.com · its alternatives →
7
Run Qwen3-Next-80B on 8GB GPU at 1tok/2s throughput▲123
Contribute to Mega4alik/ollm development by creating an account on GitHub.
2025 · github.com · its alternatives →
8
We made our own inference engine for Apple Silicon▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com · its alternatives →
9
Shoehorn – Quantize any model down to run on your machine▲99
Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn
Aug 2026 · notactuallytreyanastasio.github.io · its alternatives →
10
RunInfra▲156
Describe the AI model you need and get an optimized AI
Jul 2026 · runinfra.ai · its alternatives →
11
Luminal – Open-source, search-based GPU compiler▲153
Hi HN, I’m Joe. My friends Matthew, Jake and I are building Luminal (https://luminalai.com/), a GPU compiler for automatically generating fast GPU kernels for AI models. It uses search-based compilation to achieve high performance. We take high level model code, like you'd have in PyTorch, and generate very fast GPU code. We do that without using LLMs or AI - rather, we pose it as a search problem. Our compiler builds a search space, generates millions of possible kernels, and then searches through it to minimize runtime. You can try out a demo in `demos/matmul` on mac to…
2025 · github.com · its alternatives →
12TC
TensorDock Core GPU Cloud – GPU servers from $0.29/hr▲147
Hello HN! I’m Jonathan from TensorDock. After 7 months in beta, we’re finally launching Core Cloud, our platform to deploy GPU virtual machines in as little as 45 seconds! https://www.tensordock.com/product-core Why? Training machine learning workloads at large clouds can be extremely expensive. This left us wondering, “how did cloud ever become more expensive than on-prem?” I’ve seen too many ML startups buy their own hardware. Cheaper dedicated servers with NVIDIA GPUs are not too hard to find, but they lack the functionality and scalability of the big clouds. We thought to…
2022 · tensordock.com · its alternatives →
13
Running Gemma-4 26B at 124 tokens/SEC on a CPU, no GPU▲10
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev · its alternatives →
14
ZSE – Open-source LLM inference engine with 3.9s cold starts▲58
I've been building ZSE (Z Server Engine) for the past few weeks — an open-source LLM inference engine focused on two things nobody has fully solved together: memory efficiency and fast cold starts. The problem I was trying to solve: Running a 32B model normally requires ~64 GB VRAM. Most developers don't have that. And even when quantization helps with memory, cold starts with bitsandbytes NF4 take 2+ minutes on first load and 45–120 seconds on warm restarts — which kills serverless and autoscaling use cases. What ZSE does differently: Fits 32B in 19.3 GB VRAM (70% reduction vs FP16) — runs…
Feb 2026 · github.com · its alternatives →
15
Cuq – Formal Verification of Rust GPU Kernels▲94
A pipeline that translates Rust GPU code into formal Coq models, as a foundation for memory model proofs - neelsomani/vericuda
2025 · github.com · its alternatives →
16
Soup CLI▲107
Fine-tune an 8B LLM on a 4 GB laptop GPU
Aug 2026 · trysoup.dev · its alternatives →
17
We Beat MLPerf: Modern Storage for KV Offload and LLM Training▲35
High performance storage engine for efficient LLM inference and GPU Training.
Sep 2026 · theopenlake.com · its alternatives →
18KC
KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT▲20
If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a single notebook—covering tokenization, embeddings, causal self-attention, training, and autoregressive text generation without relying on high-level abstractions. The focus was on mechanistic clarity rather than scale or performance, and the notebook is structured to read more like a technical walkthrough than an experiment log. Feedback from people who’ve built or studied transformers would be very welcome.
May 2026 · pythongiant.github.io · its alternatives →
19
Shimmy – 5MB privacy-first, local alternative to Ollama (680MB)▲14
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary. - Michael-A-Kuykendall/shimmy
2025 · github.com · its alternatives →
20
KiviDB, a faster Redis-compatible memory database and vector cache▲69
A fast, multi-threaded Redis® and Valkey alternative written in Rust: drop-in compatible, with full-text and vector search built in - kividbio/kividb
5d ago · github.com · its alternatives →
21
I made an open-source Rust program for memory-efficient genomics▲17
My cofounder and I run a startup in oncology, where we handle cancer genomics data. It occurred to me that, thanks to a recent complexity theory result, there's a clever way to run bioinformatics algorithms using far less RAM. I built this Rust engine for running whole-genome workloads in under 100MB of RAM. Runtime is a little longer as a result - O(TlogT) instead of O(T). But it should enable whole-genome analytics on consumer-grade hardware.
Nov 2025 · github.com · its alternatives →
22
Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE▲24
Metal-first MoE inference for Apple Silicon with bounded SSD expert streaming and local OpenAI Chat, Responses, and Anthropic Messages endpoints. - hebrus-labs/hebrus
Jul 2026 · github.com · its alternatives →
23
I Built an Open Source API with Insanely Fast Whisper and Fly GPUs▲13
Hi HN! Since the launch of JigsawStack.com, we've been trying to dive deeper into fully managed AI APIs built and fine tuned for specific use cases. Audio/video transcription was one of the more basic things and we wanted the best open source model at this point it is OpenAI's whisper large v3 model based on the number of languages it supports and its accuracy. The thing is, the model is huge and requires tons of GPU power for it to run efficiently at scale. Even OpenAI doesn't provide an API for their best transcription model while only providing whisper v2 at a pretty high price. I…
2024 · github.com · its alternatives →
24
Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload▲22
Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory. A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed. Once the KV cache is offloaded,…
Jul 2026 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →