RunNburn – Run a 295B Moe from a 98GB GGUF on a 64GB RAM Desktop
runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU caches are sized from detected free/total VRAM — never from device-name presets. There is no conversion step, no sidecar cache files, no silent requantization. The GGUF on disk is the single source of truth. The result that made me want to post this: Tencent's Hy3 (295B total / 21B active sparse MoE, a single 97.8 GiB Q2_K GGUF) runs on my desktop…
In plain words
RunNburn is a Rust-based inference engine for running quantized GGUF models larger than available system memory. It keeps model weights file-backed using memory mapping while enforcing an explicit RAM budget, and sizes GPU caches based on detected available VRAM rather than device presets. Designed for developers and researchers, it requires no conversion steps or cache files, treating the GGUF file on disk as the single source of truth. The engine enables running very large models like sparse mixture-of-experts models on desktop hardware by loading only needed components on demand.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU caches are sized from detected free/total VRAM — never from device-name presets. There is no conversion step, no sidecar cache files, no silent requantization. The GGUF on disk is the single source of truth. The result that made me want to post this: Tencent's Hy3 (295B total / 21B active sparse MoE, a single 97.8 GiB Q2_K GGUF) runs on my desktop with 64 GB of RAM and one consumer NVIDIA GPU. The file is larger than RAM and VRAM combined; the selected experts for each token are pulled on demand (the newest path batches O_DIRECT reads through io_uring), while the pretrained routing is left untouched. On the same machine, same prompt, same decode length, a warm-run median gave ~5.5 tok/s decode vs ~2.0 tok/s for llama.cpp. To be upfront about scope: for models that fit comfortably in VRAM, llama.cpp is still faster than runNburn today — its CUDA kernels have years of tuning and we measure against it honestly (interleaved A/B runs, medians, and any "speedup" that changes output quality is rejected). runNburn's lane is the model that doesn't fit. What's in the box: - CLI, interactive chat, and an OpenAI-compatible server (chat/completions + responses + conversations, SSE streaming, stateful continuation with KV/SSM snapshot reuse). It's built as a single-owner personal server — one active generation is the optimization unit; continuous batching and multi-tenant throughput are explicit non-goals. - Architecture-aware paths: Llama family, Phi, Gemma, Qwen dense/hybrid/MoE (including GatedDeltaNet layers), Nemotron-H MoE, Hy3, GLM — plus in-model multi-token prediction (self-speculative decoding) with device-side verification where the GGUF ships a drafter. - Backends: CPU is the default (x86 AVX2, ARM NEON), CUDA and Metal are active, Vulkan/OpenCL are experimental. Android works through a small C ABI (rnb.h). - Native quantized kernels for Q2_K–Q6_K, Q4_0, Q8_0 — including the low-bit K-quants that big-MoE builds actually ship in. It's pre-1.0 and rough in places; recognition of an architecture doesn't mean every community variant works. But if you've got a model file bigger than your machine and you'd rather it run slowly than not at all, that's exactly the case it was built for. Happy to answer questions about the offloading design, the expert-streaming path, or the measurement protocol.
Does the same job
all alternatives →- OSOpen-source engine running Gemma 4 26B in 2 GB RAM on any M-series MacJul 2026 · github.com · ▲919
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
- WMWe made glhf.chat – run almost any open-source LLM, including 405B2024 · glhf.chat · ▲161
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
General ComputeMay 2026 · generalcompute.com · ▲315AI models that run on an inference cloud optimized for speed
- LALocalGPT – A local-first AI assistant in Rust with persistent memoryFeb 2026 · github.com · ▲331
I built LocalGPT over 4 nights as a Rust reimagining of the OpenClaw assistant pattern (markdown-based persistent memory, autonomous heartbeat tasks, skills system). It compiles to a single ~27MB binary — no Node.js, Docker, or Python required. Key features: - Persistent memory via markdown files (MEMORY, HEARTBEAT, SOUL markdown files) — compatible with OpenClaw's format - Full-text search (SQLite FTS5) + semantic search (local embeddings, no API key needed) - Autonomous heartbeat runner that checks tasks on a configurable interval - CLI + web interface + desktop GUI - Multi-provider:…
- RQRun Qwen3-Next-80B on 8GB GPU at 1tok/2s throughputSep 2025 · github.com · ▲123
Shoehorn – Quantize any model down to run on your machine19d ago · notactuallytreyanastasio.github.io · ▲97Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, July 2026
the whole month →- IR
I might be the only SRE on Earth with his own bowling center. It's a more in-depth gig than you'd think. My family and I bought an abandoned 8-lane bowling center in the rural mid-west. In our small town there weren't many recreation options for families. You've heard of a food desert? This is an R&R desert. It had been abandoned for a good reason. The roof leaks, the electrical system was constantly surging, and my 70-year-old bowling equipment (still) doesn't work perfectly. The system that keeps your score is particularly interesting to me. It's the thing you watch during your game, but…
Life & fun · Jul 2026
- EElevators▲1,680
Life & fun · Jul 2026 · john.fun
- 1W18 Words▲1,160
Life & fun · Jul 2026 · 18words.com
- BA
Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses like Claude Code, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up creating Bento, a single HTML file with everything you need in a slide tool including animations and shared editing. There's no install or cloud login, everything works offline. The default deck is around 560 KB and it doesn't need to fetch anything once you got it. Open it in a browser and then you can…
Dev tools · Jul 2026 · bento.page
- GG
A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me. But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility. I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context.…
AI · Jul 2026 · github.com
