ZSE – Open-source LLM inference engine with 3.9s cold starts
I've been building ZSE (Z Server Engine) for the past few weeks — an open-source LLM inference engine focused on two things nobody has fully solved together: memory efficiency and fast cold starts. The problem I was trying to solve: Running a 32B model normally requires ~64 GB VRAM. Most developers don't have that. And even when quantization helps with memory, cold starts with bitsandbytes NF4 take 2+ minutes on first load and 45–120 seconds on warm restarts — which kills serverless and autoscaling use cases. What ZSE does differently: Fits 32B in 19.3 GB VRAM (70% reduction vs FP16) — runs…
ZSE – Open-source LLM inference engine with 3.9s cold starts launched on February 26, 2026 with 58 votes, #564 of 16,627 launches that month and more than 95% of that year's launches.
In plain words
ZSE is an open-source LLM inference engine designed to run large language models with significantly reduced memory requirements and fast startup times. It enables 32-billion-parameter models to run on a single 40GB GPU and 7-billion-parameter models on consumer GPUs through efficient quantization and a custom pre-quantized format. The engine achieves 3.9-second cold starts for 7B models and 21.4 seconds for 32B models, making it suitable for serverless and autoscaling deployments where traditional inference engines are too slow or memory-intensive.
written from the facts on this page · September 2026
From the sources
The inference engine the open-source world built for itself. - Zyora-Dev/zse
The fastest cold start. The smallest memory footprint. On every GPU. ZSE is a production LLM inference engine that owns the full stack — no PyTorch, no Triton, no bitsandbytes, no transformers . Just pure Python, ctypes, and a kernel compiler that emits CUDA, ROCm (HIP), and Metal directly. The result: models load in seconds, not minutes , and serve at a fraction of the memory other engines need. pip install zse-engine # one package, zero transitive ML deps zse serve qwen-7b.zse # 7-second cold start. 5.8 GB on a T4. Headline Numbers Verified on Modal (T4, L4, A10G, A100), DigitalOcean (MI300X), and Apple M1. ZSE INT4 vs vLLM AWQ INT4, Qwen2.5-7B / 14B / 32B. ZSE runs 32B INT4 in 22 GB of…from github.com
In the maker’s words, at launch
I've been building ZSE (Z Server Engine) for the past few weeks — an open-source LLM inference engine focused on two things nobody has fully solved together: memory efficiency and fast cold starts. The problem I was trying to solve: Running a 32B model normally requires ~64 GB VRAM. Most developers don't have that.…
Does a similar job
all alternatives →
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series MacJul 2026 · github.com · ▲919Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Z80-μLM, a 'Conversational AI' That Fits in 40KBDec 2025 · github.com · ▲514How small can a language model be while still doing something useful? I wanted to find out, and had some spare time over the holidays. Z80-μLM is a character-level language model with 2-bit quantized weights ({-2,-1,0,+1}) that runs on a Z80 with 64KB RAM. The entire thing: inference, weights, chat UI, it all fits in a 40KB .COM file that you can run in a CP/M emulator and hopefully even real hardware! It won't write your emails, but it can be trained to play a stripped down version of 20 Questions, and is sometimes able to maintain the illusion of having simple but terse conversations…
Speeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
We Beat MLPerf: Modern Storage for KV Offload and LLM TrainingSep 2026 · theopenlake.com · ▲35High performance storage engine for efficient LLM inference and GPU Training.
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, February 2026
the whole month →
Lovon AI Therapy▲801Feel closer and have more fun with your partner every day!
AI · Feb 2026 · lovon.app
Hello my name is Ben Ward for the past 3 years I have been remastering the financial game Wall Street Raider created by Michael Jenkins originally on DOS in 1986. It has been a rough journey but I finally see the light at the end of the tunnel. I just recently redid the website and thought maybe the full story of how this project came to be would interest you all. Thank you for reading.
Work · Feb 2026 · wallstreetraider.com