Soup CLI
Fine-tune an 8B LLM on a 4 GB laptop GPU
What it does
LoRA keeps the base model frozen: read, never written. So Soup keeps it in system RAM and streams it into the GPU one decoder layer at a time. Peak VRAM becomes one layer instead of the whole model. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B trains at 119.6 tok/s in 3.32 GB peak. One YAML, one command. SFT, DPO, GRPO, KTO, plus eval, gating and export. Apache-2.0. Every number is published, including the ones I measured and threw away.
Fine-tune Llama-3.1-8B on a 4 GB laptop GPU, and align on the same card. Soup is at v0.73.3, and the v0.73 line came out of three days on somebody else's hardware: every number this project had published came from one 4 GB RTX 3050 Laptop, and a borrowed 8x H100 box was the first machine able to hold a resident reference for a model worth streaming. It confirmed the method (the streamed forward pa
The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config (task, quantization, LR and epochs come from rules, not a search), derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting. And when the model is bigger than the card, one opt-in config key streams the frozen base from RAM or NVMe one layer at a time and quantizes it to 4-bit (BETA), so Llama-3.1-8B fine-tunes on a 4 GB GPU , and DPO, ORPO, SimPO and KTO stream too. 23 methods · 144 recipes · 17 quant formats · MLX + Apple adapter . The full AI workbench built on top of Soup: playground, data, evals, fine-tuning, deploy…from trysoup.dev
Does the same job
all alternatives →
- SYSlash your LLM Inference Costs with Overnight Processing2024 · ▲5
Hey HN, If you tried running open-source models like Llama 3.1 70B or 405B, you might have noticed that it gets very expensive. The reason looks obvious enough that you might have stopped even before trying it! - GPUs are very expensive to buy or rent - Running the most performing LLMs need 4, 8 or even 16 top of the line Nvidia GPUs - And that won’t get you anywhere near the level of VRAM needed to batch enough to get a decent throughput and efficiency Some have even questioned if open-source LLM providers are not doing some shenanigans to provide the prices they offer. VC funded…
- IMI made a VRAM Calculator in Hugging Face2025 · chromewebstore.google.com · ▲9
It's a chrome extension that automatically loads the specs from the Hugging Face model card into the calculation. > To test it, install the extension (no registration/key needed) and navigate to a HF model page. Then click the "VRAM" icon on the top right to open the sidepanel. You can specify quantization, batch size, sequence length, etc. Works for inference & fine-tuning. If it does not fit on the specified GPUs, it gives you an advise on how to still run it (e.g. lowering precision). It is inspired at my work, where we were constantly exporting metrics from HF to estimate required…
Shoehorn, a library to quantize an LLM to fit your Mac's VRAM22d ago · github.com · ▲6I made this after seeing someone posit the idea online yesterday over lunch then spent some time refining it. So far it's pretty impressive IMO! Right now I am running Qwen3-30B-A3B on my 24gb unified memory m4 MacBook Pro at 50 tok/sec and this should definitely not be working for such a large model on my middling hardware. Things are detailed in the README to get up and running and DESIGN.md has details on all the choices and such made along the way.
- TCTurboPilot: Copilot clone runs code completion LLM on your CPU2023 · github.com · ▲37
Hi HN, I spent my easter weekend stuck in the house with COVID and I decided to play with llama.cpp [1] and fauxpilot [2] to see if I could get LLM code assist working on pure CPU. As a proof of concept I'd say I've proven that it's possible. However there's still a lot to do. The auto complete is quite slow at the moment. PRs welcome. [1] https://github.com/ggerganov/llama.cpp [2] https://github.com/fauxpilot/fauxpilot
- IBI built a tool to check if your computer can run LLMs locally2025 · caniusellm.com · ▲8
Built a simple web app that tells you which open-source LLMs will work on your hardware. It auto-detects your specs, shows compatible models from Hugging Face, gives realistic performance estimates (tokens/sec), and recommends quantization settings. You can also manually input specs to see "what if I upgraded my RAM?" Made this after wasting time downloading giant models only to find they crawled on my hardware. Hope it saves you some frustration!
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 16d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, August 2026
the whole month →- TL
Life & fun · 10d ago · louisabraham.github.io
- SA
Hello HN! I found that picking out plausible but diverse skin tones for my digital art and game development projects was kind of difficult, and I got curious about if there was a way to define a color space that made it easy. I've built a color picker and procedural generation algorithm based on the space as well as a bunch of other fun js features and demos throughout the page that use the equations. If you find it interesting, I have lots of explanations of how I built it and what properties the space has. The methodology might be a bit shaky, but hopefully the result is as helpful for…
Life & fun · Aug 2026 · toneyalexander.github.io
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 16d ago · simedw.com