Run Llama3.1 405B on a 8GB VRAM challenge [video]
How to run Llama3.1 405B on a 8GB VRAM
In plain words
This video demonstrates techniques for running Llama 3.1 405B, a large language model, on hardware with only 8GB of VRAM. The resource is aimed at users who want to work with this advanced model but lack high-end GPU memory. The guide addresses a practical challenge in machine learning by showing optimization methods that allow computationally demanding models to run on consumer-grade hardware constraints.
written from the facts on this page · September 2026
Does the same job
all alternatives →- L3Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPUFeb 2026 · github.com · ▲395
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
- 8F80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
- RQRun Qwen3-Next-80B on 8GB GPU at 1tok/2s throughputSep 2025 · github.com · ▲123
More life & fun this month
the category →- TL
Life & fun · 10d ago · louisabraham.github.io
Articos▲385Launch with confidence, not gut instinct Discussion | Link
Life & fun · 12d ago · producthunt.com
Nex▲351Claude Cowork for high-volume GTM workflows Discussion | Link
Life & fun · 3d ago · producthunt.com
Photosynthesis fires two of your iPhone
Life & fun · 29d ago · photosynthesis.camera
Creatium Coach▲320Your multimedia mentor that takes you from mid to great Discussion | Link
Life & fun · 11d ago · producthunt.com
SoloUno▲310Take control of hair pulling, nail biting & skin picking
Life & fun · 28d ago · solouno.io
Launched alongside, August 2024
the whole month →
- IY
Life & fun · 2024 · ytch.xyz
- IA
Hey there HN! We’re Joe and Stopa, and today we’re open sourcing InstantDB, a client-side database that makes it easy to build real-time and collaborative apps like Notion and Figma. Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can…
Dev tools · 2024 · github.com