nowfound

Life & fun · August 1, 2024

Run Llama3.1 405B on a 8GB VRAM challenge [video]

How to run Llama3.1 405B on a 8GB VRAM

Run Llama3.1 405B on a 8GB VRAM challenge [video] launched on August 1, 2024 with 19 votes, #1,011 of 5,471 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 2 years on.

Visit youtube.com ↗Alternativestop 18% of August 2024

In plain words

This video demonstrates techniques for running Llama 3.1 405B, a large language model, on hardware with only 8GB of VRAM. The resource is aimed at users who want to work with this advanced model but lack high-end GPU memory. The guide addresses a practical challenge in machine learning by showing optimization methods that allow computationally demanding models to run on consumer-grade hardware constraints.

written from the facts on this page · September 2026

Does a similar job

all alternatives →
  • Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPUFeb 2026 · github.com · ▲395

    Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho

  • Llama2024 · developer.meta.com · ▲312

    3.1-405B: an open source model to rival GPT-4o / Claude-3.5

  • Llama 3.3 70B2024 · developer.meta.com · ▲347

    Llama 405B-level performance, at a fraction of the cost

  • Soup CLIAug 2026 · trysoup.dev · ▲107

    Fine-tune an 8B LLM on a 4 GB laptop GPU

  • 80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385

    Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…

  • Run Qwen3-Next-80B on 8GB GPU at 1tok/2s throughput2025 · github.com · ▲123

    Contribute to Mega4alik/ollm development by creating an account on GitHub.

More life & fun this month

the category →

Source: Show HN post ↗

Launched alongside, August 2024

the whole month →
  • Wordware▲9,848

    Your tool for building AI agents with natural language

    AI · 2024 · wordware.ai

  • If YouTube had actual channels▲2,741

    If Youtube had actual channels.

    Life & fun · 2024 · ytch.xyz

  • Me.bot▲1,627

    The inspiring companion for your life

    AI · 2024 · me.bot

  • Upmetrics AI▲1,197

    Business plan & financial forecast using AI

    Work · 2024 · upmetrics.co

  • Jupitrr AI▲1,195

    Jupitrr AI turns raw talking videos into scroll-stopping content for coaches and consultants. Research, script, record, edit, and publish in one app.

    AI · 2024 · jupitrr.com

  • InstantDB – A Modern Firebase▲1,145

    Hey there HN! We’re Joe and Stopa, and today we’re open sourcing InstantDB, a client-side database that makes it easy to build real-time and collaborative apps like Notion and Figma. Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can…

    Dev tools · 2024 · github.com