Life & fun · alternatives · 2026
24 alternatives to Run Llama3.1 405B on a 8GB VRAM challenge [video]
How to run Llama3.1 405B on a 8GB VRAM
This video demonstrates techniques for running Llama 3.1 405B, a large language model, on hardware with only 8GB of VRAM. The resource is aimed at users who want to work with this advanced model but lack high-end GPU… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Run Llama3.1 405B on a 8GB VRAM challenge [video] launched in 2024; newer entries below may have overtaken it.
- 1
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
Feb 2026 · github.com · its alternatives →
- 2
Llama▲3123.1-405B: an open source model to rival GPT-4o / Claude-3.5
2024 · developer.meta.com · its alternatives →
- 3
Llama 3.3 70B▲347Llama 405B-level performance, at a fraction of the cost
2024 · developer.meta.com · its alternatives →
- 4
- 5
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com · its alternatives →
- 6
Contribute to Mega4alik/ollm development by creating an account on GitHub.
2025 · github.com · its alternatives →
- 7
2024 · app.wiz.chat · its alternatives →
- 8
Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla…
2024 · github.com · its alternatives →
- 9
High-speed download of LLaMA, Facebook's 65B parameter GPT model - shawwn/llama-dl
2023 · github.com · its alternatives →
- 10
Ollama▲235The easiest way to run large language models locally
2023 · github.com · its alternatives →
- 11
I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
2023 · github.com · its alternatives →
- 12WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat · its alternatives →
- 13
I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift. It ships with auto-mode, which makes a good tradeoff between memory usage and speed. I'll be implementing and porting the MTP module for speculative decoding next Local models really are the future of computing!
Sep 2026 · github.com · its alternatives →
- 14
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU. - MakazhanAlpamys/Soup
Aug 2026 · github.com · its alternatives →
- 15
Run leading vision models locally with the new engine
2025 · ollama.com · its alternatives →
- 16
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com · its alternatives →
- 17
Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…
2023 · github.com · its alternatives →
- 18
LLaMA▲118A foundational, 65-billion-parameter large language model
2023 · research.facebook.com · its alternatives →
- 19
Hey, recently I took inspiration from llama.cpp, ollama, and many other similar tools that enable inference of LLMs locally, and I just finished building a Llama inference engine for the 8B model in CUDA C. I recently wanted to explore my newly founded interest in CUDA programming and my passion for machine learning. This project only makes use of the native CUDA runtime api and cuda_fp16. The inference takes place in fp16, so it requires around 17-18GB of VRAM (~16GB for model params and some more for intermediary caches). It doesn’t use cuBLAS or any similar libraries since I wanted to be…
2025 · github.com · its alternatives →
- 20
Explore large language models in 512MB of RAM. Contribute to jncraton/languagemodels development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 21
Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization - RahulSChand/gpu_poor
2023 · github.com · its alternatives →
- 22
Llama 2▲263The next generation of Meta's open source LLM
2023 · developer.meta.com · its alternatives →
- 23IM
2023 · vram.asmirnov.xyz · its alternatives →
- 24
Dream 7B▲191Powerful Open Diffusion LLM, Beyond Autoregressive
2025 · hkunlp.github.io · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →