Life & fun · February 21, 2026
Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU launched on February 21, 2026 with 395 votes, #42 of 16,627 launches that month and more than 99% of that year's launches.
In plain words
This project enables running the Llama 3.1 70B language model on a single RTX 3090 GPU by directly accessing NVMe storage, bypassing CPU and RAM bottlenecks. Designed for experimenters and hobbyists interested in alternative GPU architectures, it streams model data directly from NVMe to GPU memory. The implementation works on consumer graphics cards, though performance should improve on professional hardware. The code is available on GitHub with supporting library documentation.
written from the facts on this page · September 2026
From the sources
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090. - xaskasdf/ntransformer
High-efficiency C++/CUDA LLM inference engine. Runs Llama 70B on a single RTX 3090 (24GB VRAM) by streaming model layers through GPU memory via PCIe, with optional NVMe direct I/O that bypasses the CPU entirely. 3-tier adaptive caching auto-sizes from hardware: VRAM-resident layers (zero I/O) + pinned RAM (H2D only) + NVMe/mmap fallback. Achieves 83x speedup over mmap baseline for 70B on consumer hardware (RTX 3090 + 48 GB RAM). Bottleneck is PCIe H2D bandwidth at Gen3 x8 (~6.5 GB/s). Q4_K_M fits 10 more layers in VRAM (36 vs 26), reducing tier B transfers. Layer skip (cosine similarity calibration) eliminates 20/80 layers per token with minimal quality loss. # Build mkdir build && cd build…from github.com
Does a similar job
all alternatives →
Tune LLaMa3.1 on Google Cloud TPUs2024 · github.com · ▲189Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla…
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
Finetune LLaMA-7B on commodity GPUs using your own text2023 · github.com · ▲449I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
I've built a locally running Perplexity clone2024 · github.com · ▲669The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
- WMWe made glhf.chat – run almost any open-source LLM, including 405B2024 · glhf.chat · ▲161
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
Run Llama3.1 405B on a 8GB VRAM challenge [video]2024 · youtube.com · ▲19How to run Llama3.1 405B on a 8GB VRAM
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 3d ago · permit.petercai.com
- IV
Life & fun · 15d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 16d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 10d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 14d ago · carrierexplode.com
Life & fun · 24d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, February 2026
the whole month →
Lovon AI Therapy▲801Feel closer and have more fun with your partner every day!
AI · Feb 2026 · lovon.app
Hello my name is Ben Ward for the past 3 years I have been remastering the financial game Wall Street Raider created by Michael Jenkins originally on DOS in 1986. It has been a rough journey but I finally see the light at the end of the tunnel. I just recently redid the website and thought maybe the full story of how this project came to be would interest you all. Thank you for reading.
Work · Feb 2026 · wallstreetraider.com