How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
I found that duplicating a specific block of 7 middle layers in Qwen2-72B, without modifying any weights, improved performance across all Open LLM Leaderboard benchmarks and took #1. As of 2026, the top 4 models on that leaderboard are still descendants. The weird finding: single-layer duplication does nothing. Too few layers, nothing. Too many, it gets worse. Only circuit-sized blocks of ~7 layers work. This suggests pretraining carves out discrete functional circuits in the layer stack that only work when preserved whole. The whole thing was developed on 2x RTX 4090s in my basement. I'm…
How I topped the HuggingFace open LLM leaderboard on two gaming GPUs launched on March 10, 2026 with 495 votes, #20 of 23,380 launches that month and more than 99% of that year's launches.
In plain words
A researcher discovered that duplicating a specific block of seven middle layers in Qwen2-72B without modifying weights improved performance across Open LLM Leaderboard benchmarks. The finding suggests that pretraining creates discrete functional circuits in neural networks that only work when preserved as complete units—single layers or incomplete blocks provide no benefit, while oversized duplications degrade performance. The technique was developed on two RTX 4090 GPUs and generated multiple derivative models that ranked highly on the leaderboard.
written from the facts on this page · September 2026
From the sources
ML, Biotech, Hardware, and Coordination Problems. Sometimes I write about hard problems and how to solve them.
In mid-2024, the HuggingFace Open LLM Leaderboard was the Colosseum for Open-Weight AI. Thousands of models were battling it out, submitted by both well-funded labs with teams of PhDs and fine-tuning wizards creating fantastically named models ( e.g. Nous-Hermes, Dolphin and NeuralBeagle14-7B… ), fighting for the top spot across six benchmarks: IFEval, BBH, MATH Lvl 5, GPQA, MuSR, and MMLU-PRO. I didn’t train a new model. I didn’t merge weights. I didn’t run a single step of gradient descent. What I did was much weirder: I took an existing 72-billion parameter model, duplicated a particular block of seven of its middle layers, and stitched the result back together. No weight was modified in…from dnhkng.github.io
In the maker’s words, at launch
I found that duplicating a specific block of 7 middle layers in Qwen2-72B, without modifying any weights, improved performance across all Open LLM Leaderboard benchmarks and took #1. As of 2026, the top 4 models on that leaderboard are still descendants. The weird finding: single-layer duplication does nothing. Too…
Does a similar job
all alternatives →
Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No trainingMar 2026 · github.com · ▲265I replicated David Ng's RYS method (https://dnhkng.github.io/posts/rys/) on consumer AMD GPUs (RX 7900 XT + RX 6950 XT) and found something I didn't expect. Transformers appear to have discrete "reasoning circuits" — contiguous blocks of 3-4 layers that act as indivisible cognitive units. Duplicate the right block and the model runs its reasoning pipeline twice. No weights change. No training. The model just thinks longer. The results on standard benchmarks (lm-evaluation-harness, n=50): Devstral-24B, layers 12-14 duplicated once: - BBH Logical Deduction: 0.22 → 0.76…
I've built a locally running Perplexity clone2024 · github.com · ▲669The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
Fine-tune an 8B model on a 4 GB laptop GPUAug 2026 · github.com · ▲139Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU. - MakazhanAlpamys/Soup
Minimal LLM Post-Training Experiments on an 8GB GPU (SFT, DPO, GRPO)Aug 2026 · github.com · ▲21Minimal, readable LLM post-training experiments on one 8GB GPU. Measures forgetting, seed variance, and RL emergence. - pochenai/nano-llm-posttraining
Multi-GPU Reinforcement Learning in Tensorflow for OpenAI Gym2016 · github.com · ▲58Multi-GPU reinforcement learning using Deep Q-Network in TensorFlow for OpenAI Gym - vishar0/dist-dqn
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
Littlebird▲709Littlebird remembers everything across your meetings, messages, docs, and apps, then takes action so you can recall any detail and never lose context.
AI · Mar 2026 · littlebird.ai
