Sparse Matrix-Vector Multiplication that works at 30–90% sparsity
To get benefits from sparsity, you usually need to have very sparse matrices, impose some structure on the sparsity pattern or have specialized hardware. None of it is the case if you want to rune pruned LLMs on consumer devices. I wanted to see how far can you push it on a GPU and ended up with this. Blog: https://www.grizzlytech.dev/blog/macko-spmv Paper: https://arxiv.org/abs/2511.13061 Code (example with torch): https://github.com/vlejd/macko_spmv
Sparse Matrix-Vector Multiplication that works at 30–90% sparsity launched on November 25, 2025 with 7 votes, #1,843 of 9,796 launches that month and more than 75% of that year's launches.
What it does
MACKO: Sparse matrix vector multiplication for low sparsity - vlejd/macko_spmv
MACKO-SpMV: M utually A ligned C ompressed coordinates K ernel O ptimised Sp arse M atrix V ector multiplication is a new format for representing sparse matrices and a cuda kernel for efficient matrix vector multiplication using this format. It is targeted at sparsities between 20-90%, commonly seen in Neural Network pruning. These sparsities were historically very hard to make use of in practice, because existing formats like CSR are not optimized for this range. We hope this library will help to spark more interest in the field of neural network pruning and find uses outside it as well. Debate whether quantization is better than pruning is beyond the scope of this work, but we hope we…from github.com
Does a similar job
all alternatives →
Speeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
I've built a locally running Perplexity clone2024 · github.com · ▲669The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
Matrix Multiplication with Half the Multiplications2024 · github.com · ▲310Algebraic enhancements for GEMM & AI accelerators. Contribute to trevorpogue/algebraic-nnhw development by creating an account on GitHub.
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
How I topped the HuggingFace open LLM leaderboard on two gaming GPUsMar 2026 · dnhkng.github.io · ▲495I found that duplicating a specific block of 7 middle layers in Qwen2-72B, without modifying any weights, improved performance across all Open LLM Leaderboard benchmarks and took #1. As of 2026, the top 4 models on that leaderboard are still descendants. The weird finding: single-layer duplication does nothing. Too few layers, nothing. Too many, it gets worse. Only circuit-sized blocks of ~7 layers work. This suggests pretraining carves out discrete functional circuits in the layer stack that only work when preserved whole. The whole thing was developed on 2x RTX 4090s in my basement. I'm…
Run TRELLIS.2 Image-to-3D generation natively on Apple SiliconApr 2026 · github.com · ▲202I ported Microsoft's TRELLIS.2 (4B parameter image-to-3D model) to run on Apple Silicon via PyTorch MPS. The original requires CUDA with flash_attn, nvdiffrast, and custom sparse convolution kernels: none of which work on Mac. I replaced the CUDA-specific ops with pure-PyTorch alternatives: a gather-scatter sparse 3D convolution, SDPA attention for sparse transformers, and a Python-based mesh extraction replacing CUDA hashmap operations. Total changes are a few hundred lines across 9 files. Generates ~400K vertex meshes from single photos in about 3.5 minutes on M4 Pro (24GB). Not as fast as…
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 27d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 25d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 22d ago · openbot.run
Talanoa▲340Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 20d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 27d ago · corespeed.io
Source: Show HN post ↗
Launched alongside, November 2025
the whole month →
Guideflow▲1,338Create interactive product demos and step-by-step guides in seconds with Guideflow, the leading AI-powered demo automation tool.
AI · Nov 2025 · guideflow.com
Hardware and software projects by Ali Roberts
Life & fun · Nov 2025 · bitsnpieces.dev
Welltory▲1,028Measure your wellness. Use data to feel great every day.
Work · Nov 2025 · welltory.com
TrustMRR▲833LLM Gateway is a b2b artificial intelligence based in United States, founded in 2025. $259,539/month in verified Stripe revenue. $78,764 MRR. Ahrefs Domain Rating 52/100. growing 55% month-over-month. Track LLM Gateway's verified metrics on TrustMRR.
Growth · Nov 2025 · trustmrr.com
Boing▲782Life & fun · Nov 2025 · boing.greg.technology