AI · alternatives · 2026

24 alternatives to KV-psi, using Linux PSI to to trim an LLM KV cache

I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
KVSplit – Run 2-3x longer contexts on Apple Silicon▲272
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com · its alternatives →
2
Find the best local LLM for your hardware, ranked by benchmarks▲283
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly. - Andyyyy64/whichllm
May 2026 · github.com · its alternatives →
3
Shoehorn, a library to quantize an LLM to fit your Mac's VRAM▲6
I made this after seeing someone posit the idea online yesterday over lunch then spent some time refining it. So far it's pretty impressive IMO! Right now I am running Qwen3-30B-A3B on my 24gb unified memory m4 MacBook Pro at 50 tok/sec and this should definitely not be working for such a large model on my middling hardware. Things are detailed in the README to get up and running and DESIGN.md has details on all the choices and such made along the way.
Aug 2026 · github.com · its alternatives →
4
Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload▲22
Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory. A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed. Once the KV cache is offloaded,…
Jul 2026 · github.com · its alternatives →
5
We Beat MLPerf: Modern Storage for KV Offload and LLM Training▲35
High performance storage engine for efficient LLM inference and GPU Training.
Sep 2026 · theopenlake.com · its alternatives →
6
Plock: Use a local LLM from anywhere in your OS▲6
After seeing a cool demo of a hack on Twitter, I built a cross platform version of it that works well and uses streaming. From anywhere on Mac and Linux, trigger Ollama and optionally feed it your clipboard. I built it yesterday and it's already very useful to me. I'm pretty excited about it and wanted to share!
2024 · github.com · its alternatives →
7
A "Cram tests" script for windows shells▲6
It's quite limited but does the job, I invite anybody not aware of what cram tests are to give them a try, being on Unix with the original cram or Windows with this one ;)
Nov 2025 · github.com · its alternatives →
8
Llmtop – Htop for LLM Inference Clusters (vLLM, SGLang, Ollama, llama)▲5
I work on inference scheduling — KV cache-aware routing, load balancing across GPU workers, that kind of thing. I wanted something like k9s but for my inference stack. Nothing existed, so I built it. llmtop is a real-time terminal dashboard for LLM inference workers. It scrapes the Prometheus /metrics endpoints that vLLM, SGLang, and LMCache already expose and shows everything in one view: KV cache usage, queue depth, TTFT/ITL latencies (P50/P99 from histogram buckets), token throughput, prefix cache hit rates. Color-coded — red means go fix it. ``` brew install…
Mar 2026 · github.com · its alternatives →
9
I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp▲5
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
Jul 2026 · github.com · its alternatives →
10
I built a tool to check if your computer can run LLMs locally▲8
Built a simple web app that tells you which open-source LLMs will work on your hardware. It auto-detects your specs, shows compatible models from Hugging Face, gives realistic performance estimates (tokens/sec), and recommends quantization settings. You can also manually input specs to see "what if I upgraded my RAM?" Made this after wasting time downloading giant models only to find they crawled on my hardware. Hope it saves you some frustration!
2025 · caniusellm.com · its alternatives →
11
Unfat, a library to easily train and distill LoRAs for LLMs▲6
Hey HN! After using a combination of Unsloth and Axolotl a lot, and finding it generally painful to figure out the right performance tuning for things like batch sizing and multi-GPU sharding, I wrote a small Python lib that sets up known-good LoRA training configurations for Llama 3.1 8B and 70B Instruct, and includes helpers for distilling from larger models or training on serverless finetuning platforms, and includes a walkthrough for distilling DeepSeek-R1 into a Llama 3.1 8B LoRA... But you can use it for pretty much any finetuning task, not just distilling large models!
2025 · github.com · its alternatives →
12
An LLM Running on a PS Vita▲5
Hello all, a couple of moons ago I ported karpathy's llama2.c code to run inference on the TinyStories 260K & 15M checkpoints on the on the PS Vita with the ability to download/delete the models on device. Runs showed that the 260K model ran at ~120 tok/s and at 15M ran at 1.8 tok/s, which probably could be a bit higher if it weren't a single threaded application. Had fun working on it as a weekend project, check it out for yourselves: https://github.com/callbacked/psvita-llm
2025 · github.com · its alternatives →
13
Nabu (TTS Reader and LLM Playground on Android)▲7
With the latest launch from Google I've added support for Gemma 3 270M, the speed for local LLM to TTS token time is incredible! This is an heavy obvious work in progress - any contributions or tips would be welcome. The idea is to have a fast moving edge model playground, and maybe have some utility (like the e reader) on the side.
2025 · github.com · its alternatives →
14
Puteron, Systemd Competitor▲7
I was debating when to post this, but I've been using it locally in a variety of situations for several months and I think it's pretty stable (and usable!). There are more features I'd like to add, but I wanted to see if there was much interest before spending much more time on it right now.
2025 · github.com · its alternatives →
15
I trained a custom LLM to create Apple Shortcuts▲5
Hi! This is a version 2.0 2x Context window 7-8x Faster Less hallucinations :)
2025 · shortcutbuilder.com · its alternatives →
16
Ksm – a lightweight hypervisor for Windows (eptp vmfunc+idt shadow)▲12
A fast, hackable and simple x64 VT-x hypervisor for Windows and Linux. Builtin userspace sandbox and introspection engine. - asamy/ksm
2016 · github.com · its alternatives →
17
Install Nix on Your Steam Deck▲5
I wrote an article about how to get Nix on your Steam Deck! Big takeaways for me: * Steam Decks are very fun to hack on * Linux Consoles are good * There are more knobs in systemd units than I remember * Flatcar Linux is a thing that exists * Remoting into a console is fun * A/B boots are less interesting than one might expect when you're used to NixOS
2022 · determinate.systems · its alternatives →
18
Trim Transformer: A transformer for physics models▲10
Trim is building a foundation model for physics. We found almost all well-maintained transformer-based architectures are optimized for language models. We’re releasing our own open-source transformer package that is optimized for physics. It achieves a 90% reduction in memory usage and a 3.5x speedup in a standard Navier-Stokes dataset.
2025 · github.com · its alternatives →
19IB
I built a local Elixir/Python pipeline to curate 14,000 RAW photos▲5
I had 14,000 photos sitting on a drive and wanted an excuse to play with local vision models and Elixir/Phoenix. I originally tried to get LLaVA to tell me if a photo was 'good' or matched my style, but quickly learned that LLMs have terrible taste. I ended up demoting the LLM to just extract metadata, and built a custom CLIP/Ridge Regression pipeline to actually learn my preferences based on how I rate things. The stack is Phoenix/Oban on the orchestrator side, and Python/FastAPI/Instructor for the AI workers. Happy to answer any questions about the architecture,…
Apr 2026 · qwelian.com · its alternatives →
20AN
A new engine to run Kimi K3 on a laptop▲7
Kimi K3 has 2.78 trillion parameters and ships as 1.42 TB of weights. It clearly does not fit in the memory of a laptop. But K3 is a Mixture-of-Experts model. For each token, only a small fraction of its 896 experts per layer is activated. That changes the problem: the entire model does not need to be resident in RAM, as long as the weights required by each token can be reached quickly enough. We built WASTE — the Weight-Aware Streaming Tensor Engine — to explore that idea. WASTE keeps the dense, repeatedly used part of the model resident in memory, stores the routed experts in an…
Jul 2026 · its alternatives →
21
We built a <60ms, open-source alternative to E2B using RustVMM and KVM▲7
Over the past few months, as we scaled our internal AI Agents, we hit a dead end: Running LLM-generated arbitrary code in Docker is basically running naked on security due to container escape risks. But using full traditional VMs takes minutes to boot and eats too much memory to support high-density concurrency. We loved the developer experience of SaaS sandboxes on the market, but they are closed-source, expensive, and have too high a barrier to entry for self-hosting. So, our team decided to build our own. After months of grinding, using RustVMM and KVM, we built a blazing-fast,…
Apr 2026 · github.com · its alternatives →
22
I made a OSS alternative to Weights and Biases▲6
Hey HN! I made a completely open sourced alternative to Weights and Biases with (insert cringe) blazingly fast performance (yes we use rust and clickhouse) Weights and Biases is super unperformant, their logger blocks user code... logging should not be blocking, yet they got away with it. We do the right thing by being non blocking. Would love any thoughts / feedbacks / roasts etc
2025 · github.com · its alternatives →
23
TurboQuant-MoE:8.5x KV-Cache Compression
8.5x KV-cache compression for LLM inference
Mar 2026 · github.com · its alternatives →
24
Tokensift, an open-sourced token-efficiency linter for LLM prompts▲6
Token-efficiency linter for LLM prompts and payloads - ritenv/tokensift
Aug 2026 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →