Alternatives
Products that do what KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT does
- 1HF
2020 · github.com
- 2
- 3CL
Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory. A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed. Once the KV cache is offloaded,…
Jul 2026 · github.com
- 4
- 5
- 6TB
Jun 2026 · medium.com
- 7IR
I don't know Rust. Friday after work I realised that 90% of my IDE time now is just the commit/diff view — and even good IDEs feel heavy for that. So over the weekend I built a dedicated native tool for just that. Kyde is a macOS git commit + diff editor with one goal: be fast, do Git well. I'm curious whether anyone else mostly opens their IDE for git operations these days. It's open source, and there's a signed app in Releases.
Jun 2026 · github.com
- 8RR
runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU caches are sized from detected free/total VRAM — never from device-name presets. There is no conversion step, no sidecar cache files, no silent requantization. The GGUF on disk is the single source of truth. The result that made me want to post this: Tencent's Hy3 (295B total / 21B active sparse MoE, a single 97.8 GiB Q2_K GGUF) runs on my desktop…
Jul 2026 · github.com
- 9AT
While building a chat application I couldn't find find a free and opensource tool to store user sessions. This led to redcache-ai. The tool helps with semantic search, Retrieval Augmented Generation(RAG) and storage. This is an early version undergoing rapid iteration. Happy to answer questions and hear feedback.
2024 · github.com
- 10
- 11KA
2015 · kvpbase.tumblr.com
- 12FA
Hi HN, We're excited to introduce Fixstars AIBooster, our new performance engineering tool designed to significantly accelerate AI model training while optimizing GPU utilization. AIBooster provides: Real-time monitoring of GPU, CPU, memory, and power consumption. Clear visibility into performance bottlenecks, helping developers optimize AI workloads. Proven acceleration of AI training processes—users commonly achieve up to 2-3x speed improvements. Significant cost savings by maximizing infrastructure efficiency. It's free to try, requires minimal setup, and integrates seamlessly into your…
2025 · fixstars.com
- 13OS
2021 · github.com
- 14IM
It's a chrome extension that automatically loads the specs from the Hugging Face model card into the calculation. > To test it, install the extension (no registration/key needed) and navigate to a HF model page. Then click the "VRAM" icon on the top right to open the sidepanel. You can specify quantization, batch size, sequence length, etc. Works for inference & fine-tuning. If it does not fit on the specified GPUs, it gives you an advise on how to still run it (e.g. lowering precision). It is inspired at my work, where we were constantly exporting metrics from HF to estimate required…
2025 · chromewebstore.google.com
- 15IE
Hey HN, when building ML systems for industrial AI, we have learned that data inspection is critical during the ML development process. We are also big fans of the Hugging Face ecosystem. That is why we built an integration to our data exploration tool Spotlight that allows you to interactively explore Hugging Face datasets with one line of code. Spotlight lets you leverage model results such as predictions and embeddings to gain a deeper understanding in data segments and model failure modes. Currently, many many NLP, CV, Audio and multimodal datasets are supported both locally and on the…
2023 · huggingface.co
- 16ML
Time to first token is 39% faster Agent wall times decrease by 46% No swaps Tracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device. Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more. Built a ton of features
Jun 2026 · autotunellm.com
- 17UA
Most of the document parsers fail on real world challenges like complex tables, handwritten documents, historical document scans, equations, multi-column layouts, complex reading order, etc. We built Unsiloed Parser to handle exactly these cases. Our latest parser v3.1 achieved #1 rank and scored 88.0 strict pass-rate on olmOCR-Bench. We ran the evaluation across 1,403 PDFs and 8,413 unit tests using the unmodified upstream Allen AI scorer (olmocr==0.4.27) and found Unsiloed beats 18 other OCR services, including GPT-5.5, Claude Opus 4.7, LlamaParse, Reducto, Azure Document Intelligence, AWS…
May 2026
- 18AA
The weekend of last week I built chat.betterdb.com as a RAG over Valkey/Redis/Dragonfly docs. The goal was to eat our own dogfood and test publicly our caching libraries. It also saved me from having to come up with various demo/test scenarios, as I could extend the building in public to the demo. There is a tool-result cache sitting between the SDK and tools. Each call is normalized and then checked before executing. If it hits we return from the cache, and if not, we check the semantic cache, which embeds the prompt and checks with KNN via valkey-search. If the cosine…
May 2026
- 19TA
https://github.com/PlumeCat/tack Hi HN! Tack is a scripting language I've been working on sporadically for the past year or so, and intensely for the past few weeks. It originated out of a desire for something that was like Lua, but with a more familiar syntax, and without some of the other surprises in Lua such as the 1-indexed tables. It's also been a great learning project, and a very satisfying challenge! While the current version is early beta at best, I hope to continue working on it and maybe see some adoption. Despite the relative lack of optimization, I'm very…
2023 · github.com
- 20KA
Double-digit speed-ups over naïve K-Means Parallel by default (Rayon) Balanced & spherical variants in one crate Memory-savvy for multi-GB datasets Builder API that just feels ergonomic Docs are live, the license is Apache-2.0, and the repo is one `cargo add kentro` away. Give it a spin—curious to hear how it performs in your pipelines!
2025 · github.com
- 21RG
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev
- 22KP
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
Jun 2026 · github.com
- 23UI
Hey everyone! I am excited to share updates on four of my & my teams' open-source projects that take large-scale search systems to the next level: USearch, UForm, UCall, and StringZilla. These projects are designed to work seamlessly together, end-to-end—covering everything from indexing and AI to storage and networking. And yeah, they're optimized for x86 AVX2/512 and Arm NEON/SVE hardware. USearch [1]: Think of it as Meta FAISS on steroids. It's now quicker, supports clustering of any granularity, and offers multi-index lookups. Plus, it's got more native bindings than probably…
2023 · usearch-images.com
- 24KH
2019 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →