Dev tools · alternatives · 2026

24 alternatives to UL-SMF – Open-source linear-complexity ~300x KV-cache compression

The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 3

UL-SMF is an open-source hardware-software compression system designed for developers working with long-context Transformer models. It reduces Key-Value cache memory requirements by combining Finite Scalar Quantization… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
Needle2: 14MB agentic LLM for phones, wearables, smart home and robots▲537
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
Aug 2026 · cactuscompute.com · its alternatives →
2AT
A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)▲79
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
Aug 2026 · mikeayles.com · its alternatives →
3
We Beat MLPerf: Modern Storage for KV Offload and LLM Training▲35
High performance storage engine for efficient LLM inference and GPU Training.
Sep 2026 · theopenlake.com · its alternatives →
4
KVSplit – Run 2-3x longer contexts on Apple Silicon▲272
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com · its alternatives →
5OA
Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB▲112
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026 · its alternatives →
6
A general-purpose encoder-decoder framework for Tensorflow▲54
A general-purpose encoder-decoder framework for Tensorflow - google/seq2seq
2017 · github.com · its alternatives →
7
I ran a language model on a PS2▲46
The Emotion Engine has 32 MB of RAM total, so the trick is streaming weights from CD-ROM one matrix at a time during the forward pass — only activations, KV cache and embeddings live in RAM. This means models bigger than the RAM can still run, they just read more from disc. Had to build a custom quantized format (PSNT), hack endianness, write a tokenizer pipeline, and most of the PS2 SDK from scratch (releasing that separately). The model itself is also custom — a 10M param Llama-style architecture I trained specifically for this. And it works. On real hardware.
Mar 2026 · github.com · its alternatives →
8
Engrim – A universal, local-first SQLite memory engine for AI CLIs▲93
The Universal Cross-Model Episodic Memory Standard. Local-first, project-scoped SQLite memory engine for Google Antigravity, Claude Code, Cursor, and Windsurf. Zero cloud lock-in. - timgordontg/engrim
Sep 2026 · github.com · its alternatives →
9
An RDMA/Infiniband Distributed Cache for Fast Inference and Training▲13
A high-performance RDMA distributed file system for fast LLM Inference and GPU Training. - GitHub - blackbird-io/blackbird: A high-performance RDMA distributed file system for fast LLM Inference and GPU Training.
2025 · github.com · its alternatives →
10
Entropy-Guided Loop – How to make small models reason▲33
TLDR: A small, vendor-agnostic inference loop that turns token logprobs/perplexity/entropy into an extra pass and reasoning for LLMs. - Captures logprobs/top-k during generation, computes perplexity and token-level entropy. - Triggers at most one refine when simple thresholds fire; passes a compact “uncertainty report” (uncertain tokens + top-k alts + local context) back to the model. - In our tests on technical Q&A / math / code, a small model recovered much of “reasoning” quality at ~⅓ the cost while refining ~⅓ of outputs. I kept seeing “reasoning” models behave…
2025 · github.com · its alternatives →
11
MCP Memory – Fast Agent Memory Using Google's OKF and SQLite FTS5▲70
An OKF-backed Model Context Protocol (MCP) server delivering persistent long-term memory and SQLite FTS5 search for AI agents. - fellowgeek/mcp-memory
Aug 2026 · github.com · its alternatives →
12
YourMemory▲93
This is a project that I have been building for a while now, YourMemory is a solution to agentic memory which focuses on pruning of noise rather than hoarding of data. In the current state of agentic memory most of the context is stored in the form of a MD file or is derived through a RAG model where you store each and everything. Both of the solution leads to bloated context which does not optimize the usage of any tokens. In this system we only keep relevant data in our memory and prune all the unnecessary data. The relevance of a data is derived through multiple factors such as recall…
Apr 2026 · yourmemoryai.vercel.app · its alternatives →
13
Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload▲22
Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory. A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed. Once the KV cache is offloaded,…
Jul 2026 · github.com · its alternatives →
14
UltraCompress – first mathematically lossless 5-bit LLM compression▲6
Verified AI infrastructure for regulated deployment. UltraCompress (our wedge): near-lossless 5-bit compression with SHA-256-reproducible reconstruction - prove the model in production is the one you validated. 23 architectures (0.6B-405B), Hermes-3-405B @ 1.0066x. OpenAI-compatible API. pip install
May 2026 · github.com · its alternatives →
15
Mini_gzip – embeddable, minimal, in-memory Gzip API▲21
Embeddable minimal in-memory GZIP decompression library for embedded systems - wkoszek/mini_gzip
2015 · github.com · its alternatives →
16
Minimal LLM Post-Training Experiments on an 8GB GPU (SFT, DPO, GRPO)▲21
Minimal, readable LLM post-training experiments on one 8GB GPU. Measures forgetting, seed variance, and RL emergence. - pochenai/nano-llm-posttraining
Aug 2026 · github.com · its alternatives →
17
Reducing the size of PCSA to that of HyperLogLog▲45
Contribute to seiflotfy/pcsa development by creating an account on GitHub.
2018 · github.com · its alternatives →
18BS
BSE – Semantic Zip Engine for Text, Image and Audio▲11
We built BSE (Bramble Semantic Engine) – a semantic compressor that transforms natural inputs into low-dimensional structured representations. It's designed as a preprocessing engine for LLMs, capable of reducing long inputs into compact, logic-preserving forms across: 1. Language Extracts SVO (Subject, Verb, Object) structure Captures modifiers: adjectives/adverbs Restores pronouns from short-term memory Detects questions Computes: Compression Rate (%) Semantic Loss (%) Compares sentence compression outputs via SDC: Subject-Subject, Verb-Verb, Object-Subject similarity Sentence…
2025 · its alternatives →
19
Running Gemma-4 26B at 124 tokens/SEC on a CPU, no GPU▲10
I wanted to know how fast a 26B mixture-of-experts model could run on a desktop CPU with no GPU. Got ~40 tok/s single-stream (lossless) and ~124 batched. The surprising part was the byte budget: for this model you compress the output head (32% of per-token bytes), not the experts (16%). The writeup has the bandwidth roofline and the dead-ends; the repo has the reproducible recipe. Happy to answer questions. Repo: https://github.com/arun-prasath2005/gemma4-cpu-moe
Jun 2026 · apeg.dev · its alternatives →
20
Model2vec – Lightning-fast Static Embeddings for RAG/Semantic Search▲28
We’ve recently open-sourced Model2vec, a method to distill sentence transformers into static embeddings that outperform all previous approaches by a large margin on MTEB. Our new models set a new state-of-the-art for static embeddings. Main features: - Our best model (potion-base-8M) has only 8M parameters, which is ~30mb on disk - Inference is ~500x faster than the distilled base model (bge-base), on a CPU - New models can be distilled in 30 seconds on a CPU without requiring a dataset - just a vocabulary - Numpy-only inference: The packaged can be install the package with minimal…
2024 · github.com · its alternatives →
21BA
Butter, a muscle memory cache for LLMs▲23
Hi HN, Erik here. Today we launch Butter, an OpenAI-compatible API proxy that caches LLM generations and serves them deterministically on revisit. Since April, we’ve been working on this concept of “muscle memory,” or deterministic replay, for agent systems performing automations. You may recall our first post in May, launching a python package called Muscle Mem: https://news.ycombinator.com/item?id=43988381 Since then, the product has evolved entirely, now taking the form of an LLM Proxy. For a deep dive into this process, check out:…
2025 · docs.butter.dev · its alternatives →
22
NeuroFlow 55.8x video inference speedup for Vision Transformers PyTorch▲8
Official PyTorch implementation of NeuroFlow: EMA-Gated Temporal Sequence Compression for Vision Transformers. Achieves up to 55.8x wall-clock speedup for video inference via semantic surprise routing and a training-free Dual-Memory Reconstruction Protocol. - ynnk-research/-NeuroFlow
May 2026 · github.com · its alternatives →
23KC
KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT▲20
If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a single notebook—covering tokenization, embeddings, causal self-attention, training, and autoregressive text generation without relying on high-level abstractions. The focus was on mechanistic clarity rather than scale or performance, and the notebook is structured to read more like a technical walkthrough than an experiment log. Feedback from people who’ve built or studied transformers would be very welcome.
May 2026 · pythongiant.github.io · its alternatives →
24
Zero-power photonic language model–code▲18
The model uses a 1024-dimensional complex Hilbert space with 32 layers of programmable Mach–Zehnder meshes (Reck architecture) and derives token probabilities directly via the Born rule. Despite using only unitary operations and no attention mechanism, a 1024×32 model achieves coherent TinyStories generation after < 1.8 hours of training on a single consumer GPU. This is Part 1 - the next step is physical implementation with $50 of optics from AliExpress.
Nov 2025 · zenodo.org · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →