AI · alternatives · 2026

24 alternatives to Llmbuffer – Python library for cache-optimized LLM conversation history

I was not getting good cache utilization when including dynamic context in agent threads. After a lot of experimentation, I found a good pattern that minimizes how often long lived conversation history gets modified while still supporting dynamic context. It has flexible hooks for doing things like truncating or summarizing tool outputs when transitioning messages to the long term history. And I'm seeing >>90% of tokens hitting the cache for my agents despite including a lot of dynamic user context. There are a wide range of agent prompting strategies so I'd love to hear where this library…

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
Context Gateway▲220
We built an open-source proxy that sits between coding agents (Claude Code, OpenClaw, etc.) and the LLM, compressing tool outputs before they enter the context window. Demo: https://www.youtube.com/watch?v=-vFZ6MPrwjw#t=9s. Motivation: Agents are terrible at managing context. A single file read or grep can dump thousands of tokens into the window, most of it noise. This isn't just expensive — it actively degrades quality. Long-context benchmarks consistently show steep accuracy drops as context grows (OpenAI's GPT-5.4 eval goes from 97.2% at 32k to 36.6% at 1M…
Mar 2026 · github.com · its alternatives →
2
An LLM response cache that's aware of dynamic data▲17
Raymond here from Butter.dev, an LLM response cache built as a chat-completions proxy. Today we're launching a key feature for the platform: the ability to generalize on dynamic, templated inputs. Caching at the HTTP request level has the obvious problem of generalizability. Nearly no request is identical, due to templated variables (like names) and metadata (like timestamps), so exact-match cache lookups rarely hit. We solve this at Butter by using LLMs to detect dynamic content in requests and derive their inter-relationships, allowing the cache entry to be stored as a template + variables…
Jan 2026 · blog.butter.dev · its alternatives →
3
Vector Cache▲9
A Python Library for Efficient LLM Query Caching
2024 · github.com · its alternatives →
4RC
Reduce ChatGPT costs 10x with distributed cache for LLMs▲22
Hello HN! We're building a caching solution for LLMs (ChatGPT, Claude). By combining cutting-edge approaches, such as edge computing, prompt compression, vectorization, and others - it can reduce your AI bills by up to 10x and significantly lower response times. Key Features: - cost efficiency: our system stores frequent queries, reducing the number of upstream (paid) API calls - fast responses: with various nodes globally, we reduce latency by serving data from the nearest location - scalability: designed to handle increasing loads and data sizes without degrading performance. The cache…
2024 · edgematic.dev · its alternatives →
5
Bindly▲76
Hi HN, I’m a product designer by background, not a traditional software engineer. Over the last year, tools like Claude and ChatGPT completely changed how I work. I started with small internal tools, and now I’m “vibe coding” multiple highly personalized apps. As I iterated between Claude Code (implementation) and ChatGPT (ideation / thinking), I kept running into the same problem. To think clearly, I had to repeatedly re-explain: - what the code currently does, - what changed recently, - and why certain decisions were made. That re-summarization step became a real bottleneck. So I…
2016 · bind.ly · its alternatives →
6
Butter – A Behavior Cache for LLMs▲50
Hi HN! I'm Erik. We built Butter, an LLM proxy that makes agent systems deterministic by caching and replaying responses, so automations behave consistently across runs. - It’s a chat completions compatible endpoint, making it easy to drop into existing agents with a custom base_url - The cache is template-aware, meaning lookups can treat dynamic content (names, addresses, etc.) as variables You can see it in action in this demo where it memorizes tic-tac-toe games: https://www.youtube.com/watch?v=PWbyeZwPjuY Why we built this: before Butter, we were Pig.dev (YC W25), where we…
Oct 2025 · butter.dev · its alternatives →
7
A local-first memory store for LLM agents (SQLite)▲48
Local persistent memory store for LLM applications including claude desktop, github copilot, codex, antigravity, etc. - CaviraOSS/LongMemory
Dec 2025 · github.com · its alternatives →
8BA
Butter, a muscle memory cache for LLMs▲23
Hi HN, Erik here. Today we launch Butter, an OpenAI-compatible API proxy that caches LLM generations and serves them deterministically on revisit. Since April, we’ve been working on this concept of “muscle memory,” or deterministic replay, for agent systems performing automations. You may recall our first post in May, launching a python package called Muscle Mem: https://news.ycombinator.com/item?id=43988381 Since then, the product has evolved entirely, now taking the form of an LLM Proxy. For a deep dive into this process, check out:…
2025 · docs.butter.dev · its alternatives →
9ZL
Zep – Long-Term Memory Store for LLM Apps▲7
Zep is a long-term memory store designed for conversational AI applications built using modern LLMs. It handles the storage, summarization, embedding, indexing, and enrichment of chat histories, and offers developers a simple, low-latency API to this data. Chat history storage is an infrastructure challenge all developers and enterprises face as they look to move from prototypes to deploying conversational AI applications that provide rich and intimate experiences to users. Key features include long-term memory persistence, auto-summarization, vector search, auto-token counting, and Python…
2023 · its alternatives →
10CA
ChatIndex – A Lossless Memory System for AI Agents▲17
Current AI chat assistants face a fundamental challenge: context management in long conversations. While current LLM apps use multiple separate conversations to bypass context limits, a truly human-like AI assistant should maintain a single, coherent conversation thread, making efficient context management critical. Although modern LLMs have longer contexts, they still suffer from the long-context problem (e.g. context rot problem) - reasoning ability decreases as context grows longer. Memory-based systems have been invented to alleviate the context rot problem, however, memory-based…
Nov 2025 · its alternatives →
11
I made a state machine framework for guided conversations with LLMs▲19
It’s written in Python and I call it GoalChain. It lets you build a conversation flow graph that the user traverses. When there’s enough input it spits out a dictionary with the defined fields. Otherwise it will jump state to state as led by the user. It was fun to write, and it’s surprisingly effective if you keep in mind you’re prompt-engineering every string and field name. README.md has a mini-tutorial. Would be cool to get some ideas for how to build it further and what improvements I could make.
2024 · github.com · its alternatives →
12AC
Agent-cache – Multi-tier LLM/tool/session caching for Valkey and Redis▲18
Multi-tier exact-match cache for AI agents backed by Valkey or Redis. LLM responses, tool results, and session state behind one connection. Framework adapters for LangChain, LangGraph, and Vercel AI SDK. OpenTelemetry and Prometheus built in. No modules required - works on vanilla Valkey 7+ and Redis 6.2+. Shipped v0.1.0 yesterday, v0.2.0 today with cluster mode. Streaming support coming next. Existing options locked you into one tier (LangChain = LLM only, LangGraph = state only) or one framework. This solves both. npm:…
Apr 2026 · its alternatives →
13
Librarian – Cut token costs by up to 85% for LangGraph and OpenClaw▲8
Hi HN, I'm building Librarian (https://uselibrarian.dev/), an open-source (MIT) context management tool that stops AI agents from burning tokens by blindly re-reading their entire conversation history on every turn. The Problem: If you're building agentic loops in frameworks like LangGraph or OpenClaw, you hit two walls fast: Financial Cost: Token usage scales quadratically over long conversations. Passing the whole history every time gets incredibly expensive. Context Rot: As the context window fills up, the LLM suffers from the "Lost in the Middle" effect. Response latency…
Feb 2026 · uselibrarian.dev · its alternatives →
14
Context Compass▲16
I built a small Chrome extension for people who write long conversations in Claude and don’t want to unexpectedly hit the 200,000-token limit. Context Compass automatically counts words and estimates token usage as you write. It updates in real time, gives you a visible percentage bar, and helps power users manage sprawling chats without losing work or flow. I built it because I kept getting the “max context” error and realized there was no way to track usage in Claude. No signups, no tracking, just a simple install. Would love any feedback or ideas for improvements. Thanks!
2025 · chromewebstore.google.com · its alternatives →
15
GPTCache – Redis for LLMs▲7
Hey folks, As much as we love GPT-4, it's expensive and can be slow at times. That's why we built GPTCache - a semantic cache for autoregressive LMs - atop the vector database Milvus and SQLite. GPTCache provides several benefits: 1) reduced expenses due to minimizing the number of requests and tokens sent to the LLM service 2) enhanced performance by fetching cached query results directly 3) improved scalability and availability by avoiding rate limits, and 4) a flexible development environment that allows developers to verify their application's features without connecting to LLM APIs or…
2023 · github.com · its alternatives →
16
A tool to give large language models better memory▲7
While building a chat application I couldn't find find a free and opensource tool to store user sessions. This led to redcache-ai. The tool helps with semantic search, Retrieval Augmented Generation(RAG) and storage. This is an early version undergoing rapid iteration. Happy to answer questions and hear feedback.
2024 · github.com · its alternatives →
17CS
Conversational search in less than 500 lines of Python▲6
Hi there! We built a Bing-chat-like conversational search engine with less than 500 lines of python code (and a similar amount of frontend script). The source code is fully open-source at https://github.com/leptonai/search_with_lepton , and we put up a live demo at https://search.lepton.run/. Let us know what you think!
2024 · search.lepton.run · its alternatives →
18LP
Llmswap – Python package to reduce LLM API costs by 50-90% with caching▲12
I built llmswap to solve a problem I kept hitting in hackathons - burning through API credits while testing the same prompts repeatedly during development. It's a simple Python package that provides a unified interface for OpenAI, Anthropic, Google Gemini, and local models (Ollama), with built-in response caching that can cut API costs by 50-90%. Key features: - Intelligent caching with TTL and memory limits - Context-aware caching for multi-user apps - Auto-fallback between providers when one fails - Zero configuration - works with environment variables from llmswap import LLMClient client…
2025 · pypi.org · its alternatives →
19
Unpack – a lightweight way to steer Codex/Claude with phased docs▲7
I've been using LLMs for long discovery and research chats (papers, repos, best practices), then distilling that into phased markdown (build plan + tests), then handing those phases to Codex/Claude to implement and test phase by phase. The annoying part was always the distillation and keeping docs and architecture current, so I built Unpack: a lightweight GitHub template plus docs structure and a few commands that turns conversations into phases/specs and keeps project docs up to date as the agent builds. It can also generate Mintlify-friendly end-user docs. There are other…
Feb 2026 · github.com · its alternatives →
20OS
OpenSkills – Stop bloating your LLM context with unused instructions▲9
Hello HN, I’ve been building AI agents lately and ran into a common "Context Bloat" problem. When an agent has 20+ skills, stuffing every system prompt, reference doc, and tool definition into a single request quickly hits token limits and degrades model performance (the "lost in the middle" problem). To solve this, I built OpenSkills, an open-source SDK that implements a Progressive Disclosure Architecture for agent skills. The Core Concept: Instead of loading everything upfront, OpenSkills splits a skill into three layers: Layer 1 (Metadata): Light-weight tags and triggers (always loaded…
Jan 2026 · its alternatives →
21
Chitchat, an easy-to-use, cross-platform, chat-based LLM interface▲5
Just finished the first draft of my weekend project. Sadly my industry is far away from all the exciting machine learning developments happening right now, so I wrote this project as my first exploration into the world of LLMs. It's not perfect, but I'm excited to see where the project goes from here! https://github.com/clarkmcc/chitchat My main motivations were: - Easy-of-use: Many models are supported out-of-the-box so users don't have to figure out how to download, where to save, etc. - Intuitive: A clean interface - Cross platform: The project is written in Rust and…
2023 · clarkmccauley.com · its alternatives →
22
N0x – LLM inference, agents, RAG, Python exec in browser, no back end▲9
Built this because I was tired of every AI tool shipping my data to someone else server n0x runs the full stack LLM inference via WebGPU, autonomous ReAct agents, RAG over your own docs, sandboxed Python execution via Pyodide all inside a single browser tab. No account No keys No backend Models download once, cache in IndexedDB permanently. Biggest challenge was context window budgeting for the agent loop and making the WASM vector search non-blocking. Happy to talk architecture. GitHub: https://github.com/ixchio/n0x | Live demo: https://n0x-three.vercel.app
Mar 2026 · n0xth.vercel.app · its alternatives →
23
Prompt-refiner – Lightweight optimization for LLM inputs and RAG▲7
Hi HN, While building RAG agents, I noticed a lot of token budget was wasted on formatting overhead (HTML tags, JSON structure, whitespace). Existing solutions felt too heavy (often requiring torch/transformers), so I wrote this lightweight, zero-dependency library to solve it. It includes strategies for context packing, PII redaction, and tool output compression. Benchmarks show it can save ~15% of tokens with negligible latency overhead (<0.5ms). Happy to answer any questions!
Dec 2025 · github.com · its alternatives →
24
XTrace MemHub▲3
Turn your GPT/Claude/Gemini history into a LLM-Wiki Mindmap
May 2026 · xtrace.ai · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →