An agent that tunes its own cache
The weekend of last week I built chat.betterdb.com as a RAG over Valkey/Redis/Dragonfly docs. The goal was to eat our own dogfood and test publicly our caching libraries. It also saved me from having to come up with various demo/test scenarios, as I could extend the building in public to the demo. There is a tool-result cache sitting between the SDK and tools. Each call is normalized and then checked before executing. If it hits we return from the cache, and if not, we check the semantic cache, which embeds the prompt and checks with KNN via valkey-search. If the cosine…
What it does
In the maker’s words, at launch
The weekend of last week I built chat.betterdb.com as a RAG over Valkey/Redis/Dragonfly docs. The goal was to eat our own dogfood and test publicly our caching libraries. It also saved me from having to come up with various demo/test scenarios, as I could extend the building in public to the demo. There is a tool-result cache sitting between the SDK and tools. Each call is normalized and then checked before executing. If it hits we return from the cache, and if not, we check the semantic cache, which embeds the prompt and checks with KNN via valkey-search. If the cosine distance is close enough, we again skip the LLM and stream the cached response. In both cases, if we miss, we store the prompt embedding, actual model, input and output tokens from OpenAI's usage report, so a future hit has the dollars avoided as data. The two tiers handle different shapes. Predefined questions, copy-pasted questions, checking the same thing again after time - produces byte-identical strings the tool cache catches. Human paraphrase is what the semantic tier exists for. This Wednesday was a bank holiday where I live, so I used to extend it further - the libraries the chat relies on now store metadata in the Valkey (or Redis if that's your preference) instance, then our monitoring reads and analyze that data and suggests improvements. These are exported also through our MCP server, so the chat's agent can check and create suggestions as well, and since this is just a demo, it can also approve its suggestions (do not do this on real production environment, unless you are a true LLM believer). The libs also read the config from the Valkey instance, so there is no restart needed. I hooked it on cron inside Vercel and let it run over the night and next day. Between Run 1 and Run 3, it started making less tool calls. The first run it suggested several different TTL changes and applied them. Run 2 and 1 had similar suggestions, because the TTL is the wrong point of control - they take natural language input (`How fast is XADD?` vs `XADD performance` are two different strings, that "mean" the same thing) so the tool cache doesn't fire and are covered by the semantic cache. An actual fix would be to move these tools from the exact-match into the semantic cache checks - a code change, not a config change. It was an indicator of a problem the system can't fix on its own. In the future the routing might also become configurable to solve this without redeploying and test and verify in quicker loops. Run 3 just didn't propose anything new - 15 -> 13 -> 8 tool calls across the three runs. Curious how others running similar loops decide what the agent can touch. Am I too skeptical of hallucinations and overly cautious? The chat can be found at https://chat.betterdb.com (it has links to all of the repos in it) And a more detailed write up can be found at https://www.betterdb.com/blog/cache-that-tunes-itself
Does the same job
all alternatives →
- ATA tool to give large language models better memory2024 · github.com · ▲7
While building a chat application I couldn't find find a free and opensource tool to store user sessions. This led to redcache-ai. The tool helps with semantic search, Retrieval Augmented Generation(RAG) and storage. This is an early version undergoing rapid iteration. Happy to answer questions and hear feedback.
- BMBetterDB, MIT Valkey-native context layer for AI agentsJun 2026 · github.com · ▲5
Today we released an open, Valkey-native context layer for AI agents as part of our packages at BetterDB (agent memory, semantic + multi-tier caching, typed retrieval) that run on a Valkey instance no matter where it is - no vendor lock-in. We even started provisioning Valkey instances starting today. Packages are shipped on npm and PyPi. Why we made it: BetterDB originally started as a monitoring and observability platform for Valkey, Redis and any RESP compatible db. This is still the core of the product, but in the process of building this, we kept seeing that one of the fastest-growing…
- CHCache Horse: An HTTP cache/batch service2025 · cache.horse · ▲5
Hi, I'm fiiv, and I'm the creator of Cache Horse. I built it because I wanted an easy plug-n-play solution to caching and simplifying HTTP requests - in particular, on frontend. First, I was fetching data like daily weather, historic currency exchange numbers, air quality readings - and many of those APIs have quota limits. And second, since I was already caching them, I thought it would be useful to batch them together - so I built that feature in. I would love to hear your feedback and thoughts on the project. Thanks!
- SLsoft-launching one of my pet projects2011 · ▲27
So a while ago I was thinking it would be neat to build a site that had the most absolute utility. The idea I came up with was a site to store those little protips that are super useful, if you know them ahead of time. I hired a guy to work on it with me, and ClueDB was born: http://cluedb.com/ Please give it a whirl (it uses Twitter auth as login but doesn't tweet anything) and tell me what you think! (Various trivia: My first try at a project coded by someone else and "product managed" by me. Runs on Flask + MongoDB. Also, this is totally unrelated to my startup.)
- SISemcache – I built a semantic cache in Rust2025 · github.com · ▲5
Have a look at my semantic caching project! It's built to easily integrate in existing LLM workflows, you can use it as a proxy where the cache forwards missed requests without modification to a specified upstream, automatically updating it's cache with the response. You can also use it as a cache-aside cache with a provided python library. It works by computing embedding vectors of input queries, and matches them to seen query + response pairs using a vector store. Everything is in-memory, so it should be blazing fast :)
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, May 2026
the whole month →
Parallel agents, diff reviewer, and multi-model comparisons
Dev tools · May 2026 · kilo.ai
- NW
Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices. We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we conducted investigations that led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly (match query to tool name, extract argument values, emit JSON), not reasoning. Cross-attention…
Life & fun · May 2026 · github.com
- FM
Dev tools · May 2026 · github.com