Alternatives
Products that do what GPTCache – Redis for LLMs does
Hey folks, As much as we love GPT-4, it's expensive and can be slow at times. That's why we built GPTCache - a semantic cache for autoregressive LMs - atop the vector database Milvus and SQLite. GPTCache provides several benefits: 1) reduced expenses due to minimizing the number of requests and tokens sent to the LLM service 2) enhanced performance by fetching cached query results directly 3) improved scalability and availability by avoiding rate limits, and 4) a flexible development environment that allows developers to verify their application's features without connecting to LLM APIs or…
- 1
- 2LS
Hi HN, I built llm.sql, an LLM inference framework that reimagines the LLM execution pipeline as a series of structured SQL queries atop SQLite. The motivation: Edge LLMs are getting better, but hardware remains a bottleneck, especially RAM (size and bandwidth). When available memory is less than the model size and KV cache, the OS incurs page faults and swaps pages using LRU-like strategies, resulting in throughput degradation that's hard to notice and even harder to debug. In fact, the memory access pattern during LLM inference is deterministic - we know exactly which weights are needed…
Apr 2026
- 3SI
Have a look at my semantic caching project! It's built to easily integrate in existing LLM workflows, you can use it as a proxy where the cache forwards missed requests without modification to a specified upstream, automatically updating it's cache with the response. You can also use it as a cache-aside cache with a provided python library. It works by computing embedding vectors of input queries, and matches them to seen query + response pairs using a vector store. Everything is in-memory, so it should be blazing fast :)
2025 · github.com
- 4LP
I was not getting good cache utilization when including dynamic context in agent threads. After a lot of experimentation, I found a good pattern that minimizes how often long lived conversation history gets modified while still supporting dynamic context. It has flexible hooks for doing things like truncating or summarizing tool outputs when transitioning messages to the long term history. And I'm seeing >>90% of tokens hitting the cache for my agents despite including a lot of dynamic user context. There are a wide range of agent prompting strategies so I'd love to hear where this library…
Jun 2026 · github.com
- 5AL
2022 · github.com
- 6
- 7OS
2022 · wundergraph.com
- 8AL
Try it out here: https://labs.refuel.ai/playground Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets. It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction.
2023
- 9IB
After fine-tuning GPT for a personal project, I realized how tedious it is to write plain text in a massive JSON file. That's why I built this app for my own use, and I want to see if others could benefit from a tool like this as well ;)
2024 · finetuna-ui.com
- 10ML
Time to first token is 39% faster Agent wall times decrease by 46% No swaps Tracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device. Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more. Built a ton of features
Jun 2026 · autotunellm.com
- 11AA
The weekend of last week I built chat.betterdb.com as a RAG over Valkey/Redis/Dragonfly docs. The goal was to eat our own dogfood and test publicly our caching libraries. It also saved me from having to come up with various demo/test scenarios, as I could extend the building in public to the demo. There is a tool-result cache sitting between the SDK and tools. Each call is normalized and then checked before executing. If it hits we return from the cache, and if not, we check the semantic cache, which embeds the prompt and checks with KNN via valkey-search. If the cosine…
May 2026
- 12NT
With the latest launch from Google I've added support for Gemma 3 270M, the speed for local LLM to TTS token time is incredible! This is an heavy obvious work in progress - any contributions or tips would be welcome. The idea is to have a fast moving edge model playground, and maybe have some utility (like the e reader) on the side.
2025 · github.com
- 13CH
Hi, I'm fiiv, and I'm the creator of Cache Horse. I built it because I wanted an easy plug-n-play solution to caching and simplifying HTTP requests - in particular, on frontend. First, I was fetching data like daily weather, historic currency exchange numbers, air quality readings - and many of those APIs have quota limits. And second, since I was already caching them, I thought it would be useful to batch them together - so I built that feature in. I would love to hear your feedback and thoughts on the project. Thanks!
2025 · cache.horse
- 14LH
I work on inference scheduling — KV cache-aware routing, load balancing across GPU workers, that kind of thing. I wanted something like k9s but for my inference stack. Nothing existed, so I built it. llmtop is a real-time terminal dashboard for LLM inference workers. It scrapes the Prometheus /metrics endpoints that vLLM, SGLang, and LMCache already expose and shows everything in one view: KV cache usage, queue depth, TTFT/ITL latencies (P50/P99 from histogram buckets), token throughput, prefix cache hit rates. Color-coded — red means go fix it. ``` brew install…
Mar 2026 · github.com
- 15CA
Jun 2026 · github.com
- 16TI
Hello HN, While browsing the Python docs yesterday, I discovered that the latest 3.12 version has added support for a `python3 -m sqlite3` interactive shell. I looked into the source code, and its implementation was simple, giving me an idea: Why not hook the beautiful llm library by simonw into such an interactive shell, and thus have direct LLM support in SQLite? Without writing a C extension, build a shared object and all that fuss. Well, now you can `pip install tsellm` and do just that. demo gif:…
2024 · github.com
- 17AT
While building a chat application I couldn't find find a free and opensource tool to store user sessions. This led to redcache-ai. The tool helps with semantic search, Retrieval Augmented Generation(RAG) and storage. This is an early version undergoing rapid iteration. Happy to answer questions and hear feedback.
2024 · github.com
- 18MI
Hi HN! I lead product at Vectara and we've just released a new LLM in our platform that outperforms GPT4 and Gemini 1.5 Pro on RAG tasks. Vectara is a Retrieval Augmented Generation (RAG) platform primarily deployed as a SaaS service which includes a generous free tier so you can try it for free. The way we've been able to offer a "better but cheaper" is that we focus a lot of our attention on taking smaller models (which can be hosted in a cost efficient way) and fine tuning them to specific tasks: in this case RAG. This ends up with a model that is less capable of arbitrary tasks like…
2024 · vectara.com
- 19NL
Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets. It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction. Here is the interactive demo: https://labs.refuel.ai/playground. Pretty fun to play with!
2023
- 20LM
I built a super easy to integrate memory storage and retrieval system for NodeJS projects because I saw a need for information to be shared and persisted across LLM chat sessions (and many other LLM feature interactions). I tried to keep the barrier to use as low as possible so I included built-in support for major LLMs (GPT, Gemini, and Claude) as well as major vector store providers (Weaviate and Pinecone). The memory store works by ingesting and automatically extracting “memories” (summarized single bits of information) from LLM interactions and vectorizing those. When you want to provide…
Mar 2026 · github.com
- 21LS
LLMStack is a low-code platform that can be used to build LLM apps, chatbots and integrate AI experiences into existing products/workflows. It comes with everything out of the box that one needs to build LLM apps locally. It can also be used in a multi-tenant setting, making it available for everyone to use in an enterprise. Some highlights of the platform: - Chain multiple LLM models allowing for complex pipelines - Includes a vector database and necessary connectors to help enrich LLM responses with private data - App templates tailored to specific use cases to quickly build LLM apps…
2023 · github.com
- 22CO
So I've been building ClawMem, an open-source context engine that gives AI coding agents persistent memory across sessions. It works with Claude Code (hooks + MCP) and OpenClaw (ContextEngine plugin + REST API), and both can share the same SQLite vault, so your CLI agent and your voice/chat agent build on the same memory without syncing anything. The retrieval architecture is a Frankenstein, which is pretty much always my process. I pulled the best parts from recent projects and research and stitched them together: [QMD](https://github.com/tobi/qmd) for the…
Mar 2026 · github.com
- 23CA
Just finished the first draft of my weekend project. Sadly my industry is far away from all the exciting machine learning developments happening right now, so I wrote this project as my first exploration into the world of LLMs. It's not perfect, but I'm excited to see where the project goes from here! https://github.com/clarkmcc/chitchat My main motivations were: - Easy-of-use: Many models are supported out-of-the-box so users don't have to figure out how to download, where to save, etc. - Intuitive: A clean interface - Cross platform: The project is written in Rust and…
2023 · clarkmccauley.com
- 24AB
All LLM user interfaces I've seen so far are somewhat frustrating: * ChatGPT web requires a lot of copy-paste, it rewrites whole document even if you need to update a part of it, etc. * Github Copilot completions are rather unreliable and do not leave an option to specify what you want; Copilot's chat sidebar is little more than ChatGPT integrated into the IDE * Google Docs have right UI for non-code text, but they use really dumb model (not Gemini 1.5 Pro). Also won't work for code. So... I wrote a Emacs Lisp function which calls LLM with contents of the buffer to generate text according to…
2024 · x.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →