Llmbuffer – Python library for cache-optimized LLM conversation history
I was not getting good cache utilization when including dynamic context in agent threads. After a lot of experimentation, I found a good pattern that minimizes how often long lived conversation history gets modified while still supporting dynamic context. It has flexible hooks for doing things like truncating or summarizing tool outputs when transitioning messages to the long term history. And I'm seeing >>90% of tokens hitting the cache for my agents despite including a lot of dynamic user context. There are a wide range of agent prompting strategies so I'd love to hear where this library…
Llmbuffer – Python library for cache-optimized LLM conversation history launched on June 10, 2026 with 7 votes, #1,808 of 22,647 launches that month and more than 75% of that year's launches.
What it does
LLM conversation buffer with cache optimization and dynamic context. - scottpurdy/llmbuffer
Most LLM applications naively concatenate their system prompt, conversation history, and any dynamic context into a single message list — and rebuild it from scratch every turn. This works, but it leaves significant money and latency on the table by constantly invalidating the provider's prompt cache. llmbuffer assembles your messages in the order that maximises cache reuse, manages the boundary between stable and changing content, and handles compaction when history grows too long — all without you having to think about it. [Static System Prompt] → [Long-Lived History] → [Dynamic Context] → [Recent Messages] cached ✓ cached ✓ not cached not cached The static system prompt and committed…from github.com
In the maker’s words, at launch
I was not getting good cache utilization when including dynamic context in agent threads. After a lot of experimentation, I found a good pattern that minimizes how often long lived conversation history gets modified while still supporting dynamic context. It has flexible hooks for doing things like truncating or…
Does a similar job
all alternatives →
Context GatewayMar 2026 · github.com · ▲220We built an open-source proxy that sits between coding agents (Claude Code, OpenClaw, etc.) and the LLM, compressing tool outputs before they enter the context window. Demo: https://www.youtube.com/watch?v=-vFZ6MPrwjw#t=9s. Motivation: Agents are terrible at managing context. A single file read or grep can dump thousands of tokens into the window, most of it noise. This isn't just expensive — it actively degrades quality. Long-context benchmarks consistently show steep accuracy drops as context grows (OpenAI's GPT-5.4 eval goes from 97.2% at 32k to 36.6% at 1M…
An LLM response cache that's aware of dynamic dataJan 2026 · blog.butter.dev · ▲17Raymond here from Butter.dev, an LLM response cache built as a chat-completions proxy. Today we're launching a key feature for the platform: the ability to generalize on dynamic, templated inputs. Caching at the HTTP request level has the obvious problem of generalizability. Nearly no request is identical, due to templated variables (like names) and metadata (like timestamps), so exact-match cache lookups rarely hit. We solve this at Butter by using LLMs to detect dynamic content in requests and derive their inter-relationships, allowing the cache entry to be stored as a template + variables…
- RCReduce ChatGPT costs 10x with distributed cache for LLMs2024 · edgematic.dev · ▲22
Hello HN! We're building a caching solution for LLMs (ChatGPT, Claude). By combining cutting-edge approaches, such as edge computing, prompt compression, vectorization, and others - it can reduce your AI bills by up to 10x and significantly lower response times. Key Features: - cost efficiency: our system stores frequent queries, reducing the number of upstream (paid) API calls - fast responses: with various nodes globally, we reduce latency by serving data from the nearest location - scalability: designed to handle increasing loads and data sizes without degrading performance. The cache…
Bindly2016 · bind.ly · ▲76Hi HN, I’m a product designer by background, not a traditional software engineer. Over the last year, tools like Claude and ChatGPT completely changed how I work. I started with small internal tools, and now I’m “vibe coding” multiple highly personalized apps. As I iterated between Claude Code (implementation) and ChatGPT (ideation / thinking), I kept running into the same problem. To think clearly, I had to repeatedly re-explain: - what the code currently does, - what changed recently, - and why certain decisions were made. That re-summarization step became a real bottleneck. So I…
Butter – A Behavior Cache for LLMsOct 2025 · butter.dev · ▲50Hi HN! I'm Erik. We built Butter, an LLM proxy that makes agent systems deterministic by caching and replaying responses, so automations behave consistently across runs. - It’s a chat completions compatible endpoint, making it easy to drop into existing agents with a custom base_url - The cache is template-aware, meaning lookups can treat dynamic content (names, addresses, etc.) as variables You can see it in action in this demo where it memorizes tic-tac-toe games: https://www.youtube.com/watch?v=PWbyeZwPjuY Why we built this: before Butter, we were Pig.dev (YC W25), where we…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 28d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, June 2026
the whole month →
Fundraisly▲1,537AI fundraising agent that finds investors and books meetings
AI · Jun 2026 · fundraisly.com
Homebrew 6.0.0▲1,481Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!
Dev tools · Jun 2026 · brew.sh
Hi HN! I was super frustrated at apps like tiktok and wechat preventing me from saving or downloading the animated gifs, so I made a little tool to turn a screen recording into a slack-sized emoji. Hope you get some value from this!
Life & fun · Jun 2026 · vorpus.github.io
Search 18 years of Hacker News and chart how any topic, tool, or person trended. Overlay terms to compare their rise and fall across 45M posts and comments.
Life & fun · Jun 2026 · hackernewstrends.com