Composable middleware for LLM inference Optimization Passes
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
What it does
In the maker’s words, at launch
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs have provider-side moderation. When you run Qwen or Llama locally, you get raw inference with zero safety net. If that model has tool access or touches a database, that's a real liability. The solution A Tower-style middleware stack for LLM inference. You wrap any provider with composable layers: ```rust let llm = PipelineBuilder::new(llama_cpp_provider) .add_layer(CacheLayer::new(CacheConfig { chat_key_mode: ChatCacheKeyMode::UserPromptOnly, ttl: Some(Duration::from_secs(900)), max_size: Some(512), ..Default::default() })) .add_layer( Guardrails::builder() .input_guard(RegexPiiRedactionGuard::default()) .input_guard(PromptInjectionGuard::default()) .enforcement_policy(EnforcementPolicy::Block) .build() .layer(), ) .build(); ``` That's it. The llm variable implements LLMProvider — you pass it to any agent and the layers are structurally enforced. Can't bypass them, can't forget them. The broader framework: AutoAgents is a full agent framework — memory, tool use, multi-agent orchestration, the works. The pipeline feature works with any provider: llama.cpp (local), Ollama, OpenAI, Anthropic, etc. Same .add_layer() API regardless of backend. Written in Rust. No GC pauses. Memory-safe. The framework has ~400 stars and is being used in production for edge AI deployments. A note on maturity: The guardrails and pipeline layers are still early — the guard implementations are basic, observability isn't there yet, and we're iterating on the API surface. But the underlying architecture is solid and stable. The middleware pattern, the trait-based guard system, and the provider-agnostic pipeline contract aren't going to change. We're building on a foundation we're confident in, and shipping the layers incrementally. Early feedback shapes what gets built next. I'd genuinely like feedback on: The layer ordering question — should we enforce a recommended order or keep it flexible? What guardrail implementations would you actually use in production? Is the Tower-middleware mental model the right framing, or is there a better analogy? Full example with local Qwen3-VL-8B: https://github.com/liquidos-ai/AutoAgents/tree/main/examples... Thanks
Does the same job
all alternatives →
- NLN0x – LLM inference, agents, RAG, Python exec in browser, no back endMar 2026 · n0xth.vercel.app · ▲9
Built this because I was tired of every AI tool shipping my data to someone else server n0x runs the full stack LLM inference via WebGPU, autonomous ReAct agents, RAG over your own docs, sandboxed Python execution via Pyodide all inside a single browser tab. No account No keys No backend Models download once, cache in IndexedDB permanently. Biggest challenge was context window budgeting for the agent loop and making the WASM vector search non-blocking. Happy to talk architecture. GitHub: https://github.com/ixchio/n0x | Live demo: https://n0x-three.vercel.app
- 5L50+ LLMs on 2 GPUs with 2-Second Swapping? We built AI-Native Runtime2025 · github.com · ▲5
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
- FSFull Stack Starter Repo Optimized2025 · abeahmed.com · ▲6
Hi everyone! I've been loving building with AI, and over the past few years I've been leaning more and more into Typescript (and bun). My team at inference.net is constantly trying to get more leverage out of AI and find ways to setup our codebase to be able to increase the level of correctness that our AI is able to write code at. This starter repo is a very opinionated way to lay out a repo to lean into AI heavily. It leverages Cloudflare Workers as a deployment target for the API (my goal is to never have to deploy an API on a AWS/Azure/GCP server ever again unless I get to a…
- CRChainFactory – Run Structured LLM Inference with Easy Parallelism2024 · github.com · ▲8
hi everyone. how does moving llm call prompts and output structure definitions away from code into configuration land sound? would you use something like this if it was stable and well documented enough? please don't hold back the criticism. i appreciate all feedback (constructive & otherwise).
- RARunning AI agents across environments needs a proper solutionMar 2026 · github.com · ▲8
Hi HN folks, I have been building AI agents for quite some time now. The shift has gone from LLM + Tools → LLM Workflows → Agent + Tools + Memory, and now we are finally seeing true agency emerge: agents as systems composed of tools, command-line access, fine-grained system capabilities, and memory. This way of building agents is powerful, and I believe it is here to stay. But the real question is: are the systems powering these agents ready for that future? I do not think so. Using Docker for a single agent is not going to scale well, because agents need to be lightweight and fast. LLMs…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
