AI · alternatives · 2026

24 alternatives to Composable middleware for LLM inference Optimization Passes

Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Prices are shown for the 1 we have checked on their own sites, 1 of them free or with a free tier.

1
We made our own inference engine for Apple Silicon▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com · its alternatives →
2AB
AutoThink – Boosts local LLM performance with adaptive reasoning▲397
I built AutoThink, a technique that makes local LLMs reason more efficiently by adaptively allocating computational resources based on query complexity. The core idea: instead of giving every query the same "thinking time," classify queries as HIGH or LOW complexity and allocate thinking tokens accordingly. Complex reasoning gets 70-90% of tokens, simple queries get 20-40%. I also implemented steering vectors derived from Pivotal Token Search (originally from Microsoft's Phi-4 paper) that guide the model's reasoning patterns during generation. These vectors encourage behaviors like numerical…
2025 · its alternatives →
3
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
4
Tiny-vLLM – high performance LLM inference engine in C++ and CUDA▲205
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
May 2026 · github.com · its alternatives →
5
An educational Local Qwen3 LLM Inference project written in Rust▲15
An educational Rust project for exporting and running inference on Qwen3 LLM family - reinterpretcat/qwen3-rs
2025 · github.com · its alternatives →
6
Trails▲81
Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Jan 2026 · trails-red.vercel.app · its alternatives →
7
LLM-Term – Simple Rust-based CLI assist tool▲32
Current AI-assisted CLI tools are often part of larger systems and work better on Linux. I built llm-term to address these. It's a Rust-based tool that compiles into a single binary file. You only need to download the binary, add it to your PATH, and configure your OpenAI key to get started. While llm-term offers an option for gpt-4o, it works great with gpt-4o-mini. So it's not costly. I appreciate any feedback or suggestions.
2024 · github.com · its alternatives →
8
N0x – LLM inference, agents, RAG, Python exec in browser, no back end▲9
Built this because I was tired of every AI tool shipping my data to someone else server n0x runs the full stack LLM inference via WebGPU, autonomous ReAct agents, RAG over your own docs, sandboxed Python execution via Pyodide all inside a single browser tab. No account No keys No backend Models download once, cache in IndexedDB permanently. Biggest challenge was context window budgeting for the agent loop and making the WASM vector search non-blocking. Happy to talk architecture. GitHub: https://github.com/ixchio/n0x | Live demo: https://n0x-three.vercel.app
Mar 2026 · n0xth.vercel.app · its alternatives →
9
50+ LLMs on 2 GPUs with 2-Second Swapping? We built AI-Native Runtime▲5
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com · its alternatives →
10
Tired of building agents? throw an LLM at this framework▲14
Enable LLMs to Program Themselves. Contribute to Osly-AI/Pocket-Flow-Framework development by creating an account on GitHub.
2025 · github.com · its alternatives →
11
Full Stack Starter Repo Optimized▲6
Hi everyone! I've been loving building with AI, and over the past few years I've been leaning more and more into Typescript (and bun). My team at inference.net is constantly trying to get more leverage out of AI and find ways to setup our codebase to be able to increase the level of correctness that our AI is able to write code at. This starter repo is a very opinionated way to lay out a repo to lean into AI heavily. It leverages Cloudflare Workers as a deployment target for the API (my goal is to never have to deploy an API on a AWS/Azure/GCP server ever again unless I get to a…
2025 · abeahmed.com · its alternatives →
12
ChainFactory – Run Structured LLM Inference with Easy Parallelism▲8
hi everyone. how does moving llm call prompts and output structure definitions away from code into configuration land sound? would you use something like this if it was stable and well documented enough? please don't hold back the criticism. i appreciate all feedback (constructive & otherwise).
2024 · github.com · its alternatives →
13
Run AI models directly in the browser – no server or internet required▲11
Local In-Browser LLM inference for private, fast, and secure chat - no server required.
2025 · private-ai-chat.vercel.app · its alternatives →
14
Running AI agents across environments needs a proper solution▲8
Hi HN folks, I have been building AI agents for quite some time now. The shift has gone from LLM + Tools → LLM Workflows → Agent + Tools + Memory, and now we are finally seeing true agency emerge: agents as systems composed of tools, command-line access, fine-grained system capabilities, and memory. This way of building agents is powerful, and I believe it is here to stay. But the real question is: are the systems powering these agents ready for that future? I do not think so. Using Docker for a single agent is not going to scale well, because agents need to be lightweight and fast. LLMs…
Mar 2026 · github.com · its alternatives →
15
Costco for LLM Tokens▲6
Inference infrastructure for AI-native teams. Run, monitor, and optimize production AI workloads with lower cost, faster latency, and dedicated support from Inference.net.
2024 · inference.net · its alternatives →
16
LLM agents that write Python to analyze execution traces at scale▲5
We combined Stanford's ACE (agents learning from execution feedback) with the Reflective Language Model pattern. Instead of reading traces in a single pass, an LLM writes and runs Python in a sandbox to programmatically explore them - finding cross-trace patterns that single-pass analysis misses. The framework achieved 2x consistency improvement on τ2-bench.
Mar 2026 · github.com · its alternatives →
17
We instrumented Claude Agent SDK using a tiny Rust proxy▲6
At Laminar (https://github.com/lmnr-ai/lmnr) we're building open source AI observability platform in Rust. We obsess over instrumentation DX for our Python and TS SDKs and in this new blog we outline how we made the most seamless way of instrumenting recently released claude agent sdk
Dec 2025 · laminar.sh · its alternatives →
18
Bhumi–OSS Python Library w Rust Underhead for 2.5x Faster LLM Inference▲8
Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…
2025 · bhumi.trilok.ai · its alternatives →
19
BonzAI – self-sovereign, local LLM inference in the browser▲5
Hi everyone, I started building BonzAI two years ago, as an attempt to fully replace my online AI stack with an offline, sovereign alternative. Back in the days, I had the intuition that open weights model would stand the comparison with private AI labs, and that the commoditization of inference was the future. When I was finally able to switch from ChatGPT/Claude to local LLMs, midjourney to Flux/SDXL/Z-image, Eleven Labs/Suno to Qwen-TTS/Ace-Step, and Sora to LTX, I implemented: - The ability to create datasets and fine-tune LoRAs; - An experimental 3D objects…
May 2026 · bonzai.sh · its alternatives →
20
Run an Agent Council of LLMs that debate and synthesize answersfree▲8
I built a local-first UI that adds two reasoning architectures on top of small models like Qwen, Llama and Mistral: a sequential Thinking Pipeline (Plan → Execute → Critique) and a parallel Agent Council where multiple expert models debate in parallel and a Judge synthesizes the best answer. No API keys, zero .env setup — just pip install multimind. Benchmark on GSM8K shows measurable accuracy gains vs. single-model inference.
Mar 2026 · github.com · its alternatives →
21
Serve 100 Large AI models on a single GPU with low impact to TTFT▲7
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com · its alternatives →
22IS
I spent 8 months trying to make LLMs Hack▲6
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025 · its alternatives →
23
I made AI agent lib that you will understand▲7
Every time I wanted to use LLMs in my existing pipelines the integration was very bloated, complex, and too slow. This is why I created a lightweight library that works just like scikit-learn, the flow generally follows a pipeline-like structure where you “fit” (learn) a skill from sample data or an instruction set, then “predict” (apply the skill) to new data, returning structured results. High-Level Concept Flow Your Data --> Load Skill / Learn Skill --> Create Tasks --> Run Tasks --> Structured Results --> Downstream Steps And the bast part: Every step can be saved and reused as…
2025 · github.com · its alternatives →
24
I applied Lyapunov stability theory to detect when LLM agents spiral▲11
An empirical validation of physics-inspired runtime monitoring for multi-turn LLM agents across 3,175 total runs spanning four benchmarks (τ³-bench, SWE-bench, MINT, custom local-model battery). A 5-condition ablation study with multi-trial validation (333 SWE-bench runs) demonstrates that Lyapunov
Jun 2026 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →