Alternatives
Products that do what Millwright – Rust-based, self-hosted LLM router does
Hey HN, With the news of OpenRouter possibly being acquired and proliferation of hosted LLM routers (i.e. Ramp Router, Vercel’s AI Gateway), I saw the need for a self hosted solution focused on cost savings, transparency, and performance. So, I built an open sourced router with a simple CLI interface that can easily sit between coding agents and GenAI workloads. For the curious and lazy, at the moment, Millwright has the tools for, - Providers: OpenAI-compatible APIs, Anthropic, Amazon Bedrock - Routing: policy-controlled model roles (cheap, mid, frontier), cheapest healthy route selection -…
- 1FA
Jul 2026 · github.com
- 2
- 3
Hi HN, we built an open source model gateway. It's a single place to manage our own self hosted, frontier, and open source models in one place. It’s is rust native, built for concurrency, and implements all the config quirks across models and providers (streaming formats, tool calls, model parameters, rate limits, and different error behavior). The gateway adds under 1 ms for BYOK requests and under 2 ms when Experiential supplies the provider key. It has every major inference provider, and 1000+ models refreshed daily via a codex agent that opens a PR. Compared to other similar projects…
9d ago · github.com
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
Dev tool that worries about your LLM API failures.
Aug 2026 · llmrouter.dpdns.org
- 12
Skip the setup and run OpenClaw & Hermes, fully managed
17d ago · cloudways.com
- 13AU
Agentpanel is an observability platform for optimizing the control flow, performance, token usage, and correctness of LLM/AI agents! Built-in @rustlang, the first release of Agent Panel currently features an AI gateway that provides seamless access to 100+ LLMs across 20+ platforms, including OpenAI GPT-4o, Gemini 1.5 Pro latest, AnthropicAI Claude 3.5, MistralAI, Cohere, Groq,Perplexity AI, and more.
2024 · github.com
- 14IB
Hey HN, I'm Daniel, solo dev from Germany. I built ClawHosters (https://clawhosters.com), a managed hosting platform for OpenClaw, the open-source AI agent framework. Quick timeline: domain registered February 5th. First paying customer six days later. I probably should have spent more time on it, but it works. If you haven't seen OpenClaw, it lets you run a personal AI assistant that connects to Telegram, Discord, Slack, and WhatsApp. Self-hosting it is absolutely possible, but it's a pain. You're dealing with Docker setup, SSL certs, port forwarding, security hardening, keeping…
Feb 2026 · clawhosters.com
- 15LT
I wanted to share a project I've been working on for the past few weeks: llgtrt. It's a Rust implementation of a HTTP REST server for hosting Large Language Models using llguidance library for constrained output with NVIDIA TensorRT-LLM. The server is compatible with the OpenAI REST API and supports structured JSON schema enforcement as well as full context-free grammars (via Guidance). It's similar in spirit to the Python-based TensorRT-LLM OpenAI server example but written entirely in Rust and built with constraints in mind. No Triton Inference Server involved. This also serves as a demo…
2024 · github.com
- 16OS
I built a network discovery tool in Rust that helps identify public Ollama LLM servers. It scans IP ranges to find Ollama instances and catalogs their available models. Important note: This is intended for educational purposes and authorized security testing only.
2025
- 17
- 18BO
Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…
2025 · bhumi.trilok.ai
- 19
- 20IB
Link: https://docs.trysoma.ai/ For the past ~9 months I’ve been building Soma, an open-source AI agent & workflow runtime written in Rust, with a TypeScript SDK (Python coming soon). It’s not a framework; it’s meant to sit underneath whatever agent/tooling code you already write (Vercel AI SDK, LangChain, custom code, etc.). It provides features around your framework + a better DX for building agents. I’ve tried to take a Next.JS model: open-source, good DX, self-deployable. I originally set out to build a vertical back-office/operations product for SMEs. I needed a…
Dec 2025 · docs.trysoma.ai
- 21
Open-source AI agent runtime — build Agents in plain English
Jul 2026 · syntheticbrew.ai
- 22SA
Hi HN, We’re building https://www.switchpoint.dev – a drop-in replacement for OpenAI’s API that reduces LLM cost by smartly routing across models (e.g., Claude, Gemini, GPT-4) depending on subject and difficulty of the task. Why we built this: LLM costs are spiraling—especially for products doing retrieval, agentic reasoning, or even just high-volume chat. We were frustrated with paying GPT-4 rates when most queries didn’t need it. So we built a router that: - Starts with cheaper/free models (like Llama 8B, 4o-mini, 2.0 flash) - Streams responses and upgrades on failure - Acts…
2025 · switchpoint.dev
- 23LF
Dec 2025 · github.com
- 24OS
Hi HN, Matvey, Ildar, Joey, and Dominik here. If you're building LLM agents that use tools, you're probably worried about prompt injection attacks that can hijack those tools. We were too, and found that solutions like prompt-based filtering or secondary "guard" LLMs can be unreliable. Our thesis is that agent security should be handled at the network level between the agent and the LLM, just like a traditional web application firewall. So we built Archestra Platform: an open-source gateway that acts as a secure proxy for your AI agents. It's designed to be a deterministic firewall against…
Oct 2025 · archestra.ai
Ranked by how close each launch is in meaning, then by votes. Refine with a description →