ArchGW – An open-source intelligent proxy server for prompts
Hi HN! This is Salman, Adil, Shuguang and Co working on ArchGW[1] - an open-source lightweight proxy server for prompts - written in Rust and built on top of Envoy[2]. Arch moves the critical but pesky handling and processing of prompts: task understanding, prompt routing, safety, and observability - outside business logic. Its an edge and egress proxy for agentic apps. We've talked to 100s of developers at places like Twilio, GE Healthcare, Redhat, Square, etc and there was a consistent theme in building AI apps: to move past a nascent demo they are left to their own devices in building out…
In plain words
ArchGW is an open-source proxy server written in Rust that handles prompt processing for AI applications. Built on Envoy, it manages task understanding, prompt routing, safety checks, and observability as a separate layer from application code. Designed for developers building agent-based AI systems, it aims to simplify infrastructure setup by centralizing prompt handling middleware so teams can move faster and deploy with greater confidence in production.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN! This is Salman, Adil, Shuguang and Co working on ArchGW[1] - an open-source lightweight proxy server for prompts - written in Rust and built on top of Envoy[2]. Arch moves the critical but pesky handling and processing of prompts: task understanding, prompt routing, safety, and observability - outside business logic. Its an edge and egress proxy for agentic apps. We've talked to 100s of developers at places like Twilio, GE Healthcare, Redhat, Square, etc and there was a consistent theme in building AI apps: to move past a nascent demo they are left to their own devices in building out middle ware capabilities so that developers can move faster and ship with confidence. Today, the approach to building an enterprise-ready AI app is cobbling together a large set of mono-functional tools, adding LLM-based preprocessing steps to determine safety (e.g. applying governance and guardrails), ask clarifying questions to improve task performance, support common agentic operations by packaging and managing function calling scenarios manually, etc. Not to mention, all the undifferentiated work in incorporating different LLM models and versions, and managing resiliency, retries and fallback logic. ArchGW was built with the belief that prompts are nuanced and opaque user requests, which require the same capabilities as traditional HTTP requests including secure handling, intelligent routing, robust observability, and integration with backend (API) systems for personalization – outside business logic. We help built Envoy while at Lyft and think its offers a great foundation to build a proxy to manage traffic for prompts. Here are some additional details about the open source project. ArchGW is written in rust, and the request path has three main parts: * Listener subsystem which handles downstream (ingress) and upstream (egress) request processing. * Prompt handler subsystem. This is where ArchGW makes decisions on the safety of the incoming request via its prompt_guard primitive and identifies where to forward the conversation to via its prompt_target primitive. * Model serving subsystem is the interface that hosts all the lightweight LLMs[3] engineered in ArchGW and offers a framework for things like hallucination detection of our these models We loved building this open source project, and our belief is that this infrastructure primitive would help developers build faster, safer and more personalized agents without all the manual prompt engineering and systems integration work needed to get there. We hope to invite other developers to use and improve Arch. Please give it a shot and leave feedback here, or at our discord channel [4] Also here is a quick demo of the project in action [5]. You can check out our public docs here at [6]. Our models are also available here [7]. [1] https://github.com/katanemo/archgw [2] https://www.envoyproxy.io/ [3] https://huggingface.co/collections/katanemo/arch-function-66... [4] https://discord.com/channels/1292630766827737088/12926307682... [5] https://www.youtube.com/watch?v=I4Lbhr-NNXk [6] https://docs.archgw.com/ [7] https://huggingface.co/katanemo
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, March 2025
the whole month →
Mimic Human Research & Save Findings in AI Knowledge Base
AI · 2025 · sider.ai
