Alternatives
Products that do what key-carousel - Key rotation for LLM agents does
I think in-process key management is the right abstraction for multi-key LLM setups. Not LiteLLM, not a Redis queue, not a custom load balancer. The failure modes are well-understood: a key gets rate-limited, you wait, you try the next one. Billing errors need a longer cooldown than rate limits. This is not a distributed systems problem — it's a state machine that fits in a library. The problem is everyone keeps solving it with infrastructure instead. Spin up LiteLLM, now you have a Python service to maintain. Reach for Redis, now you have a database for a problem that doesn't need one.…
- 1
- 2EC
Hi! I've found myself repeatedly writing little scripts to do bulk calls to LLMs for various tasks. For example, run some analysis on a large list of records. There are a few "gotchas" to doing this. For example, some service providers have rate limits, and some models will not reliably return JSON (if you're asking for it). So, I've written a command for this. What I've tried to do here is let the user break up prompts and configuration as they see fit. For example, you can have a prompt file which includes the API key, rate limit, settings, etc. all together, or break these up into…
2025 · github.com
- 3IB
Hi HN, I’m the creator of Cordum. I’ve been working in DevOps and infrastructure for years (currently in the fintech/security space), and as I started playing with AI agents, I noticed a scary pattern. Most "safety" mechanisms rely on system prompts ("Please don't do X") or flimsy Python logic inside the agent itself. If we treat agents as autonomous employees, giving them root access and hoping they listen to instructions felt insane to me. I wanted a way to enforce hard constraints that the LLM cannot override, no matter how "jailbroken" it gets. So I built Cordum. It’s an open-source…
Jan 2026 · github.com
- 4LO
Key management for multiple users and multiple cloud LLM/GenAI APIs is difficult to be both safe and convenient. Sharing keys among users risks leaking the key and makes it difficult to curb the leakage without interruptions. But assigning one key per user per cloud API results in too many keys to keep track of. Meet LlaMa(ster)Key, the secure and easy solution for API key management: * For each user, one master key for multiple APIs. * The master key is unique to each user. Granting and revoking a user's access won't impact other users. * The actual API keys to authenticate with cloud…
2024 · github.com
- 5ML
Time to first token is 39% faster Agent wall times decrease by 46% No swaps Tracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device. Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more. Built a ton of features
Jun 2026 · autotunellm.com
- 6IS
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025
- 7GB
Hey HN, We’re excited to share PySpur, an open-source tool that provides a graph-based interface for building, debugging, and evaluating LLM workflows. Why we built this: Before this, we built several LLM-powered applications that collectively served thousands of users. The biggest challenge we faced was ensuring reliability: making sure the workflows were robust enough to handle edge cases and deliver consistent results. In practice, achieving this reliability meant repeatedly: 1. Breaking down complex goals into simpler steps: Composing prompts, tool calls, parsing steps, and branching…
2024 · github.com
- 8KP
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
Jun 2026 · github.com
- 9IL
LLM Application development is extremely iterative, more so than any other types of development. This is because in addition to all the activities involved in regular application development, we also need to make the LLM Application accurate and reduce hallucination. To improve performance, we need to trial and error various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, try different agent architecture, and more. There are thousands of permutations to try. We need to be able to easily experiment with these…
2024 · palico.ai
- 10OS
Hi HN, Matvey, Ildar, Joey, and Dominik here. If you're building LLM agents that use tools, you're probably worried about prompt injection attacks that can hijack those tools. We were too, and found that solutions like prompt-based filtering or secondary "guard" LLMs can be unreliable. Our thesis is that agent security should be handled at the network level between the agent and the LLM, just like a traditional web application firewall. So we built Archestra Platform: an open-source gateway that acts as a secure proxy for your AI agents. It's designed to be a deterministic firewall against…
Oct 2025 · archestra.ai
- 11CR
hi everyone. how does moving llm call prompts and output structure definitions away from code into configuration land sound? would you use something like this if it was stable and well documented enough? please don't hold back the criticism. i appreciate all feedback (constructive & otherwise).
2024 · github.com
- 12OS
Hello HN, I’ve been building AI agents lately and ran into a common "Context Bloat" problem. When an agent has 20+ skills, stuffing every system prompt, reference doc, and tool definition into a single request quickly hits token limits and degrades model performance (the "lost in the middle" problem). To solve this, I built OpenSkills, an open-source SDK that implements a Progressive Disclosure Architecture for agent skills. The Core Concept: Instead of loading everything upfront, OpenSkills splits a skill into three layers: Layer 1 (Metadata): Light-weight tags and triggers (always loaded…
Jan 2026
- 13EA
A few months ago I was working on a flight search engine that would include pet transport costs (I know a few by hearth but storing them and make the calculations in the UI would be nice) While I was collecting pet pricing from several airlines I strugled to extract data in a common format without hallucinated values. That's when I thought: What if I use multiple LLMs and take the most common response to improve accuracy? This idea became this new project. You provide your documents, an SQLModel schema, an LLM provider, plus what you'd like to extract and Extrai does the rest. Including…
Nov 2025 · github.com
- 14CM
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com
- 15WB
Over the past few months, as we scaled our internal AI Agents, we hit a dead end: Running LLM-generated arbitrary code in Docker is basically running naked on security due to container escape risks. But using full traditional VMs takes minutes to boot and eats too much memory to support high-density concurrency. We loved the developer experience of SaaS sandboxes on the market, but they are closed-source, expensive, and have too high a barrier to entry for self-hosting. So, our team decided to build our own. After months of grinding, using RustVMM and KVM, we built a blazing-fast,…
Apr 2026 · github.com
- 16RA
Hi HN folks, I have been building AI agents for quite some time now. The shift has gone from LLM + Tools → LLM Workflows → Agent + Tools + Memory, and now we are finally seeing true agency emerge: agents as systems composed of tools, command-line access, fine-grained system capabilities, and memory. This way of building agents is powerful, and I believe it is here to stay. But the real question is: are the systems powering these agents ready for that future? I do not think so. Using Docker for a single agent is not going to scale well, because agents need to be lightweight and fast. LLMs…
Mar 2026 · github.com
- 17IM
Every time I wanted to use LLMs in my existing pipelines the integration was very bloated, complex, and too slow. This is why I created a lightweight library that works just like scikit-learn, the flow generally follows a pipeline-like structure where you “fit” (learn) a skill from sample data or an instruction set, then “predict” (apply the skill) to new data, returning structured results. High-Level Concept Flow Your Data --> Load Skill / Learn Skill --> Create Tasks --> Run Tasks --> Structured Results --> Downstream Steps And the bast part: Every step can be saved and reused as…
2025 · github.com
- 18LF
I've been building agentic apps for some large Fortune 500 companies (T-Mobile, Twilio, etc.) and developed a mental model that serves as a practical guide in building agentic apps: separate the high-level agent specific logic from low-level platform capabilities. I call it the L-MM: the Logical Mental Model for LLM applications. This mental model has not only been tremendously helpful in building agents but also helping customers think about the development process - so when I am done with a consulting engagement they can move faster across the stack and enable engineers and platform teams…
2025
- 19HP
Hi HN. I heard you like dev tools and AI, so we wanted to share our project that we’ve been working on. We’re working on Horizon [1] - a higher level abstraction for LLMs so that developers can spend less time trying to grapple with LLMs to make them work and more time with users. This is the starting feature set which takes an auto-ML approach to identify the optimal LLM model, hyperparameters, and prompt - instead of just giving you the tooling to figure it out yourself. You can read more about it in our documentations. Our view is that as LLMs become increasingly commoditized and prompts…
2023 · gethorizon.ai
- 205L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 21LF
Hey HN, I built SWE-Kit, LLM toolkit (Function callable tools) which makes building agents specialised in coding like Devin very easy. I noticed a typical pattern while building local agents: creating & perfecting LLM tools to interact with system or codebase was the repeated and time-consuming. We created a layer that simplifies building agents that can interact with code, file system, git, shell and allows you to quickly solve for a wide variety of coding agent use cases. Aren’t there open coding agents already? Well, yes, but most folks would want to solve their specific use case like a…
2024 · swekit.dev
- 22TO
I built DevClaw, an OpenClaw plugin that turns each Telegram group into an isolated, autonomous dev team: planner/orchestrator, DEVs, and QA all running on their own. I use it for all my development now. Issues on GitLab/GitHub are the single source of truth, and three things compound to save around 70% on tokens: model tiering (Haiku for typos, Opus for architecture), session reuse across tasks, and token-free scheduling that burns zero LLM calls for orchestration. Please try it and give some feedback. Also keen to hear from anyone running autonomous coding agents, especially what…
Feb 2026 · github.com
- 23
Ranked by how close each launch is in meaning, then by votes. Refine with a description →