AI · alternatives · 2026
24 alternatives to LLMs suck at writing integration code… for now
Hi HN! Stefan here from superglue and today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs. We gave LLMs API documentation and asked them to write code that makes actual API calls. Things like "create a Stripe customer" or "send a Slack message". We're not testing if they can use SDKs; we're testing if they can write raw HTTP requests (with proper auth, headers, body formatting) that actually work when executed against real API endpoints and can extract relevant information from that response. tl:dr: LLMs suck at…
Superglue has open-sourced an Agent-API Benchmark that tests how well large language models can write integration code to call real APIs. The benchmark evaluates whether LLMs can generate correct HTTP requests with… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Prices are shown for the 1 we have checked on their own sites, 1 of them free or with a free tier.
- 1
I spent a few hours last weekend testing whether AI can replace code by executing directly. Built a contact manager where every HTTP request goes to an LLM with three tools: database (SQLite), webResponse (HTML/JSON/JS), and updateMemory (feedback). No routes, no controllers, no business logic. The AI designs schemas on first request, generates UIs from paths alone, and evolves based on natural language feedback. It works—forms submit, data persists, APIs return JSON—but it's catastrophically slow (30-60s per request), absurdly expensive ($0.05/request), and has zero UI…
Nov 2025 · github.com · its alternatives →
- 2AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com · its alternatives →
- 3
Code Mode▲144I’ve been testing a simple idea inspired by Apple/Cloudflare/Anthropic: LLMs are far better at writing a small program than coordinating multiple tool calls. So instead of giving the model 10+ tools, I exposed one: a TypeScript sandbox with access to the same interfaces. The model writes a script → it runs once → done. What changed - +68% reduction in token use - No multi-step drift or retries - Local models (Llama 3.1 8B / Phi-3) became much more reliable Repo: https://github.com/universal-tool-calling-protocol/code-mode Curious whether others have seen…
Nov 2025 · github.com · its alternatives →
- 4
APIEval-20▲121An open benchmark for AI agents that test APIs
May 2026 · resources.kusho.ai · its alternatives →
- 5
Hey HN, we’re building an open specification that lets agents discover and invoke APIs with natural language, built on the OpenAPI standard. agents.json clearly defines the contract between LLMs and API as a standard that's open, observable, and replicable. Here’s a walkthrough of how it works: https://youtu.be/kby2Wdt2Dtk?si=59xGCDy48Zzwr7ND. There’s 2 parts to this: 1. An agents.json file describes how to link API calls together into outcome-based tools for LLMs. This file sits alongside an OpenAPI file. 2. The agents.json SDK loads agents.json files as tools for an LLM that…
2025 · github.com · its alternatives →
- 6
- 7
LLM Gatewayfree tier▲296I built an OpenAI-compatible LLM gateway that routes requests to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang... anything that speaks /v1/chat/completions. Single Go binary, one YAML config file, no infrastructure. It does the things you'd expect from this kind of gateway... semantic routing via a three-layer cascade (keyword heuristics, embedding similarity, LLM classifier) that picks the best model when clients omit the model field, weighted round-robin load balancing across local inference servers with health checks and failover. The part I think is most…
2025 · llmgateway.io · its alternatives →
- 8
Hi HN, we’re Stefan and Adina, and we’re building superglue (https://superglue.cloud). superglue allows you to connect to any API/data source and get the data you want in the format you need. It’s an open-source proxy server which sits between you and your target APIs. Thus, you can easily deploy it into your own infra. If you’re spending a lot of time writing code connecting to weird APIs, fumbling with custom fields in foreign language ERPs, mapping JSONs, extracting data from compressed CSVs sitting on FTP servers, and making sure your integrations don’t break when…
2025 · github.com · its alternatives →
- 9
Hi HN community, I have been working on benchmarking publicly available LLMs these past couple of weeks. More precisely, I am interested on the finetuning piece since a lot of businesses are starting to entertain the idea of self-hosting LLMs trained on their proprietary data rather than relying on third party APIs. To this point, I am tracking the following 4 pillars of evaluation that businesses are typically look into: - Performance - Time to train an LLM - Cost to train an LLM - Inference (throughput / latency / cost per token) For each LLM, my aim is to benchmark them for…
2023 · github.com · its alternatives →
- 10
Recently featured in a LangChain blog https://blog.langchain.dev/empowering-development-with-flowt... , use LLMs to construct an API first runnable workflow with an IDE experience.
2024 · github.com · its alternatives →
- 11AC
We run superglue, an OSS agentic integration platform. Last week I talked to a founder of another YC startup. She found a use case for our CLI that we hadn't officially launched yet. Her problem: customers wanted to create Opps in Salesforce from inside the chat in her app. We kept seeing this pattern: teams build agents and their users can perfectly describe what they want: "pull these three objects from Salesforce and push to nCino when X condition is true", but translating that into a generalized hard-coded tool the agent can call is a lot of work and does not scale since the logic is…
Apr 2026 · docs.superglue.cloud · its alternatives →
- 12
Check out this impressive project that enables running LLMs entirely in the browser using WebGPU. Key features: - Zero token costs, no cloud infrastructure required - Complete data privacy through local processing - Simple 3-line code integration - Built on MLC and Transformer.js The benchmarks show smaller models can effectively handle many common tasks. Currently the project roadmap includes: - No-code AI pipeline builder - Browser-based RAG for document chat - Analytics/logging - Model fine-tuning interface
2025 · github.com · its alternatives →
- 13
Single-agent LLMs suck at long-running complex tasks. We’ve open-sourced a multi-agent orchestrator that we’ve been using to handle long-running LLM tasks. We found that single LLM agents tend to stall, loop, or generate non-compiling code, so we built a harness for agents to coordinate over shared context while work is in progress. How it works: 1. Orchestrator agent that manages task decomposition 2. Sub-agents for parallel work 3. Subscriptions to task state and progress 4. Real-time sharing of intermediate discoveries between agents We tested this on a Putnam-level math problem, but the…
Feb 2026 · github.com · its alternatives →
- 14
Hi Hacker News! We’re Vadim and Chris from Highlight.io [1]. We do web app monitoring and are working on using LLMs/embeddings to add new functionality to our error monitoring product. Given that there’s a lot of founders/engineers using LLMs in their products, we figured we’d share how we built the new functionality, their impact on our workflows, and how you can try it out. Our goal was to build two features: (1) tagging errors (e.g. deeming an error as “authentication error” or a “database error”); and (2) grouping similar errors together (e.g. two errors that have a different…
2023 · github.com · its alternatives →
- 15
So, it feels like this should exist. But I couldn't find it. So I tried to build it. Agentflow lets you run complex LLM workflows from a simple JSON file. This can be as little as a list of tasks. Tasks can include variables, so you can reuse workflows for different outputs by providing different variable values. They can also include custom functions, so you can go beyond text generation to do anything you want to write a function for. Someone might say: "Why not just use ChatGPT?" Among other reasons, I'd say that you can't template a workflow with ChatGPT, trigger it with different…
2023 · github.com · its alternatives →
- 16
Henry, Matt and James here – we’re building an open source toolkit that makes it easy to integrate an LLM-powered copilot that talks to your API into software products. It works by calling API endpoints which you choose to expose to it. This lets the chatbot complete tasks within your software in response to natural language queries. It’s also open source, so you don’t have to send user data to another 3rd party. We support Llama 2, but we haven’t fine-tuned Llama 2 yet (coming soon) so highest accuracy is seen with GPT-4 or fine-tuned GPT-3.5 (much faster). We started working together 2…
2023 · github.com · its alternatives →
- 17
Hi! My name is Herve Kom, a computer science student that is interested in learning new things everyday! As one of my graduation project, I have developed a Claude Code -like Coding CLI, but with enhancement for API Testing: - Auto-generate & run tests (unit, e2e, Playwright, CI/CD, etc.) - Say bye-bye to hallucinations with built-in MCP Server to let LLM directly read from API Docs - Adding Agent.md support for better context persistence across your whole codebase - Automatic bug & security scans (logic is kind of basic but works great!) - Vibes, I want it to feel less "enterprise" but…
2025 · github.com · its alternatives →
- 18
The page you were looking for doesn
2025 · superglue.ai · its alternatives →
- 19
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
- 20SO
Last month, we open-sourced Supaglue (https://github.com/supaglue-labs/supaglue) as a developer toolkit for building customer-facing Salesforce integrations. Since then, we've been iterating on our approach with users and are re-launching Supaglue as an open source unified API, starting with CRMs. With our re-launched public alpha you can: - Interact with HubSpot and Salesforce through a unified REST API - Sync data from HubSpot and Salesforce into a local Postgres cache and make reads against a normalized CRM schema - Make writes (POST and PATCH) to HubSpot and…
2023 · its alternatives →
- 21
Hey, folks here is a peek into Jujutsu. We at Poozle are working with hundreds of APIs and it has been always frustrating to 1. Search the API in the documentation or ask ChatGPT 2. Then copy it to the postman and understand/test the API 3. Generate code to integrate into the codebase We thought how about having all of this at one place. We currently fine-tuned LLM on public REST APIs to reduce hallucination and then combined it with ChatGPT and Postman. I look forward to feedback, feature requests and discussions!
2023 · loom.com · its alternatives →
- 22
Hi HN, I've been working with LLMs in production for a while both as a solo dev building apps for clients and working at an AI startup. The one thing that always was a pain was to pay OpenAI/Gemini/Anthropic a few dollars a month just for me to say "test" or have a CI runner validate some UI code. So I built this server called ChunkBack, that mocks the popular llm provider's functionality but allows you to type in a deterministic language: `SAY "cheese"` or `TOOLCALL "tool_name" {} "tool response"` I've had to work in some test environments and give good results for experimenting…
Nov 2025 · github.com · its alternatives →
- 23OS
Docs.codes generates simple markdowns for open-source libraries that you can add to the context of your LLM assistants, helping them generate better code. Here's a quick walkthrough with pypi/mem0ai as example: https://youtu.be/SKZol8G_tIE LLMs struggle with generating correct code when using lesser-known libraries or dealing with major version changes that happen after their training cutoff. With these markdowns, you can ensure that your LLM chat/coding assistants have up-to-date knowledge of the library's API and usage patterns. We built this using the latest…
2024 · docs.codes · its alternatives →
- 24
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →