AI · alternatives · 2026
24 alternatives to A registry of agent benchmarks (including many OSS agent trajectories)
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. A registry of agent benchmarks (including many OSS agent trajectories) launched in 2024; newer entries below may have overtaken it.
- 1
Scored 65.2% vs google's official 47.8%, and the existing top closed source model Junie CLI's 64.3%. Since there are a lot of reports of deliberate cheating on TerminalBench 2.0 lately (https://debugml.github.io/cheating-agents/), I would like to also clarify a few things 1. Absolutely no {agents/skills}.md files were inserted at any point. No cheating mechanisms whatsoever 2. The cli agent was run in leaderboard compliant way (no modification of resources or timeouts) 3. The full terminal bench run was done using the fully open source version of the agent, no…
Apr 2026 · github.com · its alternatives →
- 2
Trails▲81Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Jan 2026 · trails-red.vercel.app · its alternatives →
- 3
Hey HN, We’ve been building browser agents for a while. In production, we kept converging on the same pattern: deterministic scripts for the happy path, agents only for edge cases. So we built Demonstrate Mode. The idea is simple: You perform your workflow once in a remote browser. Notte records the interactions and generates deterministic automation code. How it works: - Record clicks, inputs, navigations in a cloud browser - Compile them into deterministic code (no LLM at runtime) - Run and deploy on managed browser infrastructure Closest analog is Playwright codegen but: - Infrastructure…
2025 · notte.cc · its alternatives →
- 4
We built a computer use agent once and got it benchmarked at the 2nd position in the biggest benchmark for this field but in the process, we realized one thing, how outdated and inefficient these benchmarks are and how they don't truly capture how these tasks went. So we thought a community led benchmarking arena would be the way to go about. https://coarena.ai/. You just have to put in a task and two side by side windows with hidden models will show up and do the same task and at the end you vote which one did better(or none). We have a leaderboard where we rank all the…
Aug 2026 · coarena.ai · its alternatives →
- 5
Replicas▲239Hey HN! I'm Connor, creator of Replicas. I'm a heavy Codex/Claude Code user, and I wanted to hand off tasks to agents like I would to an engineer - kick something off before work, let it scaffold while I'm at lunch, even come back to a ready PR. The problem: existing alternatives use workspace VMs that are black boxes. I have no access to any of them - they can't run tests, read logs, or use tools like the Playwright MCP (which I use to verify agents actually fixed what they claim to fix). Replicas gives agents real cloud workspaces where they can: - Run your test suite and read the…
Jun 2026 · replicas.dev · its alternatives →
- 6
Hi HN! We’re the team behind Fiddler, and we recently built an Agent Cache feature. It works by capturing an LLM request/response pair and, once caching is enabled, serving subsequent matching calls locally so duplicate requests never hit the provider. It’s meant to reduce three pains that show up together in agent development: token cost, feedback-loop latency, and output non-determinism. We’d love to hear your thoughts.
Aug 2026 · telerik.com · its alternatives →
- 7
Hi! I’m Nathan: an ML Engineer at Mozilla.ai: I built agent-of-empires (aoe): a CLI application to help you manage all of your running Claude Code/Opencode sessions and know when they are waiting for you. - Written in rust and relies on tmux for security and reliability - Monitors state of cli sessions to tell you when an agent is running vs idle vs waiting for your input - Manage sessions by naming them, grouping them, configuring profiles for various settings I'm passionate about getting self-hosted open-weight LLMs to be valid options to compete with proprietary closed models. One…
Jan 2026 · github.com · its alternatives →
- 8
Single-agent LLMs suck at long-running complex tasks. We’ve open-sourced a multi-agent orchestrator that we’ve been using to handle long-running LLM tasks. We found that single LLM agents tend to stall, loop, or generate non-compiling code, so we built a harness for agents to coordinate over shared context while work is in progress. How it works: 1. Orchestrator agent that manages task decomposition 2. Sub-agents for parallel work 3. Subscriptions to task state and progress 4. Real-time sharing of intermediate discoveries between agents We tested this on a Putnam-level math problem, but the…
Feb 2026 · github.com · its alternatives →
- 9
I've been building computer-use tools for a while, and I quietly launched this about a month ago (122 Stars on GH). I figured it was worth sharing here. Over the last few months, a lot of computer-use agents have come out: Codex, Claude Code, CUA, and others. Most of them seem to work roughly like this: 1. Take a screenshot 2. Have the model predict pixel coordinates 3. Click x,y 4. Take another screenshot 5. Repeat That works, but it's slow, expensive in tokens, and fragile. If the UI shifts a few pixels, things break. And the model still doesn't know what any element actually is. But the…
May 2026 · github.com · its alternatives →
- 10
Hi HN, we're Kiran and Vijay! Over the past two years, we have built a columnar storage engine for observability: logs, metrics, and traces. Today, it's exciting for us to show what we've built on top of that foundation: LLM Agent Observability. Given how non-deterministic agents are, storing all traces without sampling was critical for us. But these traces tend to be in the MBs, sometimes GBs - we needed to store them inexpensively. We also needed the queries and analyses to be fast. To meet both these goals, we store them in S3 in our own parquet-like file format, and query them using AWS…
Jul 2026 · oodle.ai · its alternatives →
- 11
We recently started to use agents to update some documentation across our codebase on a weekly basis, and everything quickly turned into cron jobs, logs, and terminal output. it worked, but was hard to tell what agents were doing, why something failed, or whether a workflow was actually progressing. We thought it would be more interesting to treat agents as long-lived workers with state and responsibilities and explicit handoffs. Something you can actually see and reason about, instead of just tailing logs. So we built Clawe, a small coordination layer on top of OpenClaw that lets agent…
Feb 2026 · github.com · its alternatives →
- 12
AgenticLens▲64Visual debugging, tracing, and replay for agent workflows
Apr 2026 · agenticlens.in · its alternatives →
- 13
I'm Josh, founder of Synth. We've been working on coding agent optimization with method like GEPA and MIPRO (the latter of which, I helped to originally develop), agent evaluation via methods like RLMs, and large scale deployment for training and inference. We've also worked on patterns for memory, processing live context, and managing agent actions, combining it all in a single stack called Horizons. With the release of OpenAI's Frontier and the consumer excitement around OpenClaw, we think the timing is right to release a v0. It integrates with our sdk for evaluation and optimization but…
Feb 2026 · github.com · its alternatives →
- 14
Hi everyone! My team and I just open-sourced a bunch of cool agent dev tools: Invariant Explorer to visually inspect and understand AI traces and a testing framework, building on pytest.
2024 · github.com · its alternatives →
- 15
We built HALO (Hierarchal Agent Loop Optimizer), an open-source tool for debugging and optimizing AI agents using their execution traces. It’s a loop. Run your agent, feed the traces to HALO, get the report, apply the fixes, then re-run your agent. HALO takes in OTEL compliant traces from AI agents using tracing frameworks such as Langfuse, Arize/OpenInference, or even just plain JSONL. It uses an RLM (Recursive Language Model) to more efficiently break trace analysis into smaller subproblems in order to find recurring patterns across large amounts of data and fix systemic issues that…
Jun 2026 · github.com · its alternatives →
- 16
We combined Stanford's ACE (agents learning from execution feedback) with the Reflective Language Model pattern. Instead of reading traces in a single pass, an LLM writes and runs Python in a sandbox to programmatically explore them - finding cross-trace patterns that single-pass analysis misses. The framework achieved 2x consistency improvement on τ2-bench.
Mar 2026 · github.com · its alternatives →
- 17
we had hundreds of discussions with engineering leaders over the past few months, and everyone's trying to understand where they are in the AI journey. we collected all this data into a benchmark and built a free grader to let you know where you stand. you answer on a 1–5 scale (e.g., autonomy runs from "suggestions only" to "agents own multi-hour workflows across code, infra, and external systems") - takes about 5 minutes. https://agent-benchmarks.com/software-factory/ waiting for your results!
Jul 2026 · agent-benchmarks.com · its alternatives →
- 18
AgentWatch▲12Hi HN, I’m a solo developer and built AgentWatch to solve a problem I kept running into while building AI agents: preventing runaway loops and unexpected LLM spend before requests reach the model. AgentWatch sits in front of OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, Groq, and others to enforce budgets and runtime policies. I’d really appreciate your feedback. If you’re building AI agents, does this solve a problem you’ve experienced? I’d also love to hear what you’d improve or challenge.
Jun 2026 · agent-watch.dev · its alternatives →
- 19IS
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025 · its alternatives →
- 20
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
2025 · github.com · its alternatives →
- 21
We built meta-agent: an open-source library that automatically and continuously improves agent harnesses from production traces. Point it at an existing agent, a stream of unlabeled production traces, and a small labeled holdout set. An LLM judge scores unlabeled production traces as they stream. A proposer reads failed traces and writes one targeted harness update at a time, such as changes to prompts, hooks, tools, or subagents. The update is kept only if it improves holdout accuracy. On tau-bench v3 airline, meta-agent improved holdout accuracy from 67% to 87%. We open-sourced meta-agent.…
Apr 2026 · github.com · its alternatives →
- 22
Hi HN folks, I have been building AI agents for quite some time now. The shift has gone from LLM + Tools → LLM Workflows → Agent + Tools + Memory, and now we are finally seeing true agency emerge: agents as systems composed of tools, command-line access, fine-grained system capabilities, and memory. This way of building agents is powerful, and I believe it is here to stay. But the real question is: are the systems powering these agents ready for that future? I do not think so. Using Docker for a single agent is not going to scale well, because agents need to be lightweight and fast. LLMs…
Mar 2026 · github.com · its alternatives →
- 23
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com · its alternatives →
- 24YA
After adding "Human" as a LLM provider to OpenCode a few months ago as a joke, it turns-out that acting as a LLM is quite painful. But it was surprisingly useful for understanding real agent harnesses dev. So I thought I wouldn't leave anyone out! I made a small oss game - You Are An Agent - youareanagent.app - to share in the (useful?) frustration It's a bit ridiculous. To tell you about some entirely necessary features, we've got: - A full WASM arch-linux vm that runs in your browser for the agent coding level - A bad desktop simulation with a beautiful excel simulation for our computer…
Feb 2026 · youareanagent.app · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →