Alternatives
Products that do what AgentDiff does
Trajectory regression testing for AI agents
- 1
Trace, evaluate, and improve AI agents in production
30d ago · telerik.com
- 2
- 3
- 4
- 5
- 6
- 7
- 8IB
I built a tool to roast landing pages with AI agents. I was gathering feedback from watching landing page roast videos, and figured out I could prompt LLMs to analyse a screenshot and roast based on the same criteria. It's not 100% accurate yet, but it has been really insightful when I've tested it on my own websites. Let me know what you think!
2024 · roastmylandingpage.io
- 9AL
Hi HN, I built this to address what I see as the fundamental problem with ReAct-style agents: compounding errors. Even a small mistake made early enough in the loop can snowball and ruin the final output. But with search, agents can look multiple steps ahead and backtrack before committing to a particular trajectory. This has already been shown in a few papers to help agents avoid mistakes and boost overall task performance, but there's no easy way to actually build these kinds of agents. So that's why I made this framework. I believe search will eventually become table stakes for building…
2024 · github.com
- 10AB
Hi HN, Zidan here. I’ve been experimenting with AI-assisted debugging and noticed a recurring gap: most tools optimize for agent-led exploration (ex: giving claude code a browser to click around and try to reproduce an issue). But in many cases, I've already found the bug myself. What I actually want is a way to hand the agent the exact context I just saw - without retyping steps, copying logs, or hoping it can reproduce the behavior. So we built FlowLens, an open-source MCP server + Chrome extension that captures browser context and lets coding agents inspect it as structured, queryable…
Nov 2025 · github.com
- 118B
Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Jan 2026 · trails-red.vercel.app
- 12AT
Hi Hacker News! We're launching Zalor, an agent testing platform. Agents often break when you tweak system prompts, swap models, or add tools. Zalor automatically generates test scenarios and evaluates your agent so you know it's reliable before deploying to production. We currently support the OpenAI Agents SDK and are onboarding other frameworks. A GitHub integration is coming so you can get feedback on every update. Looking forward to hearing feedback from people building agents.
Mar 2026 · agents.zalor.ai
- 13AB
Hi everyone! My team and I just open-sourced a bunch of cool agent dev tools: Invariant Explorer to visually inspect and understand AI traces and a testing framework, building on pytest.
2024 · github.com
- 14AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
- 15RS
What relai-sdk is an open-source toolkit for making AI agents reliable via a complete learning loop: simulate → evaluate → optimize. Why Agent runs are stochastic; tool-calls fail; hard to reproduce, measure, and fix at scale. It’s also hard to align behavior with goals across output quality/format, cost, and latency. We need a loop that integrates user feedback and LLM evaluators directly into the agent code (prompts, configs, models, graphs) without overfitting. How - Simulation: LLM personas, mocked MCP servers/tools, synthetic data; can condition on real traces - Evaluation:…
Oct 2025 · github.com
- 16FC
Hi everyone, I’ve been working on an open-source tool called Flakestorm to test the reliability of AI agents before they hit production. Most agent testing today focuses on eval scores or happy-path prompts. In practice, agents tend to fail in more mundane ways: typos, tone shifts, long context, malformed input, or simple prompt injections — especially when running on smaller or local models. Flakestorm applies chaos-engineering ideas to agents. Instead of testing one prompt, it takes a “golden prompt”, generates adversarial mutations (semantic variations, noise, injections, encoding edge…
Jan 2026
- 17
- 18OS
Hey HN! We built EvalKit, a library you embed to capture agent actions and a UI where domain experts give feedback, evaluate and improve AI agents. We experienced, in large agentic systems, prompt-engineering or auto-prompt improvement tool can get accuracy from 0 to 50% but for increasing accuracy to 100% we had to work with domain experts. Example -> In a law ai agent, lawyers are needed because law is complex and lawyers have a deeper context compared to non-lawyers. Other evaluation tools in the market focus on the experience of the developer and we are focusing on making as easy as…
2025 · github.com
- 19AP
Hi HN, I’m a solo developer and built AgentWatch to solve a problem I kept running into while building AI agents: preventing runaway loops and unexpected LLM spend before requests reach the model. AgentWatch sits in front of OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, Groq, and others to enforce budgets and runtime policies. I’d really appreciate your feedback. If you’re building AI agents, does this solve a problem you’ve experienced? I’d also love to hear what you’d improve or challenge.
Jun 2026 · agent-watch.dev
- 20SE
Hey HN! I built self-driving sim and eval at Waymo. Now I’m building Scorecard to bring that approach to agent eval: reproducible, automated scoring for AI. Scorecard lets you: - Run LLM-as-judge evals on agent workflows: test tool usage, multi-step reasoning, and task completion in CI/CD or in a playground. - Debug failures with OpenTelemetry traces: see which tool failed, why your agent looped, and where reasoning went wrong. - Collaborate on datasets, simulated agents, and evaluation metrics. Try it out → https://app.scorecard.io (free tier, no payment required!) Docs →…
Oct 2025 · docs.scorecard.io
- 21HA
Hi HN, I am Umer. I recently built an experimental framework called HyperFlow to explore the idea of self-improving AI agents. Usually, when an agent fails a task, we developers step in to manually tweak the prompt or adjust the code logic. I wanted to see if an agent could automate its own improvement loop. Built on LangChain and LangGraph, HyperFlow uses two agents: - A TaskAgent that solves the domain problem. - A MetaAgent that acts as the improver. The MetaAgent looks at the TaskAgent's evaluation logs, rewrites the underlying Python code, tools, and prompt files, and then tests the new…
Apr 2026
- 22IB
The main goal of this was to be able to not just run multiple Claude Code sessions at once, but actually manage them and keep track of what I was doing. Sometimes this is multiple attempts on the same task, sometimes I work several tasks at once. Really I was just sick of twiddling my thumbs waiting for the coding agent to finish, and I wanted it to be easy to work on/review/test another change while I waited.
2025 · github.com
- 23SR
Hello all, I'm a software developer. Over the last few months more and more of my work has turned into using coding agents instead of typing the whole code myself. Usually a few claude sessions at once, sometimes codex, one per feature or per revealed bug. I ran them in a split terminal for a few weeks, and quickly spotted two main problems. The first is that I couldn't easily tell which agent was stuck waiting on me and which was still working, so I'd cycle through sessions and checking on them. The second one: agents sharing a single branch step on each other. Two of them could be editing…
Jul 2026 · shikigami.dev
- 24CA
I built this because I was tired of creating pull requests in 20 repositories just to change a single line of workflow job version. With Infra as AI, just mention the change. Agents work on all repos in parallel, read the docs, make a bunch of PRs and fill in the description. You can see the demo of the actual dashboard in the landing. Let me know your thoughts :) It means a lot to me!
Sep 2025 · infrastructureas.ai
Ranked by how close each launch is in meaning, then by votes. Refine with a description →