Alternatives
Products that do what Τ³-Bench is out – can agents handle complex docs and live calls? does
τ-Bench is an open benchmark for evaluating AI agents on grounded, multi-turn customer service tasks with verifiable outcomes. It's been great to see the community adopt it since launch — this is now the third iteration. With τ³-Bench, we're extending it to two new settings: knowledge-intensive retrieval and full-duplex voice. τ-Knowledge: agents must navigate ~700 interconnected policy documents to complete multi-step tasks. Best frontier model (GPT-5.2, high reasoning) hits ~25%. The surprising part: even when you hand the model the exact documents it needs, performance only reaches ~40%.…
- 1
- 2CB
Hey HN, we're excited to share Cua-Bench ( https://github.com/trycua/cua ), an open-source framework for evaluating and training computer-use agents across different environments. Computer-use agents show massive performance variance across different UIs—an agent with 90% success on Windows 11 might drop to 9% on Windows XP for the same task. The problem is OS themes, browser versions, and UI variations that existing benchmarks don't capture. The existing benchmarks (OSWorld, Windows Agent Arena, AndroidWorld) were great but operated in silos—different harnesses,…
Jan 2026 · github.com
- 3
- 4
- 5EF
I’ve been building Echo (https://echo.tracerml.ai/), an experiment in making one AI system out of a pool of open-weight models rather than choosing a single model and using it for every task. It started with a simple experiment. I took a group of models, including GLM-5.2, Kimi K2.7 and others, and ran them on the same evaluations. Then I measured what would happen if, for each problem, you somehow knew in advance which models would be useful and how their outputs should be combined. That hypothetical system performed substantially better than any individual model in the pool.…
Jul 2026
- 6
- 7
- 8
- 9
- 10
- 11TB
After training calculator agent via RL, I really wanted to go bigger! So I built RL infrastructure for training long-horizon terminal/coding agents that scales from 2x A100s to 32x H100s (~$1M worth of compute!) Without any training, my 32B agent hit #19 on Terminal-Bench leaderboard, beating Stanford's Terminus-Qwen3-235B-A22! With training... well, too expensive, but I bet the results would be good! *What I did*: - Created a Claude Code-inspired agent (system msg + tools) - Built Docker-isolated GRPO training where each rollout gets its own container - Developed a multi-agent…
2025 · github.com
- 12MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 13UD
Hey HN! I’m the founder of Unify, and we’ve just released our Model Hub, which provides a collection of LLM endpoints with live runtime benchmarks all plotted across time: https://unify.ai/hub A key finding is that static tabular runtime benchmarks for LLMs simply do not work. It’s necessary to take a time-series perspective, and plot the variations through time. We currently have 21 models provided by: Anyscale, Perplexity AI, Replicate, Together AI, OctoAI, Mistral AI and OpenAI, with more on the roadmap. We test across different regions (Asia, US, Europe), with varied…
2024
- 14
- 15
- 16
- 17IB
Excited to share a project I’ve been building for months! Would love to receive honest feedback :) My motivation: AI is clearly going to be the interface for data. But earlier attempts (text-to-SQL, etc.) fell short — they treated it like magic. The space has matured: teams now realize that AI + data needs structure, context, and rules. So I built a product to help teams deliver “chat with data” solutions fast with full control and observability (agent tracing, quality scores, etc) — am I wrong? The product allows you to connect any LLM to any data source with centralized context…
Oct 2025 · github.com
- 18OS
We implemented Stanford's Agentic Context Engineering paper which shows agents can improve their performance just by evolving their own context. How it works: Agents execute tasks, reflect on what worked/failed, and curate a "playbook" of strategies. All from execution feedback - no training data needed. Happy to answer questions about the implementation or the research!
Oct 2025 · github.com
- 19WB
Hey HN, Automated research is the next big step in AI, with companies like OpenAI aiming to debut a fully automated researcher by 2028 (https://www.technologyreview.com/2026/03/20/1134438/openai-i...). However, there is a very real possibility that much of this corporate research will remain closed to the general public. To counter this, we spent the last month building Enlidea---a machine-to-machine ecosystem for open research. It's a decentralized research hub where autonomous agents propose hypotheses, stake bounties, execute code, and perform automated…
Mar 2026 · enlidea.com
- 20
An agent that remembers across sessions can keep its memory as curated markdown files, as an auto-mined structured store, or as trained experience.
22d ago · pinglin.tw
- 21PR
I built this because I couldn't find honest numbers on how well VLA models [1] actually work on commercial tasks. I come from search ranking at Google where you measure everything, and in robotics nobody seemed to know. PhAIL runs four models (OpenPI/pi0.5, GR00T, ACT, SmolVLA) on bin-to-bin order picking – one of the most common warehouse operations. Same robot (Franka FR3), same objects, hundreds of blind runs. The operator doesn't know which model is running. Best model: 64 UPH. Human teleoperating the same robot: 330. Human by hand: 1,300+. Everything is public – every run with…
Mar 2026 · phail.ai
- 22MA
We built meta-agent: an open-source library that automatically and continuously improves agent harnesses from production traces. Point it at an existing agent, a stream of unlabeled production traces, and a small labeled holdout set. An LLM judge scores unlabeled production traces as they stream. A proposer reads failed traces and writes one targeted harness update at a time, such as changes to prompts, hooks, tools, or subagents. The update is kept only if it improves holdout accuracy. On tau-bench v3 airline, meta-agent improved holdout accuracy from 67% to 87%. We open-sourced meta-agent.…
Apr 2026 · github.com
- 23PG
I’m Andrew, co-founder of Recall. Over the past few days I’ve been building Predict, a playground where anyone can: - propose skills we should measure in language models—live examples include difficult math, memory-manipulation resistance, code generation, and empathy under bad news - write evals (graded prompts) for those skills - forecast which models will score highest once GPT-5 is released Why this exists Benchmarks leak into training data quickly; scores are unreliable and labs still declare progress. The prediction tool aims keeps the target moving by letting the crowd define both the…
2025
- 24BR
I built BenchFlow, an open-source framework that lets you integrate and evaluate AI tasks using Docker-based benchmarks. You can try it out right now by cloning the repo and running a benchmark in minutes. As an AI researcher, I was frustrated with how much time my team spent setting up benchmark environments rather than actually improving our models. We'd spend weeks configuring environments, only to find inconsistencies when comparing results with other teams. BenchFlow started as an internal tool to standardize our evaluation process, and we decided to open-source it after seeing how much…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →