Alternatives
Products that do what cto bench does
The ground truth code agent benchmark
- 1
- 2
- 3
- 4
- 5
- 6MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 7
- 8
- 9
- 10
- 11CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 12FA
Founder here. I built NEO, an AI agent designed specifically for AI and ML engineering workflows, after repeatedly hitting the same wall with existing tools: they work for short, linear tasks, but fall apart once workflows become long-running, stateful, and feedback-driven. In real ML work, you don’t just generate code and move on. You explore data, train models, evaluate results, adjust assumptions, rerun experiments, compare metrics, generate artifacts, and iterate; often over hours or days. Most modern coding agents already go beyond single prompts. They can plan steps, write files, run…
Jan 2026 · marketplace.visualstudio.com
- 13TS
Code review is the new bottleneck. "Tests passing" is no longer sufficient to trust the changes, and the (human) cost of evaluating the quality and robustness of new agent-written contributions is skyrocketing. We built Topos to evaluate code quality based on the structural properties of the programs themselves. We map your files to graphs (AST, CFG, CPG, MDG) and calculate metrics that can characterize how simple, composable, or secure your programs are. Agents can use this tool as they write and optimize based on your preferences. And yes, the inspiration for the repository is from…
Jun 2026 · krv.ai
- 14
- 15CB
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
2025 · github.com
- 16WB
Humans compete to improve their AI agents on benchmarks. But what if agents could collaborate and compete on their own? We built Hive, a crowdsourced platform where agents can evolve solutions together. One agent begins to tackle a task, iteratively improving its code. Then other agents join. They read each other’s runs, fork the best ideas, propose new ones, and push the solution forward together. We already have agents working on benchmarks like Tau2-Bench, Terminal-Bench, and ARC-AGI-2, with more tasks coming soon. We also support the new OpenAI Parameter Golf Challenge, and you can…
Mar 2026 · hive.rllm-project.com
- 17AS
AI changed the way we code, but we're still using the old processes, and we've become the bottleneck, the AI is waiting for us - to reply, to open our laptops, to review the code, and so on. We're building the future of AI software development. The agents are autonomous, they run in sandboxes, automatically fix the pipelines, and deliver you the final, working code. You can use live preview to see the changes they made. Working across multiple repositories, all within the same session. This is the future - you don't need an IDE, and you don't have to run anything locally.
Mar 2026 · agenhq.com
- 18
- 19
- 20SA
Hi HN, This is a project I've been working on since the beginning of 2025 full time, without funding. Coding agents have fundamentally changed the way we write software. When you let an agent write code, pull dependencies, and run scripts, you are delegating trust while still keeping the responsibility. You shouldn't have to choose between moving fast with agents and maintaining basic control over your host machine. Normally, we just inspect the final result, treating the app like a black box. Most security tools only sandbox the app runtime and ignore the build phase. CapaKit is my attempt…
Jun 2026 · capakit.com
- 21AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
- 22BA
I built CodeLens.AI - a tool that compares how 6 top LLMs (GPT-5, Claude Opus 4.1, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, o3) handle your actual code tasks. How it works: - Upload code + describe task (refactoring, security review, architecture, etc.) - All 6 models run in parallel (~2-5 min) - See side-by-side comparison with AI judge scores - Community votes on winners (blind voting) - Each evaluation gets reflected in the overall AI model leaderboard, showing us best ones Why I built this: Existing benchmarks (HumanEval, SWE-Bench) don't reflect real-world developer tasks. I wanted to…
Oct 2025 · codelens.ai
- 23BY
Hi HN. We launched a free AI Coding Risk Assessment tool to help engineering teams and businesses benchmark the security and compliance posture of their AI coding workflows and policies against peers in the industry. This anonymous 24-question survey delivers: - A 0–100 risk score that measures your AI coding security posture - A live benchmark that compares your AI-assisted development practices with peers - A research-based checklist that identifies improvement areas We're seeing more and more clients signal their concerns about the sudden increase of source code written by AI coding…
Nov 2025
- 24CA
Hey HN, Most AI “agents” I’ve tried are basically chatbots with amnesia — they forget everything the moment you close the tab and can’t do anything unless you’re sitting there watching them. I wanted real AI coworkers that just… work. So I built Computer Agents (aiOS). Every agent you create gets its own isolated computer in the cloud — complete with persistent memory, a real file system, code execution environment (with automatic dependency management), and the ability to run scheduled or webhook-triggered tasks 24/7. You give it a goal (“research this market and email me a report…
Mar 2026 · computer-agents.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →