AI · alternatives · 2026
24 alternatives to CVE-Bench, the first LLM benchmark using real-world web vulnerabilities
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
APIEval-20▲121An open benchmark for AI agents that test APIs
May 2026 · resources.kusho.ai · its alternatives →
- 2
Fabraix▲196We built Adversarial Cost to Exploit (ACE), a benchmark that measures the token expenditure an autonomous adversary must invest to breach an LLM agent. Instead of binary pass/fail, ACE quantifies adversarial effort in dollars, enabling game-theoretic analysis of when an attack is economically rational. We tested six budget-tier models (Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, Claude Haiku 4.5) with identical agent configs and an autonomous red-teaming attacker. Haiku 4.5 was an order of magnitude harder to break than every other model; $10.21 mean…
May 2026 · fabraix.com · its alternatives →
- 3
- 4
We built a computer use agent once and got it benchmarked at the 2nd position in the biggest benchmark for this field but in the process, we realized one thing, how outdated and inefficient these benchmarks are and how they don't truly capture how these tasks went. So we thought a community led benchmarking arena would be the way to go about. https://coarena.ai/. You just have to put in a task and two side by side windows with hidden models will show up and do the same task and at the end you vote which one did better(or none). We have a leaderboard where we rank all the…
Aug 2026 · coarena.ai · its alternatives →
- 5
After training calculator agent via RL, I really wanted to go bigger! So I built RL infrastructure for training long-horizon terminal/coding agents that scales from 2x A100s to 32x H100s (~$1M worth of compute!) Without any training, my 32B agent hit #19 on Terminal-Bench leaderboard, beating Stanford's Terminus-Qwen3-235B-A22! With training... well, too expensive, but I bet the results would be good! *What I did*: - Created a Claude Code-inspired agent (system msg + tools) - Built Docker-isolated GRPO training where each rollout gets its own container - Developed a multi-agent…
2025 · github.com · its alternatives →
- 6
Anthropic and OpenAI's publicly available models are explicitly guard-railed so that they refuse offensive tasks. And their cyber-focussed models are gated for enterprises. This leaves SMEs and mid market open to major vulnerabilities. AI can be used as both an adversarial and defensive tool in the world of cyber. A worst case outcome is if only the adversaries have access. Meanwhile, most existing AI cyber tools are just wrappers. The problem is that they still have all the guardrails on from the foundation model where they will inherit its refusals. For this project we've post-trained a…
Jun 2026 · argusred.com · its alternatives →
- 7
Trails▲81Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Jan 2026 · trails-red.vercel.app · its alternatives →
- 8
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai · its alternatives →
- 9
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com · its alternatives →
- 10
A vendor-neutral Kubernetes incident benchmark for testing AI investigation tools across detection, diagnosis, and mitigation. Deploy faults, capture investigations, and compare results with a configurable AI judge. - edgedelta/project-arena
15d ago · github.com · its alternatives →
- 11
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com · its alternatives →
- 12
We build runtime security for AI agents. The playground started as an internal tool that we used to test our own guardrails. But we kept finding the same types of vulnerabilities because we think about attacks a certain way. At some point you need people who don't think like you. So we open-sourced it. Each challenge is a live agent with real tools and a published system prompt. Whenever a challenge is over, the full winning conversation transcript and guardrail logs get documented publicly. Building the general-purpose agent itself was probably the most fun part. Getting it to reliably use…
Mar 2026 · github.com · its alternatives →
- 13
New ways of breaking app-integrated LLMs . Contribute to greshake/llm-security development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 14ΤB
τ-Bench is an open benchmark for evaluating AI agents on grounded, multi-turn customer service tasks with verifiable outcomes. It's been great to see the community adopt it since launch — this is now the third iteration. With τ³-Bench, we're extending it to two new settings: knowledge-intensive retrieval and full-duplex voice. τ-Knowledge: agents must navigate ~700 interconnected policy documents to complete multi-step tasks. Best frontier model (GPT-5.2, high reasoning) hits ~25%. The surprising part: even when you hand the model the exact documents it needs, performance only reaches ~40%.…
Mar 2026 · its alternatives →
- 15
PwnScan▲8For at least a few years (since before COVID), in my circle everbody wants an AI tool that finds vulnerabilities in binaries. I have my reservations about such a tool because of my perception that AI is overhyped, so I decided to build one myself to see how useful it could be (or not). I was actually kinda surprised how well it worked for my collection of binaries with known CVEs. It definitely has false positives and negatives, as any static analysis tool would, but I think its actually somewhat usable. I dont think the results are super actionable, but interesting nonetheless. I still…
2025 · pwnscan.com · its alternatives →
- 16
Hi HN, I’m the creator of Cordum. I’ve been working in DevOps and infrastructure for years (currently in the fintech/security space), and as I started playing with AI agents, I noticed a scary pattern. Most "safety" mechanisms rely on system prompts ("Please don't do X") or flimsy Python logic inside the agent itself. If we treat agents as autonomous employees, giving them root access and hoping they listen to instructions felt insane to me. I wanted a way to enforce hard constraints that the LLM cannot override, no matter how "jailbroken" it gets. So I built Cordum. It’s an open-source…
Jan 2026 · github.com · its alternatives →
- 17BY
Hi HN. We launched a free AI Coding Risk Assessment tool to help engineering teams and businesses benchmark the security and compliance posture of their AI coding workflows and policies against peers in the industry. This anonymous 24-question survey delivers: - A 0–100 risk score that measures your AI coding security posture - A live benchmark that compares your AI-assisted development practices with peers - A research-based checklist that identifies improvement areas We're seeing more and more clients signal their concerns about the sudden increase of source code written by AI coding…
Nov 2025 · its alternatives →
- 18
We built FullHunt Agentic AI, an autonomous tool that maps an organization’s internet-facing assets and possible attack paths in seconds. It automatically discovers exposed assets, correlates CVEs (EPSS, CISA KEV, FullHunt data), and produces a prioritized action plan. Try a prompt like: "Discover the attack surface of acme.com and identify possible attack paths."
2021 · fullhunt.io · its alternatives →
- 19IS
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025 · its alternatives →
- 20
we had hundreds of discussions with engineering leaders over the past few months, and everyone's trying to understand where they are in the AI journey. we collected all this data into a benchmark and built a free grader to let you know where you stand. you answer on a 1–5 scale (e.g., autonomy runs from "suggestions only" to "agents own multi-hour workflows across code, infra, and external systems") - takes about 5 minutes. https://agent-benchmarks.com/software-factory/ waiting for your results!
Jul 2026 · agent-benchmarks.com · its alternatives →
- 21
The first benchmark that evaluates what AI agents actually need from document parsing. 2,000 human-verified pages, 169K deterministic test rules, 5 capability dimensions.
Apr 2026 · parsebench.ai · its alternatives →
- 22
- 23
Find AI vulnerabilities before hackers do
Mar 2026 · promptbrake.com · its alternatives →
- 24
ModelRed▲10Hi everyone, We built ModelRed to test AI models and apps for security issues. Ran 4,182 attack probes against 9 leading models to see what would break. Leaderboard: https://modelred.ai (no signup, just check it out) Claude scored 9.5/10 but still failed on medical/financial prompts. Mistral Large scored 3.3/10. The gap between best and worst is huge. We test for prompt injections, data leaks, jailbreaks, risky tool calls, domain specific hacks, basically everything that goes wrong when your LLM has access to real data and APIs. The platform runs these tests…
Nov 2025 · modelred.ai · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →