Alternatives
Products that do what CVE-Bench, the first LLM benchmark using real-world web vulnerabilities does
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
- 1
- 2
- 3WP
Anthropic and OpenAI's publicly available models are explicitly guard-railed so that they refuse offensive tasks. And their cyber-focussed models are gated for enterprises. This leaves SMEs and mid market open to major vulnerabilities. AI can be used as both an adversarial and defensive tool in the world of cyber. A worst case outcome is if only the adversaries have access. Meanwhile, most existing AI cyber tools are just wrappers. The problem is that they still have all the guardrails on from the foundation model where they will inherit its refusals. For this project we've post-trained a…
Jun 2026 · argusred.com
- 4VC
2020 · snyk.io
- 5AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
- 6MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 7CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 8OA
Hi HN, we're Kiran and Vijay! Over the past two years, we have built a columnar storage engine for observability: logs, metrics, and traces. Today, it's exciting for us to show what we've built on top of that foundation: LLM Agent Observability. Given how non-deterministic agents are, storing all traces without sampling was critical for us. But these traces tend to be in the MBs, sometimes GBs - we needed to store them inexpensively. We also needed the queries and analyses to be fast. To meet both these goals, we store them in S3 in our own parquet-like file format, and query them using AWS…
Jul 2026 · oodle.ai
- 9IB
Hi HN, I’m the creator of Cordum. I’ve been working in DevOps and infrastructure for years (currently in the fintech/security space), and as I started playing with AI agents, I noticed a scary pattern. Most "safety" mechanisms rely on system prompts ("Please don't do X") or flimsy Python logic inside the agent itself. If we treat agents as autonomous employees, giving them root access and hoping they listen to instructions felt insane to me. I wanted a way to enforce hard constraints that the LLM cannot override, no matter how "jailbroken" it gets. So I built Cordum. It’s an open-source…
Jan 2026 · github.com
- 10BY
Hi HN. We launched a free AI Coding Risk Assessment tool to help engineering teams and businesses benchmark the security and compliance posture of their AI coding workflows and policies against peers in the industry. This anonymous 24-question survey delivers: - A 0–100 risk score that measures your AI coding security posture - A live benchmark that compares your AI-assisted development practices with peers - A research-based checklist that identifies improvement areas We're seeing more and more clients signal their concerns about the sudden increase of source code written by AI coding…
Nov 2025
- 11IS
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025
- 12BY
we had hundreds of discussions with engineering leaders over the past few months, and everyone's trying to understand where they are in the AI journey. we collected all this data into a benchmark and built a free grader to let you know where you stand. you answer on a 1–5 scale (e.g., autonomy runs from "suggestions only" to "agents own multi-hour workflows across code, infra, and external systems") - takes about 5 minutes. https://agent-benchmarks.com/software-factory/ waiting for your results!
Jul 2026 · agent-benchmarks.com
- 13ME
We are building a VM that helps you simulate realistic production conditions, model latencies, different interleaving, user requests, and find bugs. Every non-deterministic property is turned into a knob you or a coding agent can control. We have helped teams perfectly reproduce support incidents and found bugs in some of the world's most well tested software (including a database).
Jun 2026 · workers.io
- 14AB
Hi HN, Zidan here. I’ve been experimenting with AI-assisted debugging and noticed a recurring gap: most tools optimize for agent-led exploration (ex: giving claude code a browser to click around and try to reproduce an issue). But in many cases, I've already found the bug myself. What I actually want is a way to hand the agent the exact context I just saw - without retyping steps, copying logs, or hoping it can reproduce the behavior. So we built FlowLens, an open-source MCP server + Chrome extension that captures browser context and lets coding agents inspect it as structured, queryable…
Nov 2025 · github.com
- 15WB
Over the past few months, as we scaled our internal AI Agents, we hit a dead end: Running LLM-generated arbitrary code in Docker is basically running naked on security due to container escape risks. But using full traditional VMs takes minutes to boot and eats too much memory to support high-density concurrency. We loved the developer experience of SaaS sandboxes on the market, but they are closed-source, expensive, and have too high a barrier to entry for self-hosting. So, our team decided to build our own. After months of grinding, using RustVMM and KVM, we built a blazing-fast,…
Apr 2026 · github.com
- 16BA
Hi HN, I’d like to share an open-source project we’ve been working on for a while: Browser4. The motivation came from a recurring frustration: most browser automation tools (Playwright, Selenium, Puppeteer) are excellent for human-written scripts, but start to show friction when used as a core execution layer for AI agents or at very high concurrency. So instead of building “another wrapper around Playwright”, we experimented with a different direction: designing a browser engine where AI agents are first-class citizens. ### What Browser4 is Browser4 is a browser automation engine built on…
Dec 2025 · github.com
- 17TS
2016 · tucloud.com
- 18WI
At Laminar (https://github.com/lmnr-ai/lmnr) we're building open source AI observability platform in Rust. We obsess over instrumentation DX for our Python and TS SDKs and in this new blog we outline how we made the most seamless way of instrumenting recently released claude agent sdk
Dec 2025 · laminar.sh
- 19SO
hello everyone, my first post! AA here, founder of ⌘ Langbase.com — we are a developer platform for building and scaling serverless AI memory agents. I know surveys can be boring, but this one’s different—it’s interactive! That's very much intentional. My team and I have been up for the last 21 hours putting together this report. This was a looot of work, so I hope y'all like it. Introducing … State of AI Agents 2024 report On Langbase, we processed 184 billion tokens and handled 786 million AI agent runs from 36K developers. From all that data plus insights from 3.4K builders who filled out…
2024 · langbase.com
- 20
Triage CVEs by what attackers are actually exploiting
Jun 2026 · precursorintelligence.com
- 21AR
Feb 2026 · trypillar.com
- 22
- 23LA
We combined Stanford's ACE (agents learning from execution feedback) with the Reflective Language Model pattern. Instead of reading traces in a single pass, an LLM writes and runs Python in a sandbox to programmatically explore them - finding cross-trace patterns that single-pass analysis misses. The framework achieved 2x consistency improvement on τ2-bench.
Mar 2026 · github.com
- 24
5 AI agents review every PR for bugs in 10 seconds
Jun 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →