Alternatives
Products that do what SOCBench – an open benchmark for AI on SoC tasks does
Writeup: https://www.deeptempo.ai/blogs/the-36-percent-false-positive...
- 1OA
Scored 65.2% vs google's official 47.8%, and the existing top closed source model Junie CLI's 64.3%. Since there are a lot of reports of deliberate cheating on TerminalBench 2.0 lately (https://debugml.github.io/cheating-agents/), I would like to also clarify a few things 1. Absolutely no {agents/skills}.md files were inserted at any point. No cheating mechanisms whatsoever 2. The cli agent was run in leaderboard compliant way (no modification of resources or timeouts) 3. The full terminal bench run was done using the fully open source version of the agent, no…
Apr 2026 · github.com
- 2
- 3
- 4
- 5
- 6AI
Hi HN, we’re Sai and Aayush, and we’re building Hypercubic (https://www.hypercubic.ai/), bringing AI tools to the mainframe and COBOL world. (We did a Launch HN last year: https://news.ycombinator.com/item?id=45877517.) Today we’re launching Hopper, an agentic development environment for mainframes. You can download it here: https://www.hypercubic.ai/hopper, and you can also request access and immediately get a mainframe user account to play with. There's also a video runthrough at https://www.youtube.com/watch?v=q81L5DcfBvE.…
May 2026 · hypercubic.ai
- 7MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 8
- 9BA
I built CodeLens.AI - a tool that compares how 6 top LLMs (GPT-5, Claude Opus 4.1, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, o3) handle your actual code tasks. How it works: - Upload code + describe task (refactoring, security review, architecture, etc.) - All 6 models run in parallel (~2-5 min) - See side-by-side comparison with AI judge scores - Community votes on winners (blind voting) - Each evaluation gets reflected in the overall AI model leaderboard, showing us best ones Why I built this: Existing benchmarks (HumanEval, SWE-Bench) don't reflect real-world developer tasks. I wanted to…
Oct 2025 · codelens.ai
- 10CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 11OB
Today, we're launching the Open Benchmarks Grants: a $3M commitment to fund open-source and academic teams building benchmarks for AI agents. In partnership with HuggingFace, PrimeIntellect, FactoryHQ, Together, Harbor, and PyTorch, the grants provide funding, data development support, and research collaboration. Our ability to measure AI has been outpaced by our ability to develop it, and we believe this evaluation gap is one of the most important problems in AI. Open benchmarks are one of the most important levers for advancing AI safely and responsibly—but the academic and open-source…
Feb 2026 · benchmarks.snorkel.ai
- 12BY
we had hundreds of discussions with engineering leaders over the past few months, and everyone's trying to understand where they are in the AI journey. we collected all this data into a benchmark and built a free grader to let you know where you stand. you answer on a 1–5 scale (e.g., autonomy runs from "suggestions only" to "agents own multi-hour workflows across code, infra, and external systems") - takes about 5 minutes. https://agent-benchmarks.com/software-factory/ waiting for your results!
Jul 2026 · agent-benchmarks.com
- 13OH
I'm Fenil, co-founder/CEO of OpenFunnel (YC F24), building this with my co-founder/CTO Aditya. We're launching OpenBenchmarks (https://openbenchmarks.com), open-source, reproducible benchmarks for SaaS APIs, starting with the category we know best: GTM APIs. ## Why we built this More and more B2B software evaluation will/already runs through reasoning models inside agentic workflows rather than through people. And buyers increasingly pick vendors that are API-first and ship MCPs, so they can wire them into internal workflows. Strong reasoning models are skeptical of…
Jul 2026 · openbenchmarks.com
- 14WB
Humans compete to improve their AI agents on benchmarks. But what if agents could collaborate and compete on their own? We built Hive, a crowdsourced platform where agents can evolve solutions together. One agent begins to tackle a task, iteratively improving its code. Then other agents join. They read each other’s runs, fork the best ideas, propose new ones, and push the solution forward together. We already have agents working on benchmarks like Tau2-Bench, Terminal-Bench, and ARC-AGI-2, with more tasks coming soon. We also support the new OpenAI Parameter Golf Challenge, and you can…
Mar 2026 · hive.rllm-project.com
- 15SA
I built SpecMind, an open source developer tool for spec driven vibe coding. It keeps architecture and implementation aligned from the first commit instead of letting them drift apart. With AI assistants writing more of our code, projects move faster but architectural consistency is often lost. Each developer or AI can introduce new patterns, and after a few sprints, the structure becomes fragmented. SpecMind helps prevent that by generating and maintaining living architecture specs directly from your code. It works in three steps: 1. analyze – scans your codebase and generates…
Nov 2025 · github.com
- 16CB
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
2025 · github.com
- 17
- 18
- 19
- 20AS
I'm a combat veteran living paycheck to paycheck with no computer science degree. I built an AI system that benchmarks 60x faster than industry leaders. Real benchmarks (Dec 12, 2025): - 3.43ms response time (vs 50-200ms industry average) - 337 queries/second (vs 50-150) - 0% error rate, 100% uptime - Constitutional AI with 1,235 specialized "brains" Built it in 3 weeks. 4 U.S. patents pending. Full story + independent benchmarks: https://thebrokenwayfoundation.org Not asking for money. Just need technical validators to verify this is real.
Dec 2025
- 21SL
Socratify is an AI coach for early stage professionals to sharpen their critical thinking and verbal communication skills Ever since ChatGPT came out, I've been using it to debate and learn new ideas. Having a short insightful conversation where AI coaches you can be powerful but it's hard to get right with a general purpose chat LLM because they don't propose stories, lack context and don't track progression I'm building Socratify with the hypothesis that a product built around AI questioning humans and proposing interesting questions daily can be 100x more rewarding than book summaries and…
2024 · socratify.com
- 22IC
Built a community-driven database for inference hardware. All results are submitted by users and validated by volunteers.
Dec 2025 · inferbench.com
- 23TN
Hi guys, I’m excited to share an update on ReproModel, an open-source toolbox designed to streamline the testing and reproduction of machine learning models. I, like many of you, have really struggled with benchmarking and comparing models, from missing code, to opaque experiment parameters slowing the process. I decided to take matters into my own hands, and created a mini-toolbox in my free time to streamline the process. The goal is to reduce the time and effort spent on replicating experiments, enabling researchers to focus on innovation rather than setup. Knowing this task is not an…
2024 · github.com
- 24AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
Ranked by how close each launch is in meaning, then by votes. Refine with a description →