New eval from SWE-bench team evalutes LMs based on goals not tickets
Current evals test LMs on tasks: "fix this bug," "write a test" But we code to achieve goals: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals. Because real software dev isn’t about following instructions. It’s about achieving outcomes. Here's how it works: Two LMs enter a tournament. Each maintains its own codebase. Every round: 1. Edit Phase: LMs modify their codebases however they like 2. Competition phase: Codebases battle in an arena. 3. Repeat The LM that wins the majority of rounds is…
What it does
In the maker’s words, at launch
Current evals test LMs on tasks: "fix this bug," "write a test" But we code to achieve goals: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals. Because real software dev isn’t about following instructions. It’s about achieving outcomes. Here's how it works: Two LMs enter a tournament. Each maintains its own codebase. Every round: 1. Edit Phase: LMs modify their codebases however they like 2. Competition phase: Codebases battle in an arena. 3. Repeat The LM that wins the majority of rounds is declared winner. Arenas can be anything like games, trading sims, cybersec envs. We currently have 6 arenas implemented and support for 8 different programming languages. This has been one of our biggest projects in terms of scale to date. Over the past few months, we've completed 1.5k tournaments, totalling more than 50,400 agent runs. And you can look at all of these runs right now from your browser (links below!) You can find the rankings on our website (spoiler: Sonnet 4.5 tops the list), but almost more interesting: Humans are still way ahead! In one of our arena, even the worst solution from the human leaderboard is miles ahead of the best LM! And we're not surprised: LMs consistently fail to properly adapt to outcomes, hallucinate about reasons for failure, and produce ever messier codebases with every round. More information: https://codeclash.ai/ https://arxiv.org/pdf/2511.00839 https://github.com/codeclash-ai/codeclash
Does the same job
all alternatives →
- CVCodebase visualizer (I'll build the diagram for the first 20 repos)2025 · pie-crepe-38f.notion.site · ▲8
I worked as a software engineer at Amazon, SAP, and on open source. In all 3 places I have struggled with the friction of understanding codebases before I can make a contribution. I think this brain-fatiguing process can be improved. I am trying to solve it with a tool I built over the last 4 years called CodeCanvas: https://docs.code-canvas.com CodeCanvas visualizes codebases through interactive diagrams linked directly to source code. Users can record 'simulations' to demonstrate data flow and business logic. I’ve also recently added an LLM chat where it takes only the relevant…
- HWHow we use LLMs to find testing gaps, vulnerabilities in codebases2024 · github.com · ▲9
Hello everyone! I’m thrilled to announce the latest feature from Mutahunter.ai, the ultimate tool for finding and fixing weaknesses in your code. We’ve designed Mutahunter to leverage mutation testing powered by advanced LLMs, helping you uncover vulnerabilities and enhance your code quality effortlessly. Introducing our newest feature: Detailed Mutation Testing Reports! After running our mutation tests, Mutahunter now generates comprehensive reports that clearly summarize: • Vulnerable code gaps • Test case gaps These reports significantly reduce the cognitive load on developers by…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, November 2025
the whole month →
- IB
Life & fun · Nov 2025 · bitsnpieces.dev
- BBoing▲782
Life & fun · Nov 2025 · boing.greg.technology