Alternatives
Products that do what New eval from SWE-bench team evalutes LMs based on goals not tickets does
Current evals test LMs on tasks: "fix this bug," "write a test" But we code to achieve goals: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals. Because real software dev isn’t about following instructions. It’s about achieving outcomes. Here's how it works: Two LMs enter a tournament. Each maintains its own codebase. Every round: 1. Edit Phase: LMs modify their codebases however they like 2. Competition phase: Codebases battle in an arena. 3. Repeat The LM that wins the majority of rounds is…
- 1SP
Hey HN! We're Charles and Dean, and we're building Stage: a code review tool that guides you through reading a PR step by step, instead of piecing together a giant diff. Here's a demo video: https://www.tella.tv/video/stage-demo-1pph. You can play around with some example PRs here: https://stagereview.app/explore. Teams are moving faster than ever with AI these days, but more and more engineers are merging changes that they don't really understand. The bottleneck isn't writing code anymore, it's reviewing it. We're two engineers who got frustrated with…
Apr 2026 · stagereview.app
- 2
- 3
- 4
- 5
- 6IL
I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process…
2024 · github.com
- 7LS
Hi HN! Stefan here from superglue and today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs. We gave LLMs API documentation and asked them to write code that makes actual API calls. Things like "create a Stripe customer" or "send a Slack message". We're not testing if they can use SDKs; we're testing if they can write raw HTTP requests (with proper auth, headers, body formatting) that actually work when executed against real API endpoints and can extract relevant information from that response. tl:dr: LLMs suck at…
2025 · github.com
- 8
- 9SC
Hey HN! We're Charles and Dean. A few weeks ago we posted about Stage, a code review tool that guides you through reading a PR step by step - https://news.ycombinator.com/item?id=47796818. We got a lot of great feedback but also heard from many people that they wanted to have the chapters experience even before opening a PR… so we built the Stage CLI as the local, open-source version that anyone can try. Here’s a quick demo video: https://www.tella.tv/video/stage-cli-demo-f55q It works with any coding agent of your choice. The skill instructs the agent to…
May 2026 · github.com
- 10
- 11CL
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com
- 122C
Single-agent LLMs suck at long-running complex tasks. We’ve open-sourced a multi-agent orchestrator that we’ve been using to handle long-running LLM tasks. We found that single LLM agents tend to stall, loop, or generate non-compiling code, so we built a harness for agents to coordinate over shared context while work is in progress. How it works: 1. Orchestrator agent that manages task decomposition 2. Sub-agents for parallel work 3. Subscriptions to task state and progress 4. Real-time sharing of intermediate discoveries between agents We tested this on a Putnam-level math problem, but the…
Feb 2026 · github.com
- 13CT
I've been building a tool that changes how LLM coding agents explore codebases, and I wanted to share it along with some early observations. Typically claude code globs directories, greps for patterns, and reads files with minimal guidance. It works in kind of the same way you'd learn to navigate a city by walking every street. You'll eventually build a mental map, but claude never does - at least not any that persists across different contexts. The Recursive Language Models paper from Zhang, Kraska, and Khattab at MIT CSAIL introduced a cleaner framing. Instead of cramming everything into…
Feb 2026 · github.com
- 14AR
Hi HN, I built this open-source LLM red teaming tool based on my experience scaling LLMs at a big co to millions of users... and seeing all the bad things people did. How it works: - Uses an unaligned model to create toxic inputs - Runs these inputs through your app using different techniques: raw, prompt injection, and a chain-of-thought jailbreak that tries to re-frame the request to trick the LLM. - Probes a bunch of other failure cases (e.g. will your customer support bot recommend a competitor? Does it think it can process a refund when it can't? Will it leak your user's address?) -…
2024 · promptfoo.dev
- 15AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 16II
2023 · github.com
- 17
- 18NB
May 2026 · programbench.com
- 19IB
Apr 2026 · github.com
- 20IM
It’s written in Python and I call it GoalChain. It lets you build a conversation flow graph that the user traverses. When there’s enough input it spits out a dictionary with the defined fields. Otherwise it will jump state to state as led by the user. It was fun to write, and it’s surprisingly effective if you keep in mind you’re prompt-engineering every string and field name. README.md has a mini-tutorial. Would be cool to get some ideas for how to build it further and what improvements I could make.
2024 · github.com
- 21CV
I worked as a software engineer at Amazon, SAP, and on open source. In all 3 places I have struggled with the friction of understanding codebases before I can make a contribution. I think this brain-fatiguing process can be improved. I am trying to solve it with a tool I built over the last 4 years called CodeCanvas: https://docs.code-canvas.com CodeCanvas visualizes codebases through interactive diagrams linked directly to source code. Users can record 'simulations' to demonstrate data flow and business logic. I’ve also recently added an LLM chat where it takes only the relevant…
2025 · pie-crepe-38f.notion.site
- 22HW
Hello everyone! I’m thrilled to announce the latest feature from Mutahunter.ai, the ultimate tool for finding and fixing weaknesses in your code. We’ve designed Mutahunter to leverage mutation testing powered by advanced LLMs, helping you uncover vulnerabilities and enhance your code quality effortlessly. Introducing our newest feature: Detailed Mutation Testing Reports! After running our mutation tests, Mutahunter now generates comprehensive reports that clearly summarize: • Vulnerable code gaps • Test case gaps These reports significantly reduce the cognitive load on developers by…
2024 · github.com
- 23WW
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best…
2024
- 24AC
I had an idea on a competition in a highly constrained environment: 32x32 matrix, with bots running code you write! The live site is https://kingofthegrid.com/. Each bot emulates Z80 CPU and they all get equal execution time, so the most efficient algorithm wins. Bots can move, eat, clone themselves, and most importantly, survive the competition. Each submission, leaderboard is re-calculated, such that each participant has a match against each other, and the bot that makes the most wins gets to the top. It includes includes an online in-browser IDE:…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →