New Benchmark from SWE-bench team is 0% solved
ProgramBench evaluates whether language models can rebuild programs from scratch.
New Benchmark from SWE-bench team is 0% solved launched on May 5, 2026 with 24 votes, #867 of 20,538 launches that month and more than 90% of that year's launches.
In plain words
New Benchmark is a tool from the SWE-bench team designed for software engineers and researchers working on code generation and problem-solving tasks. It provides a testing framework with unsolved problems that challenge AI systems and developers to write code solutions. The benchmark launched in May 2026 and appears aimed at those evaluating or improving software engineering automation capabilities.
written from the facts on this page · September 2026
From the sources
Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior. In each task, the agent receives an executable and its documentation, and it must re-implement the given executable. It does not get access to any of the executable's source code, it cannot de-compile the executable , and cannot use the internet . There are 200 tasks in total covering different program complexities, ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler , FFmpeg , and SQLite . The agent must choose a language, design the architecture, write all source code, and…from programbench.com
Does a similar job
all alternatives →
LocalScore – Local LLM Benchmark2025 · localscore.ai · ▲124Hey Folks! I've been building an open source benchmark for measuring local LLM performance on your own hardware. The benchmarking tool is a CLI written on top of Llamafile to allow for portability across different hardware setups and operating systems. The website is a database of results from the benchmark, allowing you to explore the performance of different models and hardware configurations. Please give it a try! Any feedback and contribution is much appreciated. I'd love for this to serve as a helpful resource for the local AI community. For more check out: - Website:…
- RSRandomly switching between LMs at every step boosts SWE-bench score2025 · swebench.com · ▲5
Hello from the SWE-bench/SWE-agent team at Princeton/Stanford. When we created the SWE-bench benchmark in 2023 from hundreds of real-life GitHub issues/pull requests, the highest score was just a couple of percent. The tasks were so challenging for LMs, that most people didn't even want to work on them. Half a year later, SWE-agent showed that the early 2024 LMs were actually good enough to resolve up to 20% of the GitHub issues in the benchmark. This kicked off a whole wave of coding agents. Back then, developing agents was all about working around tons of silly behavior from…
Stage – Putting humans back in control of code reviewApr 2026 · stagereview.app · ▲130Hey HN! We're Charles and Dean, and we're building Stage: a code review tool that guides you through reading a PR step by step, instead of piecing together a giant diff. Here's a demo video: https://www.tella.tv/video/stage-demo-1pph. You can play around with some example PRs here: https://stagereview.app/explore. Teams are moving faster than ever with AI these days, but more and more engineers are merging changes that they don't really understand. The bottleneck isn't writing code anymore, it's reviewing it. We're two engineers who got frustrated with…
More work this month
the category →
Endless lofi beats composed live in your browser, over pixel-art nights in Paris, Tokyo, New York, Hong Kong, Sydney, San Francisco and more. Free, no sign-up.
Work · 25d ago · loficities.com
Smooth Recorder▲257Mac screen recordings that look edited when you hit stop
Work · 19d ago · smoothrecorder.com
Work · 13d ago · readrare.com
Source: Show HN post ↗
Launched alongside, May 2026
the whole month →
Parallel agents, diff reviewer, and multi-model comparisons
Dev tools · May 2026 · kilo.ai
Hey HN, Henry here from Cactus. We open-sourced Needle, a 26M parameter function-calling (tool use) model. It runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices. We were always frustrated by the little effort made towards building agentic models that run on budget phones, so we conducted investigations that led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly (match query to tool name, extract argument values, emit JSON), not reasoning. Cross-attention…
Life & fun · May 2026 · github.com
🌱 Private, quiet space for thinking. Simple app for .md files. - zakirullin/files.md
Dev tools · May 2026 · github.com