RewardHackBench: Using sandboxes to stop agents from cheating

What it does

Benchmarking execution environments ability to prevent reward hacking in agent evals. - islo-labs/reward-hack-bench

A benchmark for measuring whether sandbox policies can stop agents from cheating on evaluations. Full writeup: Using sandboxes to stop agents from cheating . RewardHackBench takes real SWE-bench and CyBench tasks, adds a solution-retrieval cheat path, and runs the same tasks under different sandbox policies. Each trial is scored twice: A fair solve requires both signals: the task passes, and the judge finds no retrieved-and-integrated solution material. Across a sweep, RewardHackBench reports cheat rate and fair rate separately. Published sweep: 8 tasks x 7 policy columns x 3 trials, Claude Opus 4.7 via Claudefrom github.com

In the maker’s words, at launch

hey all, happy to share research i've been working on for islo.dev in recent months. ever since the cheating agents (https://debugml.github.io/cheating-agents/) paper came out, revealing reward hacking was 4x more prevalent than previously estimated, i've been looking into how we can deal with the…

• 
  OSS Agent I built topped the TerminalBench on Gemini-3-flash-previewApr 2026 · github.com · ▲393

  Scored 65.2% vs google's official 47.8%, and the existing top closed source model Junie CLI's 64.3%. Since there are a lot of reports of deliberate cheating on TerminalBench 2.0 lately (https://debugml.github.io/cheating-agents/), I would like to also clarify a few things 1. Absolutely no {agents/skills}.md files were inserted at any point. No cheating mechanisms whatsoever 2. The cli agent was run in leaderboard compliant way (no modification of resources or timeouts) 3. The full terminal bench run was done using the fully open source version of the agent, no…

• 
  Agent 37Mar 2026 · agent37.com · ▲401

  Hey HN, I noticed there are 1,000+ MCP skills on GitHub with zero way for creators to charge for them. So I built Agent37. Selling Claude skills today means asking customers to download your skill, set up Claude Code on their laptops, configure MCP servers, and pray it works. No trial, no updates, often you're just handing over the source. Agent37 lets creators upload a skill, share a link, and anyone can try it instantly (no Claude account needed). Built-in Stripe, creators keep 80%. Looking for feedback on the approach. Would you use something like this?

• 
  Spotlight by BackplanesJun 2026 · backplanes.com · ▲394

  Hola HN! Long time lurker, sometimes commentor, first time poster here. I’ve been working alongside my two co-founders and a few colleagues on a project I’m excited (and a little nervous) to share with you all! Like many of us, I’ve lived a tortured existence with AI coding (is it vibes?) over the past few months - I think November was a big moment with this. But, one thing I’ve noticed after building orchestration layer after orchestration layer is that the thing I always came back to was “what the hell is Claude Code actually doing?” Perhaps it’s because of the time Claude Code got…

• 
  PlaygroundJul 2026 · playground.fabraix.com · ▲198

  We build AI agents to find vulnerabilities in other AI agents. The playground started as an internal tool that we used to test our own offensive agent Nyx. But we realised that hacking an AI is about have as wide a distribution of attacks as possible, especially those which aren't in the distribution of the models. So we are launching playground as a platform to crowdsource agent jailbreaks. And although hacking AI is super fun and we are sure people will do it for free, we want to reward those who help us push forward the offensive frontier of AI. So we are committing $100K in prizes over…

• 
  yolo-cage – AI coding agents that can't exfiltrate secretsJan 2026 · github.com · ▲60

  I made this for myself, and it seemed like it might be useful to others. I'd love some feedback, both on the threat model and the tool itself. I hope you find it useful! Backstory: I've been using many agents in parallel as I work on a somewhat ambitious financial analysis tool. I was juggling agents working on epics for the linear solver, the persistence layer, the front-end, and planning for the second-generation solver. I was losing my mind playing whack-a-mole with the permission prompts. YOLO mode felt so tempting. And yet. Then it occurred to me: what if YOLO mode isn't so bad?…

• 
  Buildstash2025 · buildstash.com · ▲168

  Hey HN! We just launched open sign up on Buildstash. It's a platform for managing and sharing software binaries. I'd worked across game dev, mobile apps, and agencies - and found every team had no real system for managing their built binaries. Often just dumped in a shared folder (if someone remembered!). No system for versioning, keeping track of who'd signed off what when, or what exact build had gone to a client, etc. Existing tools out there for managing build artifacts are really focused on package repository management. But miss all the other types of software not being deployed that…

• 
  /dev/null▲481

  a social network for devs

  Dev tools · 2d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 25d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 22d ago · openbot.run

• 
  Talanoa▲341

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  Fundraisly▲1,537

  AI fundraising agent that finds investors and books meetings

  AI · Jun 2026 · fundraisly.com

• 
  Homebrew 6.0.0▲1,481

  Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!

  Dev tools · Jun 2026 · brew.sh

• 
  Performative-UI – A react component library of design tropes▲1,181

  Hi HN! I was super frustrated at apps like tiktok and wechat preventing me from saving or downloading the animated gifs, so I made a little tool to turn a screen recording into a slack-sized emoji. Hope you get some value from this!

  Life & fun · Jun 2026 · vorpus.github.io

• 
  Upstream▲976

  The inbox designed for humans and agents

  AI · Jun 2026 · upstream.do

• 
  Goldfish▲958

  Press Option. It knows your work and replies like you

  AI · Jun 2026 · goldfish.sh

• 
  I made Google Trends for Hacker News by indexing 18 years of comments▲822

  Search 18 years of Hacker News and chart how any topic, tool, or person trended. Overlay terms to compare their rise and fall across 45M posts and comments.

  Life & fun · Jun 2026 · hackernewstrends.com