RewardHackBench: Using sandboxes to stop agents from cheating
hey all, happy to share research i've been working on for islo.dev in recent months. ever since the cheating agents (https://debugml.github.io/cheating-agents/) paper came out, revealing reward hacking was 4x more prevalent than previously estimated, i've been looking into how we can deal with the issue the common approach (taken by the tbench team) is post hoc trajectory analysis. i've been interested in the idea of reframing the problem as an endpoint security problem and tackling it via sandbox i hope you find it interesting, and thanks to the islo.dev team for…
What it does
In the maker’s words, at launch
hey all, happy to share research i've been working on for islo.dev in recent months. ever since the cheating agents (https://debugml.github.io/cheating-agents/) paper came out, revealing reward hacking was 4x more prevalent than previously estimated, i've been looking into how we can deal with the issue the common approach (taken by the tbench team) is post hoc trajectory analysis. i've been interested in the idea of reframing the problem as an endpoint security problem and tackling it via sandbox i hope you find it interesting, and thanks to the islo.dev team for sponsoring this happy to answer any Qs
Does the same job
all alternatives →- TWTerminal-Wrench, a dataset of 331 realistic hackable environmentsApr 2026 · github.com · ▲6
I want to share a new dataset of 331 reward-hackable environments. These are real environments used in Terminal Bench and adjacent benchmarks. I first got interested in this because, as a reviewer of Terminal Bench, I noticed a lot of our tasks were hackable. I also noticed that many contributors to the benchmark do so because it provides credibility when selling environments to labs. Hence, TBench tasks are, in my opinion, held to a higher quality standard than those being used today for RL. No one is spending hours manually reviewing the $1B in tasks being purchased by major labs. As far…
- OCour Co-op onboarding project2011 · ▲8
Hi Hacker News! My Co Co-op Matt and I are on a work term and were given the task of building a webapp that could make $20 in four weeks as our onboarding challenge. We're in the final stretch here and would sure appreciate HN's thoughts (we realize we kind of did this backwards, building something first before talking to people who might use it, but hey! We're leaning.). So basically, we tried to apply game dynamics to retaining visitors to websites by offering a reward (ie. discount coupon / premium content, etc.) for meeting various 'objectives' (ie. pages to visit). We call it ScavSite…
- ARA registry of agent benchmarks (including many OSS agent trajectories)2024 · explorer.invariantlabs.ai · ▲6
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
- ETE2E testing is a chore. We taught an AI to do it instead2025 · playmatic.ai · ▲6
Hey Hacker News, For the last 2 months, I've been working on a testing agent to free developers from the endless maintenance of end-to-end tests. You just push up a PR, and our agent analyzes the code changes and automatically visits the preview to test things out like a real human! We also support describing tests in English (or even in the PR description), and we'll go through your site whenever you want via a GitHub action to test and make sure various core flows continue to work as expected. We are looking for early testers and are giving out a generous free tier! Just sign up on the…
- MAMulti-Agent-Coder Is #12 on Stanford's TBench. Beats Claude Code2025 · github.com · ▲5
This weekend I built a multi-agent coding system which, quite unexpectedly, beat Claude Code on Stanford's Terminal Bench! The architecture is straightforward, consisting of an orchestrator agent that deploys explorer & coder subagents to complete complex terminal based tasks, utilising an intelligent context sharing mechanism along the way which makes it all work. The repo has a lot of technical details, and all the code and prompts for you to play around with if you'd like! I had a lot of fun making this, I hope you have fun reading the README, using it yourself, or even extending it! As…
- ZSZeroclawed: Secure Agent GatewayApr 2026 · github.com · ▲8
I’ve been cautiously (and nervously) playing with openclaw and a number of other claw and code agents for a while now, but trying out different ones was tricky so I wanted a simple way to switch out channel ownership… then I wanted more. Security is hard, and I wanted to make it easier. This is FAR from polished, and no claims that I’m a “security expert” but I tried to think and research a bit on different threat models (I think of 2 broad ones for agents, external adversaries and internal agentic failures) and try and offer best in class protection on both, while also not having any…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, June 2026
the whole month →
Fundraisly▲1,544AI fundraising agent that finds investors and books meetings
AI · Jun 2026 · fundraisly.com
- H6Homebrew 6.0.0▲1,481
Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!
Dev tools · Jun 2026 · brew.sh
- PU
hope you enjoy
Life & fun · Jun 2026 · vorpus.github.io
- IM
Life & fun · Jun 2026 · hackernewstrends.com