Cua-Bench – a benchmark for AI agents in GUI environments
Hey HN, we're excited to share Cua-Bench ( https://github.com/trycua/cua ), an open-source framework for evaluating and training computer-use agents across different environments. Computer-use agents show massive performance variance across different UIs—an agent with 90% success on Windows 11 might drop to 9% on Windows XP for the same task. The problem is OS themes, browser versions, and UI variations that existing benchmarks don't capture. The existing benchmarks (OSWorld, Windows Agent Arena, AndroidWorld) were great but operated in silos—different harnesses,…
In plain words
Cua-Bench is an open-source framework for evaluating and training AI agents that interact with graphical user interfaces across different operating systems and applications. It addresses the problem that computer-use agents perform inconsistently across different UI environments, operating systems, and browser versions. Unlike existing benchmarks that operate separately, Cua-Bench provides a unified testing framework that standardizes how agents are evaluated and can generate training data for reinforcement learning, rather than only measuring performance.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN, we're excited to share Cua-Bench ( https://github.com/trycua/cua ), an open-source framework for evaluating and training computer-use agents across different environments. Computer-use agents show massive performance variance across different UIs—an agent with 90% success on Windows 11 might drop to 9% on Windows XP for the same task. The problem is OS themes, browser versions, and UI variations that existing benchmarks don't capture. The existing benchmarks (OSWorld, Windows Agent Arena, AndroidWorld) were great but operated in silos—different harnesses, different formats, no standardized way to test the same agent across platforms. More importantly, they were evaluation-only. We needed environments that could generate training data and run RL loops, not just measure performance. Cua-Bench takes a different approach: it's a unified framework that standardizes environments across platforms and supports the full agent development lifecycle—benchmark, train, deploy. With Cua-Bench, you can: - Evaluate agents across multiple benchmarks with one CLI (native tasks + OSWorld + Windows Agent Arena adapters) - Test the same agent on different OS variations (Windows 11/XP/Vista, macOS themes, Linux, Android via QEMU) - Generate new tasks from natural language prompts - Create simulated environments for RL training (shell apps like Spotify, Slack with programmatic rewards) - Run oracle validations to verify environments before agent evaluation - Monitor agent runs in real-time with traces and screenshots All of this works on macOS, Linux, Windows, and Android, and is self-hostable. To get started: Install cua-bench: % pip install cua-bench Run a basic evaluation: % cb run dataset datasets/cua-bench-basic --agent demo Open the monitoring dashboard: % cb run watch <run_id> For parallelized evaluations across multiple workers: % cb run dataset datasets/cua-bench-basic --agent your-agent --max-parallel 8 Want to test across different OS variations? Just specify the environment: % cb run task slack_message --agent your-agent --env windows_xp % cb run task slack_message --agent your-agent --env macos_sonoma Generate new tasks from prompts: % cb task generate "book a flight on kayak.com" Validate environments with oracle implementations: % cb run dataset datasets/cua-bench-basic --oracle The simulated environments are particularly useful for RL training—they're HTML/JS apps that render across 10+ OS themes with programmatic reward verification. No need to spin up actual VMs for training loops. We're seeing teams use Cua-Bench for: - Training computer-use models on mobile and desktop environments - Generating large-scale training datasets (working with labs on millions of screenshots across OS variations) - RL fine-tuning with shell app simulators - Systematic evaluation across OS themes and browser versions - Building task registries (collaborating with Snorkel AI on task design and data curation, similar to their Terminal-Bench work) Cua-Bench is 100% open-source under the MIT license. We're actively developing it as part of Cua (https://github.com/trycua/cua), our Computer Use Agent SDK, and we'd love your feedback, bug reports, or feature ideas. GitHub: https://github.com/trycua/cua Docs: https://cua.ai/docs/cuabench Technical Report: https://cuabench.ai We'll be here to answer any technical questions and look forward to your comments!
Does the same job
all alternatives →
- DODemo of Agent Based Model on GPU with CUDA and OpenGL (Windows/Linux)2023 · github.com · ▲55
Demo of agent based model on GPU with CUDA and OpenGL (Windows/Linux) Agent instances on GPU memory Uses SSBO for instanced objects (with GLSL 450 shaders) CUDA OpenGL interops Renders with GLFW3 window manager Dynamic camera views in OpenGL (pan,zoom with mouse) Libraries installed using vcpkg (https://github.com/KienTTran/ABMGPU)
- ADAgent-desktop – Native desktop automation CLI for AI agentsMay 2026 · github.com · ▲99
I've been building computer-use tools for a while, and I quietly launched this about a month ago (122 Stars on GH). I figured it was worth sharing here. Over the last few months, a lot of computer-use agents have come out: Codex, Claude Code, CUA, and others. Most of them seem to work roughly like this: 1. Take a screenshot 2. Have the model predict pixel coordinates 3. Click x,y 4. Take another screenshot 5. Repeat That works, but it's slow, expensive in tokens, and fragile. If the UI shifts a few pixels, things break. And the model still doesn't know what any element actually is. But the…
- ΤBΤ³-Bench is out – can agents handle complex docs and live calls?Mar 2026 · ▲12
τ-Bench is an open benchmark for evaluating AI agents on grounded, multi-turn customer service tasks with verifiable outcomes. It's been great to see the community adopt it since launch — this is now the third iteration. With τ³-Bench, we're extending it to two new settings: knowledge-intensive retrieval and full-duplex voice. τ-Knowledge: agents must navigate ~700 interconnected policy documents to complete multi-step tasks. Best frontier model (GPT-5.2, high reasoning) hits ~25%. The surprising part: even when you hand the model the exact documents it needs, performance only reaches ~40%.…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, January 2026
the whole month →- IN
Hey HN! I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents. I didn't write a single line of code. Of course no-code doesn't mean no-engineering. This project took a lot more manual labor than I'd hoped! I wrote a deep dive on the workflow and some thoughts about the future of AI coding and creativity: http://cannoneyed.com/projects/isometric-nyc
AI · Jan 2026 · cannoneyed.com
Automatic AI-powered code reviews the moment you open a PR
Dev tools · Jan 2026 · kilo.ai
