I built a new software primitive. It replaces AI screenshot agents
I built a Rust binary (~700KB, two dependencies) that snaps onto any Windows application, continuously reads its UI Automation accessibility tree into a SQLite database, and lets external processes control the app via SQL INSERT. The core insight: every GUI application already describes itself as structured text through the accessibility layer (built for screen readers since 1997). Every major AI lab is taking screenshots of this and running vision models on it. DirectShell just reads the text directly. What it does: - Reads every button, field, menu item into a queryable SQLite DB…
What it does
In the maker’s words, at launch
I built a Rust binary (~700KB, two dependencies) that snaps onto any Windows application, continuously reads its UI Automation accessibility tree into a SQLite database, and lets external processes control the app via SQL INSERT. The core insight: every GUI application already describes itself as structured text through the accessibility layer (built for screen readers since 1997). Every major AI lab is taking screenshots of this and running vision models on it. DirectShell just reads the text directly. What it does: - Reads every button, field, menu item into a queryable SQLite DB (refreshed every 500ms) - Generates multiple output formats: full DB, interactive elements list, LLM-optimized snapshots (50-200 tokens vs 1,200-5,000 for a screenshot) - Controls apps via 5 action types: click by element name, set text via UIA ValuePattern, type character-by-character, send key combos, scroll - Includes an MCP server so Claude/GPT can use it directly Day 1 demo: filled 360 Google Sheets cells in 90 seconds, read and replied to a Claude.ai conversation cross-app, wrote to Notepad instantly. No screenshots, no vision model, no coordinate guessing. Limitations (honest): built in 8.5 hours, single-app scope, Chromium apps need a 4-phase activation hack to expose their tree, accessibility quality varies by app. AGPL-3.0. Demo: https://youtu.be/nvZobyt0KBg Full technical paper: in the repo under Dokumentation/ and on https://dev.to/tlrag/i-built-a-new-software-primitive-in-85-...
Does the same job
all alternatives →- ADAgent-desktop – Native desktop automation CLI for AI agentsMay 2026 · github.com · ▲99
I've been building computer-use tools for a while, and I quietly launched this about a month ago (122 Stars on GH). I figured it was worth sharing here. Over the last few months, a lot of computer-use agents have come out: Codex, Claude Code, CUA, and others. Most of them seem to work roughly like this: 1. Take a screenshot 2. Have the model predict pixel coordinates 3. Click x,y 4. Take another screenshot 5. Repeat That works, but it's slow, expensive in tokens, and fragile. If the UI shifts a few pixels, things break. And the model still doesn't know what any element actually is. But the…
TestSprite 3.0May 2026 · testsprite.com · ▲472Let a fleet of parallel agents test your app in minutes
- IAIota – a text editor written in Rust2014 · github.com · ▲221
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
