Autotab Instruct – Claude Computer Use with Guardrails for Reliability
Hi HN, We’ve built a desktop app to create highly reliable AI agents that use a computer with mouse and keyboard. Until last week, we had tried many different approaches to open-ended agentic features but none of them had met our reliability bar. With Anthropic’s Computer Use this finally changed, and we just shipped a feature we’re calling Instruct. Instruct allows users to create agentic blocks as part of a larger Autotab skill that provides the structured logical flow to keep the automation on track. If you haven’t had a chance to try Computer Use yet, it is an impressive leap from the…
In plain words
Autotab Instruct is a desktop app that creates AI agents capable of automating computer tasks using mouse and keyboard control. It uses Anthropic's Computer Use technology combined with structured logical flow to maintain reliability. Users build agentic blocks within larger Autotab skills, giving the AI detailed instructions to keep automation on track rather than operating without guidance. The tool is designed for users who need dependable automation of complex workflows.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN, We’ve built a desktop app to create highly reliable AI agents that use a computer with mouse and keyboard. Until last week, we had tried many different approaches to open-ended agentic features but none of them had met our reliability bar. With Anthropic’s Computer Use this finally changed, and we just shipped a feature we’re calling Instruct. Instruct allows users to create agentic blocks as part of a larger Autotab skill that provides the structured logical flow to keep the automation on track. If you haven’t had a chance to try Computer Use yet, it is an impressive leap from the last generation of vision models (e.g. gpt4o struggles with relative positions, let alone coordinates). At the same time, it is still not good enough to be given a prompt and let loose. One of the big surprises to us early on was just how much intent specification is required for most real world workflows to run reliably. What looks at first like a simple form filling task usually turns out to have dozens of edge cases and super specific, hidden rules. Even human employees need to be shown how to perform these tasks, and then refined with question-asking + feedback over time. We wanted to build a tool for specifying intent, and iterating with the model to make it reliable enough for real work. - Automations run on top of an action scaffold, which works kind of like a very fuzzy programming language with strict types. This gives the model a high level plan that guides execution, and makes it easy to break out discrete steps to get the reliability you need. (Interestingly, this has also proven useful not just for the agent, but also for the human trying to create, verify and edit the automation.) - When the model is unsure it asks for clarification. For example, if you are in editing mode and the model thinks that an element looks meaningfully different than before, it will ask you to verify that it is the same element. - The agent has access to a memory system that lets it recall information from past runs as well as instructions and feedback from the user. Here's a short video of Autotab Instruct in action: https://www.loom.com/share/ccf4e9d8c798450da3324a6cff024971?.... There are a few more demos at https://twitter.com/autotabai/status/1852393973165199425 a75f06f82cab521bc78672ed35d85e8a. We’d love to hear what you think!
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, November 2024
the whole month →
- IB
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
