Tarsier – Vision utilities for web interaction agents
Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be. Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance. Here's the…
In plain words
Tarsier is a tool that enables language models to understand webpage visual structure through text-based representations rather than images. It converts screenshots into whitespace-structured text using OCR and tags interactive elements with IDs for agents to reference. The tool is designed for developers building web interaction agents and AI codegen systems. According to its creator, text-only models using Tarsier's output outperform multimodal models using screenshots on web automation tasks.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be. Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance. Here's the Tarsier pipeline in a nutshell: 1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot 2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo) 3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai). By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems! https://reworkd.ai/careers
Does the same job
all alternatives →
Google Gemma 4 12BJun 2026 · blog.google · ▲310Run multimodal AI locally with an encoder-free architecture
AgentmemoryMay 2026 · agent-memory.dev · ▲322Persistent memory for Claude Code, Codex & coding agents
- MRMorph Reflexes – Multi-head classifiers for agent tracesJun 2026 · ▲20
The most common failures for production agents are behavioral: looping, reasoning leakage, user frustration, and more. Using a frontier model like GPT or Sonnet to judge every turn is too expensive and slow to run at scale. To solve this, we built Reflexes: semantic signals from agent traces, served fast and cheap over API. Built on custom kernels and a custom inference engine forked from vLLM. Under the hood, it is a small LLM architected around multi-head inference. Small models need to be trained for specific tasks, but running 50 separate small models on the same input for 50 tasks makes…
/show-me: agent skill for compact visual representations25d ago · humanlayer.com · ▲12Was so sick of reading walls of codex/claude prose, in markdown plans and just in the chat, that I started playing with ideas that force coding agents to display information differently. The human visual cortex is an amazing thing, and getting coding agents to let me use it has been pretty nice so far. Been iterating on this a lot internally for the last few months, polishing and mostly removing stuff.
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
