Turn a video of an app into a functional prototype with Claude Opus
Hey everyone, I’m the maintainer of the popular screenshot-to-code repo on Github (46k+ stars). When Claude Opus was released, I thought to myself what if you could send in a video of yourself using a website or app, would the LLM be able to build it as a functional prototype? To my surprise, it worked quite well. Here are two examples: * In this video, you can see the AI replicating Google with auto-complete suggestions and a search results page (failed at putting the results on a separate page). https://streamable.com/s24pq6 * Here, we show it a multi-step form…
In plain words
This tool converts videos of app or website interactions into functional prototypes using Claude Opus. Users submit a video of themselves using an application, and the AI extracts frames to analyze the interface and behavior, then generates working code that replicates the demonstrated functionality. It successfully recreates features like search interfaces with autocomplete and multi-step forms. Built by the maintainer of the popular screenshot-to-code repository, it's designed for developers and designers who want to quickly prototype interfaces based on video demonstrations.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey everyone, I’m the maintainer of the popular screenshot-to-code repo on Github (46k+ stars). When Claude Opus was released, I thought to myself what if you could send in a video of yourself using a website or app, would the LLM be able to build it as a functional prototype? To my surprise, it worked quite well. Here are two examples: * In this video, you can see the AI replicating Google with auto-complete suggestions and a search results page (failed at putting the results on a separate page). https://streamable.com/s24pq6 * Here, we show it a multi-step form (https://tally.so/templates/online-quiz/V3qOnk) and ask Claude to re-create it. It does a really good job! https://streamable.com/gstsgn The technical details: Claude Opus only allows you to send a max of 20 images so 20 frames are extracted from the video, and passed along with a prompt that uses a lot of Claude-specific techniques such as using XML tags and pre-filling an assistant response. In total, 2 passes are performed with the second pass instructing the AI improve on the first attempt. More passes might help as well. While I think the model has Google.com memorized but for many other multi-page/screen apps, it tends to work quite well. You can try it out by downloading the Github repo and setting up a Anthropic API key in backend/.env Be warned that one creation/iteration (with 2 passes) can be quite expensive ($3-6 dollars).
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, March 2024
the whole month →
- 3Y
Life & fun · 2024 · github.com
Microlaunch▲1,116Launch and get feedback on both the idea and product
Dev tools · 2024 · microlaunch.net
