nowfound

Life & fun · December 21, 2022

HT

How to use ChatGPT+ARKit to script experiences with natural language

Video demonstration: https://youtu.be/bJFa5HOmhl4 I created a demo involving ChatGPT, OpenAI's Whisper running on-device for speech-to-text (because why not?), and ARKit. By defining a custom JavaScript environment with JavaScriptCore and describing it to ChatGPT, I was able to get it to produce functioning scripts that instantiate and manipulate 3D objects. Sketchfab is used to import 3D assets on the fly. Google Poly had a much better catalog of assets for this sort of thing but sadly, it's no longer available. The README.md describes how it all works but the gist of it is…

Visit github.com ↗Alternatives

In plain words

This project demonstrates how to create augmented reality experiences using natural language commands. It combines ChatGPT with on-device speech recognition via OpenAI's Whisper and Apple's ARKit to let users describe 3D scenes that the system then scripts and renders. A custom JavaScript environment interprets user prompts and generates code to instantiate and manipulate 3D objects pulled from Sketchfab. It's designed for developers interested in natural language interfaces for AR applications.

written from the facts on this page · September 2026

From the sources

In the maker’s words, at launch

Video demonstration: https://youtu.be/bJFa5HOmhl4 I created a demo involving ChatGPT, OpenAI's Whisper running on-device for speech-to-text (because why not?), and ARKit. By defining a custom JavaScript environment with JavaScriptCore and describing it to ChatGPT, I was able to get it to produce functioning scripts that instantiate and manipulate 3D objects. Sketchfab is used to import 3D assets on the fly. Google Poly had a much better catalog of assets for this sort of thing but sadly, it's no longer available. The README.md describes how it all works but the gist of it is that user prompts are wrapped in a larger prompt that describes the environment (i.e., which functions are available to use and how they work) and constraints. The code is then executed directly. Assets for objects are fetched as needed by searching Sketchfab.

More life & fun this month

the category →
  • TL
    The load-bearing vocabulary of Claude▲703

    Life & fun · 10d ago · louisabraham.github.io

  • iPhone app takes simultaneous images from 2 lenses, fuses into 1 photo▲329

    Photosynthesis fires two of your iPhone

    Life & fun · 28d ago · photosynthesis.camera

  • SoloUno▲310

    Take control of hair pulling, nail biting & skin picking

    Life & fun · 28d ago · solouno.io

  • Scroll through all 43252003274489856000 Rubik's Cube states▲295

    Scroll through all 43,252,003,274,489,856,000 reachable Rubik's Cube permutations.

    Life & fun · 26d ago · everycube.alen.is

  • Expeditione▲265

    The Interactive 3D Encyclopedia

    Life & fun · 21d ago · expeditione.fun

  • Eigendrum - Draw any shape and hear what it sounds like as a drum▲243

    Hi HN, I built Eigendrum, a web tool that solves the 2D wave equation for arbitrary shapes so you can hear what they sound like as drums. How it works: * Solves -∇²u = λu using finite element analysis (Kφ = λMφ) on a triangle mesh. * Validated to <0.1% error against closed-form solutions for circles (Bessel zeros) and rectangles. * Sound model factors in strike location, Rayleigh damping, and mallet width. * Includes Kac drums I & II to demonstrate identical sound spectra from different geometries. * No frameworks, build steps, or dependencies. Repo and tests:…

    Life & fun · 26d ago · baselashraf81.github.io

Launched alongside, December 2022

the whole month →