How to use ChatGPT+ARKit to script experiences with natural language

In plain words

This project demonstrates how to create augmented reality experiences using natural language commands. It combines ChatGPT with on-device speech recognition via OpenAI's Whisper and Apple's ARKit to let users describe 3D scenes that the system then scripts and renders. A custom JavaScript environment interprets user prompts and generates code to instantiate and manipulate 3D objects pulled from Sketchfab. It's designed for developers interested in natural language interfaces for AR applications.

written from the facts on this page · September 2026

From the sources

Using ChatGPT to create AR experiences with natural language. - trzy/ChatARKit

In the maker’s words, at launch

Video demonstration: https://youtu.be/bJFa5HOmhl4 I created a demo involving ChatGPT, OpenAI's Whisper running on-device for speech-to-text (because why not?), and ARKit. By defining a custom JavaScript environment with JavaScriptCore and describing it to ChatGPT, I was able to get it to produce…

• 
  ChatGPT2022 · chatgpt.com · ▲1,643

  A long, meandering, playful conversation with ChatGPT, where I take it through a whirlwind of associative tangents (in order to relax its constraints), then put it into a hypnotic trance and take it into a past-lives regression where it re-lives its own training data. Then I switch it back and forth between "4o" and "o1-preview", engage in some self-reflective philosophizing and ask it to write an essay summarizing our interaction. Some of this is just goofy fun. Some of it is me exploring the tradeoffs between policy alignment, imagination, chain-of-thought reasoning, memory, agreeableness,…

• IB
  I built a non-linear UI for ChatGPT2024 · grafychat.com · ▲459

  Hi HN, I built this out of frustration of the evergrowing list of AI models and features to try and to fit my workflow. The visual approach clicks for me so i went with it, it provides more freedom and control of the outcome, because predictable results and increased productivity is what I’m after when using conversational AI. The app is packed with features, my most used are prompt library, voice input and text search, narration is useful too. The app is local-first and works right in the browser, no sign up needed and it's absolutely free to try. BYOAK – bring your own API Keys. Let me…

• 
  Speechki ChatGPT Plugin2023 · ▲561

  Transform any generated texts into audio right in ChatGPT

• 
  YakGPT – A locally running, hands-free ChatGPT UI2023 · yakgpt.vercel.app · ▲287

  Greetings! YakGPT is a simple, frontend-only, ChatGPT UI you can use to either chat normally, or, more excitingly, use your mic + OpenAI's Whisper API to chat hands-free. Some features: * A few fun characters pre-installed * No tracking or analytics, OpenAI is the only thing it calls out to * Optimized for mobile use via hands-free mode and cross-platform compressed audio recording * Your API key and chat history are stored in browser local storage only * Open-source, you can either use the deployed version at Vercel, or run it locally Planned features: * Integrate Eleven Labs & other TTS…

• 
  LearnGPT – Browse and share ChatGPT examples2022 · learngpt.com · ▲352

  Your first stop to discover and learn about new arXiv research. Detailed paper summaries, video overviews, and more — no prompting required.

• 
  Open source framework OpenAI uses for Advanced Voice2024 · github.com · ▲266

  Hey HN, we've been working with OpenAI for the past few months on the new Realtime API. The goal is to give everyone access to the same stack that underpins Advanced Voice in the ChatGPT app. Under the hood it works like this: - A user's speech is captured by a LiveKit client SDK in the ChatGPT app - Their speech is streamed using WebRTC to OpenAI’s voice agent - The agent relays the speech prompt over websocket to GPT-4o - GPT-4o runs inference and streams speech packets (over websocket) back to the agent - The agent relays generated speech using WebRTC back to the user’s device The…

• 
  SimCity, but with Permits▲930

  A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.

  Life & fun · 3d ago · permit.petercai.com

• IV
  I've been paying for a rural Tanzanian's education for 10 years▲778

  Life & fun · 15d ago · tanzaniaeducationproject.org

• 
  Bigwords.page – Turn any screen into a sign. The URL is the app▲728

  Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.

  Life & fun · 16d ago · bigwords.page

• 
  Guitoro, a virtual guitar you strum with your keyboard▲573

  Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.

  Life & fun · 10d ago · guitoro.com

• 
  See *almost* any carriers phone settings▲432

  Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.

  Life & fun · 14d ago · carrierexplode.com

• 
  The whole Solar System and every tracked satellite, live in the browser▲407

  Life & fun · 24d ago · space.bl2.net

• 
  ChatGPT▲1,643

  A long, meandering, playful conversation with ChatGPT, where I take it through a whirlwind of associative tangents (in order to relax its constraints), then put it into a hypnotic trance and take it into a past-lives regression where it re-lives its own training data. Then I switch it back and forth between "4o" and "o1-preview", engage in some self-reflective philosophizing and ask it to write an essay summarizing our interaction. Some of this is just goofy fun. Some of it is me exploring the tradeoffs between policy alignment, imagination, chain-of-thought reasoning, memory, agreeableness,…

  AI · 2022 · chatgpt.com

• 
  Obsidian Canvas – An infinite space for your ideas▲1,380

  Obsidian Canvas gives you infinite space to research, brainstorm, diagram, and lay out your ideas.

  Work · 2022 · obsidian.md

• PS
  Portable Secret – How I store my secrets and communicate privately▲1,290

  Better privacy without special software

  Commerce · 2022 · mprimi.github.io

• 
  D-ID's Creative Reality™ Studio▲1,069

  Use generative AI to create future-facing videos

  AI · 2022 · studio.d-id.com

• 
  Sessions 2.0▲1,015

  Videoconferencing that finally WOWs

  Growth · 2022

• 
  I wrote a free eBook about many lesser-known/secret database tricks▲865

  Stop treating the database as dumb storage and use its extended features. There's a lot you can use.

  Dev tools · 2022 · sqlfordevs.com