I built a sub-500ms latency voice agent from scratch
I built a voice agent from scratch that averages ~400ms end-to-end latency (phone stop → first syllable). That’s with full STT → LLM → TTS in the loop, clean barge-ins, and no precomputed responses. What moved the needle: Voice is a turn-taking problem, not a transcription problem. VAD alone fails; you need semantic end-of-turn detection. The system reduces to one loop: speaking vs listening. The two transitions - cancel instantly on barge-in, respond instantly on end-of-turn - define the experience. STT → LLM → TTS must stream. Sequential pipelines are dead on arrival for natural…
I built a sub-500ms latency voice agent from scratch launched on March 2, 2026 with 570 votes, #15 of 23,380 launches that month and more than 99% of that year's launches.
In plain words
A voice agent built from scratch that achieves approximately 400 milliseconds of end-to-end latency while running speech-to-text, language model, and text-to-speech in parallel. The system prioritizes turn-taking detection over transcription accuracy, enabling users to interrupt naturally with instant cancellation and immediate responses. It uses streaming architecture throughout the pipeline and colocates components to minimize latency. The builder emphasizes that semantic understanding of conversation turns, rather than voice activity detection alone, creates a natural conversational experience.
written from the facts on this page · September 2026
From the sources
Nick Tikhonov's blog
This post made it to the Hacker News front page! I'm genuinely grateful - and pleasantly surprised by the response. If you're building an AI or voice product and want hands-on help, I do focused consulting. Learn more . I’ve spent the last six months working on a startup, building agent prototypes for one of the largest consumer packaged goods companies in the world. As part of that work, our team relied on off-the-shelf voice agent platforms to help the company operate more effectively. Though I can’t go into the business details, the technical takeaway was clear: voice agents are powerful, and there are brilliant off-the-shelf abstractions like Vapi and ElevenLabs that make spinning up…from ntik.me
In the maker’s words, at launch
I built a voice agent from scratch that averages ~400ms end-to-end latency (phone stop → first syllable). That’s with full STT → LLM → TTS in the loop, clean barge-ins, and no precomputed responses. What moved the needle: Voice is a turn-taking problem, not a transcription problem. VAD alone fails; you need semantic…
Does a similar job
all alternatives →
Open-source, native audio turn detection model2025 · github.com · ▲126Our goal with this project is to build a completely open source, state of the art turn detection model that can be used in any voice AI application. I've been experimenting with LLM voice conversations since GPT-4 was first released. (There's a previous front page Show HN about Pipecat, the open source voice AI orchestration framework I work on. [1]) It's been almost two years, and for most of that time, I've been expecting that someone would "solve" turn detection. We all built initial, pretty good 80/20 versions of turn detection on top of VAD (voice activity detection) models. And…
Conversational Replicas by Tavus2024 · tavus.io · ▲694Hey there, I’m Brian. I've been shipping conversational models over here at Tavus for the past two years. I want to tell you about our new audio-understanding model: Sparrow-2! It’s a new category of model and a unique new approach to conversational audio. Earlier this year we launched Sparrow-1, (at the time) our SoTA turn taking model. Since our Sparrow-1 launch, I’ve spent a lot of time listening to humans talking and trying to really understand how people know when to talk, when to listen, and when to wait. I’ve also been hunting down failure modes of the current SoTA models. And while…
Voice bots with 500ms response times2024 · fastvoiceagent.cerebrium.ai · ▲315Last year when GPT-4 was released I started making lots of little voice + LLM experiments. Voice interfaces are fun; there are several interesting new problem spaces to explore. I'm convinced that voice is going to be a bigger and bigger part of how we all interact with generative AI. But one thing that's hard, today, is building voice bots that respond as quickly as humans do in conversation. A 500ms voice-to-voice response time is just barely possible with today's AI models. You can get down to 500ms if you: host transcription, LLM inference, and voice generation all together in one place;…
Conversational AI 2.0 From ElevenLabs2025 · elevenlabs.io · ▲652Luke from ElevenLabs here. We just released Scribe Realtime v2, our new real-time Speech to Text model. We rebuilt it from the ground up around a new architecture that uses predictive transcription – it anticipates the next likely word before it’s spoken. That makes it both fast (around 150 ms latency) and highly accurate. Scribe v2 Realtime outperforms every low latency transcription model across 30 commonly used EU and Asian languages – 93.5% accuracy, compared to Gemini 2.5 Flash (91.4%), GPT-4o MiniTranscribe (90.7%), and Deepgram Nova 3 (85%). It supports 90+ languages including…
An open source framework for voice assistants2024 · github.com · ▲346I've been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational/real-time AI needs to solve on the way to building/shipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models/services, handling voice interruptions, swapping out different…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
Littlebird▲709Littlebird remembers everything across your meetings, messages, docs, and apps, then takes action so you can recall any detail and never lose context.
AI · Mar 2026 · littlebird.ai
