Open-source turn detection model for voice AI
Hey HN, it’s Russ - cofounder of LiveKit. An open source stack for building realtime AI applications. We’re sharing our first homegrown AI model for turn detection. Here’s a live demo: https://cerebras.vercel.app/ Voice AI has come a long way in the last year. We now have end-to-end systems that can generate a response to user input in 300-500ms — human level speeds! As latency reduces, a common problem that surfaces is the LLM responds too quickly. Any time there’s a short pause in a user’s speech, it ends up interrupting them. This is largely due to how voice AI applications…
What it does
In the maker’s words, at launch
Hey HN, it’s Russ - cofounder of LiveKit. An open source stack for building realtime AI applications. We’re sharing our first homegrown AI model for turn detection. Here’s a live demo: https://cerebras.vercel.app/ Voice AI has come a long way in the last year. We now have end-to-end systems that can generate a response to user input in 300-500ms — human level speeds! As latency reduces, a common problem that surfaces is the LLM responds too quickly. Any time there’s a short pause in a user’s speech, it ends up interrupting them. This is largely due to how voice AI applications perform “turn detection” — that is, figuring out when the user has finished speaking and when the model can run inference and respond. Pretty much everyone uses a signal processing technique called voice activity detection (VAD). In a nutshell, it figures out when the audio signal switches from speech to silence and then triggers an end of turn once a configurable amount of silence has transpired. One obvious delta between VAD and how humans do turn detection is we also consider the content of speech (i.e. what someone says). These past few months, we’ve been working on an open weights, content-aware turn detection model for voice AI applications. It was fine-tuned from SmolLM v2 on text, runs on CPU (currently takes 50ms for inference), and uses speech transcriptions as input to predict when a user has completed a thought (also called an “utterance”). Since it was trained on text, notably it works well for pipeline-based architectures (i.e. STT ⇒ LLM ⇒ TTS). We use this model together with VAD to make better predictions about whether a user is done speaking. Here’s some demos -- - Podcast interview: https://youtu.be/EYDrSSEP0h0 - Ordering food: https://youtu.be/fcr8Y-3c4E0 - Providing shipping address: https://youtu.be/2pQWvd6xozw - Customer support: https://youtu.be/YoSRg3ORKtQ In our testing we’ve found: - 85% reduction in unintentional interruptions - 3% false positives (where the user is done speaking, but the model thinks they aren’t) In practice, we still have work to do. We currently delay inference if the model predicts a < 15% chance the user is done speaking. This threshold misses a bunch of middle-of-the-pack probabilities. Next steps are improving the model accuracy, tuning performance, and expanding to support more languages (only supports English rn). Separately, we’re starting to explore an audio-based model that considers not just what someone says but how they say it, which can be used with natively multimodal models like GPT-4o that directly process and generate audio. Code here: https://github.com/livekit/agents/tree/main/livekit-plugins/... Let us know what you think!
Does the same job
all alternatives →
- RTReal-time AI voice conversations with <0.5s latency2024 · dashboard.avaturn.live · ▲6
Avaturn.live is a voice-to-voice AI assistant that lets you speak to an avatar in real time. We managed to reach <0.5 sec response time so that the conversation with an avatar feels more natural. Also what do you think about our lip sync? We were struggling a lot to reach our current level. Initial use case we're exploring: Automated product demos/sales, where the AI avatar can showcase features and answer questions in real-time. Demo: https://dashboard.avaturn.live/demo You can try talking to an avatar about anything
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
