Multimodal perception system for real-time conversation
I work on real-time voice/video AI at Tavus and for the past few years, I’ve mostly focused on how machines respond in a conversation. One thing that’s always bothered me is that almost all conversational systems still reduce everything to transcripts, and throw away a ton of signals that need to be used downstream. Some existing emotion understanding models try to analyze and classify into small sets of arbitrary boxes, but they either aren’t fast / rich enough to do this with conviction in real-time. So I built a multimodal perception system which gives us a way to encode visual…
Multimodal perception system for real-time conversation launched on February 10, 2026 with 54 votes, #570 of 16,628 launches that month and more than 95% of that year's launches.
In plain words
This multimodal perception system processes real-time voice and video during conversations, capturing visual and audio signals that traditional text-based systems discard. Rather than forcing emotions into predefined categories, it encodes these signals and translates them into natural language descriptions via a fine-tuned language model. Built for developers working with conversational AI, it enables agents to genuinely perceive both sight and sound, with results accessible through an OpenAI-compatible tool interface for live interactions.
written from the facts on this page · September 2026
From the sources
Experience Raven-1, Tavus
In the maker’s words, at launch
I work on real-time voice/video AI at Tavus and for the past few years, I’ve mostly focused on how machines respond in a conversation. One thing that’s always bothered me is that almost all conversational systems still reduce everything to transcripts, and throw away a ton of signals that need to be used…
Does a similar job
all alternatives →
Conversational AI 2.0 From ElevenLabs2025 · elevenlabs.io · ▲652Luke from ElevenLabs here. We just released Scribe Realtime v2, our new real-time Speech to Text model. We rebuilt it from the ground up around a new architecture that uses predictive transcription – it anticipates the next likely word before it’s spoken. That makes it both fast (around 150 ms latency) and highly accurate. Scribe v2 Realtime outperforms every low latency transcription model across 30 commonly used EU and Asian languages – 93.5% accuracy, compared to Gemini 2.5 Flash (91.4%), GPT-4o MiniTranscribe (90.7%), and Deepgram Nova 3 (85%). It supports 90+ languages including…
Conversational Replicas by Tavus2024 · tavus.io · ▲694Hey there, I’m Brian. I've been shipping conversational models over here at Tavus for the past two years. I want to tell you about our new audio-understanding model: Sparrow-2! It’s a new category of model and a unique new approach to conversational audio. Earlier this year we launched Sparrow-1, (at the time) our SoTA turn taking model. Since our Sparrow-1 launch, I’ve spent a lot of time listening to humans talking and trying to really understand how people know when to talk, when to listen, and when to wait. I’ve also been hunting down failure modes of the current SoTA models. And while…
Real-time AI Voice Chat at ~500ms Latency2025 · github.com · ▲524Have a natural, spoken conversation with AI! Contribute to KoljaB/RealtimeVoiceChat development by creating an account on GitHub.
- ARA real time AI video agent with under 1 second of latency2024 · ▲455
Hey it’s Hassaan & Quinn – co-founders of Tavus, an AI research company and developer platform for video APIs. We’ve been building AI video models for ‘digital twins’ or ‘avatars’ since 2020. We’re sharing some of the challenges we faced building an AI video interface that has realistic conversations with a human, including getting it to under 1 second of latency. To try it, talk to Hassaan’s digital twin: https://www.hassaanraza.com, or to our "demo twin" Carter: https://www.tavus.io We built this because until now, we've had to adapt communication to the limits of…
Tough Tongue AI 2.02025 · toughtongueai.com · ▲481A multimodal agent to create, share and rehearse scenarios
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 28d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, February 2026
the whole month →
Lovon AI Therapy▲801Feel closer and have more fun with your partner every day!
AI · Feb 2026 · lovon.app
Hello my name is Ben Ward for the past 3 years I have been remastering the financial game Wall Street Raider created by Michael Jenkins originally on DOS in 1986. It has been a rough journey but I finally see the light at the end of the tunnel. I just recently redid the website and thought maybe the full story of how this project came to be would interest you all. Thank you for reading.
Work · Feb 2026 · wallstreetraider.com