Building a Real-Time AI Avatar for Coaching and Training Simulations
cerebrium.ai/blog/how-to-build-a-real-time-ai-avatar-for-training-and-coaching At Cerebrium, we have recently built a few demos showing voice AI capabilities (worlds fastest voice agent & realtime RAG agent) but we wanted to push the boundary and see if we could create realistic, human-like situations to train and onboard teams to perform better - recreating real life scenarios! An example of this is a sales coach for your sales team, an investor pitch or even prep for a notoriously stressful YC interview . To achieve this there were a few difficult problems to solve, namely: - How…
What it does
In the maker’s words, at launch
cerebrium.ai/blog/how-to-build-a-real-time-ai-avatar-for-training-and-coaching At Cerebrium, we have recently built a few demos showing voice AI capabilities (worlds fastest voice agent & realtime RAG agent) but we wanted to push the boundary and see if we could create realistic, human-like situations to train and onboard teams to perform better - recreating real life scenarios! An example of this is a sales coach for your sales team, an investor pitch or even prep for a notoriously stressful YC interview . To achieve this there were a few difficult problems to solve, namely: - How do you recreate a human-like video call both physically and/or emotionally (an angry customer, a fast speaker etc)? - How do you steer the conversation to a specific outcome? - How do you do function calling at low latency? Here's how we solved each of these problems: How do you recreate a human-like video call both physically and/or emotionally (an angry customer, a fast speaker etc) - We used Tavus (tavus.io) to create a realistic AI avatar. Tavus allows you to build AI-generated video experiences with an API. They have created very modular API’s whereby you can select an OpenAI compatible endpoint as your LLM as well as any TTS service. You are also able to train my own human replica with a few video clips. - To get across emotion, we used the new emotional control released by Cartesia (cartesia.ai) - a fast, realistic voice API. They allow you to select from a range of emotions (Angry, Sad, Positive etc) as well as different talking speeds. This allows us to convey various emotions in these simulated environments such as angry customers. How to you steer the conversation to a specific outcome - This wasn’t the most complex issue - we simply used function calling to steer a conversation in a specific direction based on answers given by a user. Our implementation isn’t bulletproof, but given more time you could implement some robust methods. Make the above have low latency interactions? - What's tough about implementing function-calling is latency. If you are using a API that has function-calling capabilities (Mistral, OpenAI etc) the latencies are very high because the process of function calling is: - Make a request to the API with an instruction. - API determines that you need to use a function. You then run your function and get a result. - You send the result to the API (AGAIN) and then get a response which you can show to the user. - Both API calls incur network time and so we saw the average roundtrip response time of 600-800ms for EACH API call with the TTFT fluctuating around ~300ms each time. Therefore you are looking at a minimum TTFT of ~1s when you take into consideration the two requests and your TTS service. - To get around this we implemented Mistral-7B locally on Cerebrium (cerebrium.ai), which has function calling capabilities. Our TTFT was ~80ms in us-east-1 and we never had to go over the network for the second request (since we were calling the model locally). Therefore our TTFT was roughly 150ms for our LLM which put our response time of voice-to-voice responses at roughly ~300-400ms - 3x lower! Room for improvement There is definitely some room for improvement in our implementation that you would need to make this production ready for many company use cases, most notably is the issue of user pauses and detecting when a user has finished their response. Models are not sophisticated enough to know if a user is still thinking or formulating their response. Currently when we hear silence, the model starts responding which really stresses you out in the interview use case! We made all the code available as well as wrote a tutorial of how we got this up and running. We would love feedback or ideas from the community on how to make this application better. More importantly, it would be great if you commit to the GitHub repo so the community can benefit from it.
Does the same job
all alternatives →
AVTR-1 Real-Time Open Weights ModelMay 2026 · avaturn.live · ▲197Generating uncanny AI avatars is now open source
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
