The voice AI startup ElevenLabs has launched its latest conversational AI system, “Conversational AI 2.0,” setting a new standard for AI conversations that seamlessly transition between voice and text. This upgrade goes beyond simple functional improvements, focusing on implementing AI that listens and speaks as naturally as a real person.
Even responds to ‘Um… hang on’
Traditional voice AI often produced awkward silences or breaks when recognizing and reacting to user speech. However, this new system analyzes hesitation, pauses, and interjections in real-time to adjust the timing of responses. For instance, when it hears “Um… hang on…”, the AI recognizes this not as mere silence but as a signal that the user is thinking, and waits accordingly. This makes the flow of conversation smooth and natural.
Freedom of speech and text conversion, with multilingual support
“Conversational AI 2.0” supports ‘multimodal’ functions that freely transition between voice and text. Whether a user switches to typing mid-conversation or the reverse, the AI continues seamlessly without interruption. It’s particularly useful in noisy backgrounds or when precise input, like numbers or addresses, is needed.
In addition, an automatic language detection feature is also included. Even if a user starts in English and switches to Japanese or Spanish halfway through, the AI detects this in real-time and responds in the appropriate language. This functionality is especially attractive for companies dealing with multinational clientele, with expectations of application across various sectors like customer service, call centers, and marketing.
Connection to external knowledge for accurate responses
The system also incorporates a ‘Retrieval-Augmented Generation’ (RAG) feature, which allows it to connect with external data to provide answers reflecting up-to-date knowledge. For example, in medical settings, it can deliver the latest treatment guidelines or current policies and product information in customer support services instantly.
Security and operational reliability have been significantly enhanced for immediate application in corporate environments. This includes compliance with HIPAA, EU regional data storage options, and improved system stability. The system also supports a ‘mass messaging’ feature, enabling efficient use in customer notifications and surveys.
Smarter voice AI
This update from ElevenLabs suggests that AI has reached a stage where it recognizes the ‘etiquette’ and ‘context’ expected in human-to-human communication.
As AI understands the basic flow of communication—listening, pausing, responding—the threshold for introducing AI in business settings is expected to drop significantly. Collaboration with AI in areas like voice-based customer support, multilingual service, and real-time information provision across various industries is likely to become more seamless.