Jack Dorsey’s Block has released a new audio model from its lab that is running locally and makes expressions like laughing, pausing, and crying sound uncannily human. The company says the model, called “Dessa,” is fully trained on human voice and a single NVIDIA GPU.
Dessa is a full duplex model. This means you can call and listen at the same time, similar to the GPT-Live model OpenAI released earlier this year. Speech models are typically paired with larger language models for knowledge retrieval. Block said that Dessa’s knowledge comes from conversation data Dessa has.
Block said the current model includes text in the pipeline that drives speech production, which can sound unnatural and annoying. Many models, such as Eleven Lab v4 and Fish Audio Drama 3, use voice tags to drive emotion with voice production. Dorsey’s company claims that its model incorporates emotion. It is unclear whether that emotion is manipulated by some kind of tag.
“Interacting with current voice AI is simply the worst. It converts your voice into text and throws away an incredible amount of information that text alone can’t convey. The world’s most powerful AI can’t hear how you’re feeling. We think it’s stupid, leads to a worse experience, and is potentially dangerous,” the company said in a blog post.
The company’s researchers posted multiple examples of conversations with Dessa in which the agent’s voice, rhythm, and emotion varied. For example, in a conversation, the bot said it was tired from work, but it sounded tired at the time.
In the example below, the bot was emotional and on the verge of tears while talking about things that made the person feel comfortable.
Another photo shows members of the Block team playing the game using a voice assistant.
The company says this model can run locally on consumer-grade computers starting with the MacBook M4 Max. However, there is currently no public access to Dessa, and Block is calling on safety researchers to test the model. This could test cases where a voice agent might fool people on the other end.
Block is in the financial business and sells solutions to small and medium-sized businesses. Square already uses voice AI to process orders. But with unique voice models that run on-device, you can enable more business and consumer experiences at lower cost.
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
