Text to Speech Evaluation
Evaluating the text to speech engines that would give every Dynamic Sym its voice, and finding out the most natural voice was the wrong one.
In a Dynamic Sym, the voice is the customer
A Dynamic Sym is an AI powered roleplay where a simulated customer talks to a trainee in real time. The trainee hears a person on the other end of the line. That person is a text to speech engine.
So the voice is not a detail. If it lags for twenty seconds, the conversation dies. If it says a price or an email address wrong, the Sym quietly trains the wrong behaviour. Before Symtrain committed one engine to power every Dynamic Sym, someone had to answer a deceptively simple question, which voice engine should we use. That was the research.
A voice that mispronounces teaches the wrong thing
Early testing surfaced a pattern of failures that would break a training Sym. The voice sounded human, until it hit the exact things a customer service call is made of.
Names and terms
Organisation names, people, and technical terms read incorrectly.
Wrong language slipping in
Spanish Syms output numbers in Portuguese. Hinglish read as Hindi, with sentiment scoring unavailable.
Numbers, currency, email
The exact tokens a real call runs on, prices, account numbers, emails, came out mangled.
Naturalness, latency, and accuracy pull against each other. The most natural voice was the slowest. The fastest voice mispronounced the most. Which tradeoff is right for a live training conversation?
The evaluation criteria
I judged every engine on the dimensions that decide whether a Dynamic Sym feels real and stays usable.
Naturalness
Does it sound like a human customer, with believable prosody and tone.
Latency
Does it reply fast enough to hold a live conversation.
Pronunciation accuracy
Does it read numbers, currency, emails, and codes correctly.
Language coverage
How many languages and accents, and how well it localises.
Control
Can we steer tone, emotion, and style predictably.
Cost and scale
What does it cost per hour of audio at production volume.
Six engines, very different bets
I scanned the market from hosted enterprise APIs to open source models, and worked out the real cost per hour of audio for each.
| Engine | Best for | Languages | Cost per hour of audio | Live latency |
|---|---|---|---|---|
| ElevenLabs | Expressive, conversational voices | 32 and up to 70 plus | ~$8.10 Flash | Fast |
| Google Gemini TTS | Enterprise and multilingual | 75 plus, 380 plus voices | ~$0.90 Flash, ~$1.80 Pro | Fast |
| Amazon Polly | System voices and IVR | Dozens | ~$0.86 neural, ~$0.22 standard | Fast |
| Qwen3 TTS | Open source experimentation | 10 mainstream | Open source, free to run | Low latency |
| Chatterbox | Open source voice agents | 23, cross language transfer | Open source, free to run | Low latency |
| NVIDIA PersonaPlex | Interactive persona agents | Not published | Research, not publicly priced | Real time |
Cost per hour derived from public pricing at roughly 15 characters per second of speech. Directional, for comparison.
Expressive, or predictable
Every engine sits somewhere on the same tension. The expressive ones feel human but drift. The predictable ones stay in line but sound flat. A live Sym needs the top right, human enough, and controllable.
One brutal test script
To compare engines fairly, I wrote a single test passage that packs in every hard case a real service call throws at a voice, numbers, an ID, dates, a time, a blood pressure reading, two currencies, an email, a phone number, and a URL. I ran it in English and Spanish, through the ElevenLabs API, and logged status, latency, and exactly how each token came out.
Real API tests
Voices Gabriela and Matilda, mp3 output, run with and without a language code.
Status and latency
Every run logged a 200 status and a precise response time in milliseconds.
Token by token
Each phone number, email, price, and ID was checked against how it should sound.
Three ElevenLabs models, one clear split
The deep dive came down to three models. The most natural one was the slowest by far. Latency turned out to be the deciding factor.
Measured response time per request. A twenty three second reply is unusable in a live conversation, no matter how good it sounds.
Best overall balance
Fast, natural, and handled numerics best. Occasional stutter on IDs and edge tokens.
Fast but risky
Comparable speed, but louder, less natural, and more gibberish on emails and addresses.
Beautiful, unusable live
The most natural voice, but around twenty three seconds per reply and no language code support.
Expected, versus what I heard
The failures were specific and repeatable. These are the moments that would break a trainee's belief, or teach them the wrong thing.
The last one was the key clue. The model does not know a region's mannerisms, the way machine translation misses local habits. That pointed straight at the fix.
The most natural voice was the wrong voice.
For a live training conversation, latency is not a technical detail, it is a feature. A voice that replies in five seconds and reads a price correctly beats a beautiful voice that takes twenty three seconds and drops the euros.
Pick the fast voice, then fix the edges
The initial hypothesis favoured the most natural model. The testing flipped it. My recommendation was a pairing, the right engine plus a layer to handle the tokens it fumbles.
ElevenLabs Flash v2.5
The best balance of naturalness, low latency, and numeric accuracy for a real time Sym. Fast enough to hold a conversation, natural enough to believe.
A pronunciation library
Normalise numbers, currency, emails, and abbreviations before they reach the engine, and standardise how they are read aloud. This is where the fix lives, not in the voice itself.
Latency outranks polish
Multilingual v2 sounded best but replied in around twenty three seconds and could not enforce numeric localisation. Unusable live.
Run a side by side pilot
Validate Flash against real Symtrain scripts, then tune the pronunciation layer for the edge cases that matter most to each client.
A technical decision, made on evidence
De risked a core decision
Turned the voice engine choice from an opinion into an evidence backed recommendation, and surfaced the pronunciation library as a needed feature.
A reusable evaluation method
A single edge case test harness and a clear scoring approach the team could rerun against any new model.
Clear direction to build on
Gave product and engineering one recommendation, the reasons behind it, and the next step to validate it.
As UX Researcher, I
- Framed the voice engine decision as a research question
- Evaluated six voice technologies across the market
- Worked out the real cost per hour of audio for each
- Designed a single edge case test passage
- Ran structured API tests in English and Spanish
- Measured latency and logged every response
- Scored pronunciation token by token
- Compared three ElevenLabs models closely
- Traced the root cause to regional pronunciation
- Recommended the engine and the pronunciation layer
- Defined a side by side pilot to validate the choice
- Documented the findings for product and engineering
A product's voice is a UX decision, not an audio one.
The right voice for a conversation is not the most beautiful. It is the one you stop noticing, because it never makes you wait and never gets the details wrong.
← Back to all work