Neural TTS: What Makes Text to Speech Sound Realistic

Neural TTS (neural text to speech) uses neural networks trained on recordings of real speech to turn text into audio. It is the reason synthetic voices now sound far more natural than the robotic voices of earlier systems: in published listening tests, neural models such as Tacotron 2 and VITS were rated close to human recordings. This guide explains what makes speech sound realistic, how neural TTS differs from older methods, how realism is measured, what the research actually reports, and how to get natural results in practice. Updated 15 October 2026.

What Is Neural TTS?

Text-to-speech systems have gone through three broad generations. Concatenative systems cut recordings of a voice into small units and joined the right units together; they could sound clear but often had audible seams and flat intonation. Statistical parametric systems generated speech from acoustic parameters predicted by a model; they were smoother but often sounded muffled or buzzy. Neural TTS replaced both with neural networks that learn directly from recorded speech.

A typical neural pipeline has two parts. An acoustic model reads the text (or the sequence of sounds derived from it) and predicts an intermediate representation such as a mel spectrogram, which describes how the energy of the voice is distributed over frequencies and time. A vocoder then turns this representation into an audio waveform. Tacotron 2 is a well-known example of this two-stage design. End-to-end models such as VITS go further and learn the whole path from text to waveform in a single model.

The turning point came with WaveNet, published by DeepMind researchers in 2016, whose authors reported that listeners rated it significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. Since then, neural TTS has become the standard for products where voice quality matters, from audiobooks and accessibility tools to virtual assistants and phone agents.

What Makes TTS Sound Realistic?

Realism in text to speech is not a single property. It comes from several acoustic characteristics that listeners use, mostly without noticing, to judge whether speech sounds natural. When one of them is wrong, the voice starts to sound artificial:

Neural TTS vs. Concatenative TTS

Concatenative systems are still found in older products, while neural TTS is the standard for new deployments where quality matters. The comparison below summarises the practical differences; the naturalness row uses the scores reported in the WaveNet paper, where both approaches were rated in the same listening test:

For a new project, neural TTS is the default choice. Concatenative synthesis can still make sense on very constrained hardware or for a small, fixed set of prompts, but it cannot adapt its intonation to new sentences the way a neural model can. Vocalis AI uses neural speech synthesis throughout, from voice generation to the streaming voice of its phone agents.

How TTS Realism Is Measured: MOS Listening Tests

The most widely used measure of TTS quality is the mean opinion score (MOS). Listeners hear short samples and rate each one on a five-point scale, from 1 (bad) to 5 (excellent); the ratings are averaged, usually with a confidence interval. The method comes from telecommunications: ITU-T Recommendation P.800 describes methods for the subjective determination of transmission quality, including this kind of absolute category rating.

Good studies include real recordings of the same speaker in the same test. This matters because human speech never scores a perfect 5: in the studies below, the recordings themselves scored between 4.46 and 4.58. A model is therefore judged by how close it comes to the recordings in the same test, not by its absolute score. Some papers also use a comparative MOS (CMOS), where listeners hear two versions of the same sentence and say which they prefer and by how much.

MOS has clear limits. Scores depend on the listeners, the sentences, the voice, the playback conditions and what the samples are compared with, so a 4.4 in one paper and a 4.4 in another do not mean the same thing. Vendor claims of a MOS score without a published method, sample set and human reference cannot be compared with anything.

What the Research Reports: Published Naturalness Scores

The table below lists scores reported in the original research papers of landmark neural TTS systems, each next to the score of human recordings in the same test. The figures come from different studies and are not comparable across rows; they show how close each system came to real speech in its own evaluation:

Scores as reported in each paper (see Sources). WaveNet: Table 1, US English, natural speech as 16-bit linear PCM. NaturalSpeech: comparative MOS against human recordings, with no statistically significant difference reported (Wilcoxon signed-rank test, p >> 0.05). Commercial voices are not listed because no comparable public evaluation exists for them.

Two lessons follow. First, on clean read speech from a single speaker, research systems have reached the point where listeners rate them close to real recordings. Second, those results come from controlled conditions: one voice, read sentences, good audio. Spontaneous conversation, emotional content, rare names, other languages and phone audio remain harder, which is why you should always judge a voice on your own text and your own channel.

From WaveNet to Zero-Shot Voices: How Neural TTS Evolved

The history of neural TTS since 2016 explains both what modern voices can do and where their limits come from. WaveNet showed that a neural network could generate the speech waveform sample by sample and be judged far more natural than earlier methods, but it was slow to run. Later work made generation much faster and split the job into two parts: an acoustic model that turns text into a spectrogram, and a neural vocoder that turns the spectrogram into audio.

Tacotron 2, published in 2017, combined these two parts and was rated 4.53 on the mean opinion score scale against 4.58 for professionally recorded speech. End-to-end models such as VITS (2021) then learned the whole path from text to waveform in a single model, which simplified training and improved naturalness on single-speaker datasets.

A more recent family treats speech like language. VALL-E, published by Microsoft researchers in 2023, encodes audio into discrete codes and trains a language model on them. Its authors report that it can synthesise speech in the voice of an unseen speaker from a 3-second enrolled recording used as an acoustic prompt, after scaling training data to 60,000 hours of English speech. This kind of zero-shot voice imitation is what makes voice cloning so easy today, and also why consent and disclosure rules matter.

For buyers, the practical consequence is simple: most commercial voices now use one of these neural approaches, so the differences you will hear come less from the generation of the model and more from the voice data, the tuning for your language, and how well the system handles your particular text.

Realistic TTS on the Phone

Phone agents add two constraints. The first is audio quality: classic G.711 telephony samples speech at 8 kHz, which keeps the voice intelligible but removes much of the high-frequency detail that makes a voice sound rich. Voices should therefore be chosen and tuned on a real phone line, not only on headphones.

The second is timing. In a live conversation the answer must start quickly, so TTS is used in streaming mode: the first words are synthesised and sent while the rest of the sentence is still being generated. The voice must also stop instantly when the caller interrupts. A voice that sounds slightly less polished but answers promptly and handles interruptions well usually gives a better phone experience than a perfect voice that leaves silences.

How to Get the Most Natural TTS Results

Even the best TTS engine produces weaker results with poorly prepared text. These five practices consistently give more natural output:

Common TTS Errors and How to Fix Them

Even excellent voices make predictable mistakes. Knowing them in advance saves hours of re-recording and is the fastest way to make synthetic speech sound professional.

Text normalisation errors.Before speaking, the engine must decide how to read numbers, dates, currencies, units and abbreviations. "1/2" can be a date or a fraction, "St." can be Saint or Street. Write ambiguous items in full, or use the SSML say-as element to state the type explicitly.

Homographs.Words spelled the same but pronounced differently, such as "read" (present or past) or "lead" (the verb or the metal), depend on context. Neural models usually guess right, but check them in your script and rephrase when needed.

Names and brand terms. Surnames, place names, product names and foreign words are the most common source of mispronunciation. Most engines let you add a custom lexicon or an SSML phoneme element with the exact pronunciation; build that list once and reuse it.

Wrong stress and flat questions. A sentence can be pronounced correctly but stressed on the wrong word, which changes its meaning. Short sentences, clear punctuation and SSML emphasis on the key word fix most cases.

Unnatural pauses. Pauses in the middle of a phrase, or no pause between two ideas, are quickly noticed. Adjust punctuation first, then use SSML break elements for the moments that really need a longer silence.

Inconsistency over long texts. In long narration, some engines drift in speed or tone. Generate long content in paragraphs, listen to the transitions, and keep the same voice settings for every part of a project.

How to Compare Realistic TTS Voices Yourself

Since published scores rarely cover commercial voices in comparable conditions, the most reliable comparison is a small listening test of your own. Pick 10 to 20 sentences that represent your real content, including names, numbers, questions and a few difficult words. Generate them with each voice you are considering, in the format your audience will hear (for example phone audio).

Play the samples in random order to several listeners who do not know which system produced which sample, and ask them to rate naturalness from 1 to 5, as in a MOS test. If possible, include a real recording of a person reading the same sentences as a reference. Note mispronunciations and odd stresses separately: they matter more to your customers than a small difference in average score.

Finally, check practical criteria alongside quality: the languages and accents you need, support for SSML and custom pronunciations, streaming speed for live use, the rights attached to the voice, and how your text and audio data are handled.

Responsible Use of Realistic Voices

The more realistic synthetic voices become, the more important it is to use them openly. Tell listeners when they are hearing an AI voice, especially in conversations: in the EU, Article 50 of the AI Act requires AI systems that interact with people to be designed so that people know, unless this is obvious, and requires synthetic audio to be marked in a machine-readable format as far as technically feasible. Only clone or imitate a voice with the documented consent of the person it belongs to.

Frequently Asked Questions

Sources

van den Oord A. et al., "WaveNet: A Generative Model for Raw Audio", 2016, Table 1 — arxiv.org/abs/1609.03499

Shen J. et al., "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (Tacotron 2), 2017 — arxiv.org/abs/1712.05884

Kim J., Kong J., Son J., "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech" (VITS), 2021 — arxiv.org/abs/2106.06103

Tan X. et al., "NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality", 2022 — arxiv.org/abs/2205.04421

Wang C. et al., "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (VALL-E), 2023 — arxiv.org/abs/2301.02111

ITU-T Recommendation P.800 (08/1996), "Methods for subjective determination of transmission quality" — itu.int

W3C, "Speech Synthesis Markup Language (SSML) Version 1.1", W3C Recommendation, 7 September 2010 — w3.org/TR/speech-synthesis11

ITU-T Recommendation G.711, "Pulse code modulation (PCM) of voice frequencies" — itu.int