Word error rate (WER)

The word error rate (WER) is a metric that states how many words in a transcript are wrong, missing or added when compared with a reference text. It is calculated as the sum of substitutions, deletions and insertions divided by the number of words in the reference, and it is the standard measure for evaluating automatic speech recognition.

How it works

The word error rate compares two word sequences: the reference, meaning the verified wording of a recording, and the hypothesis, meaning the transcript produced by the speech recognition system. Morris, Maier and Green (2004) describe WER as the cost of turning the hypothesis back into the reference word by word: replacing a word ($S$), inserting a word that is missing ($D$ for a deletion from the reference) or removing a surplus word ($I$ for an insertion in the hypothesis). The formula is

$$\text{WER} = \frac{S + D + I}{N}$$

where $N$ is the number of words in the reference.

An example: the reference “The witness entered the room at seven o’clock” has eight words. If the software returns “The witness entered the room at seven o’clock” with “the” replaced by “a”, that is one substitution, so $S = 1$, $D = 0$, $I = 0$ and $\text{WER} = \frac{1 + 0 + 0}{8} = 0.125$. A word that is missing entirely counts as a deletion; a word the system invents counts as an insertion.

Because insertions are counted, the word error rate can exceed 1.0.

What it is used for

The word error rate is a comparable measure for testing speech recognition systems against each other and for naming their limits. Koenecke et al. (2020) compared five commercial systems and measured, on matched interviews, a word error rate of 0.35 for Black speakers against 0.19 for white speakers. The figure shows that a system does not have a single error level; it performs differently depending on speaker, dialect and recording situation.

For practice this means not treating the word error rate as one blanket value, but checking it separately for the recording types that actually occur. A system that performs well on clean dictation can perform far worse on overlapping speech or strong dialect.

Related measures

The character error rate (CER) applies the same approach at the level of characters instead of words. A misspelled abbreviation counts as a whole error for the word error rate, but only as the differing characters for the character error rate. The Hugging Face audio course therefore calls the word error rate prone to punishing spelling mistakes regardless of their severity.

The diarization error rate (DER) does not measure wording but the assignment of speech segments to speakers. The pyannote.metrics documentation defines it as the share of speech time that was counted as silence, falsely detected as speech, or assigned to the wrong speaker. The word error rate and the diarization error rate therefore measure independent things: a transcript can have every word right and still swap the speakers.

Accuracy is often calculated as $1 - \text{WER}$, which makes it a derived figure rather than a measure in its own right. Because insertions can push the word error rate above 1.0, accuracy calculated this way turns negative as soon as a system invents more words than the reference contains.

Relevance in a forensic context

Forensic transcripts from interviews, expert examinations or lawful interception are quoted, evaluated and placed in case files. A high share of errors at the passages that are later interpreted directly affects how the transcript can be used.

The word error rate depends heavily on the type of recording, for example on the number of speakers and whether they talk over each other. A single figure on a product page says little about how a system will perform on a specific interview or interception recording.

For work on the transcript this means the editor does not replace listening back with the error rate, but uses it as a guide to how much checking is needed. Recordings with many speakers or background noise get more time against the audio than a clean dictation.

Frequently asked questions

What does a word error rate of 0.1 mean?

Out of 100 words in the reference, 10 differ in the hypothesis through substitution, deletion or insertion. The figure alone does not say which words are affected. The editor checks that by reading the text against the recording.

Does a low word error rate mean the transcript can be used as evidence?

No. The word error rate assesses only the wording, not the speaker assignment, the punctuation, or whether a single error changes the meaning of a sentence. A wrong year in a quotation can weigh more than several missing filler words, even though both enter the figure equally.

Why do word error rates differ so much between studies?

Because the reference text, the recording conditions and the language determine the result. Koenecke et al. (2020) measured 0.35 against 0.19 with the same systems, depending on which group of people was speaking. Values from different test sets are only comparable when the reference text and the recording type match.

Sources

1. Morris, A. C., Maier, V., Green, P. (2004): From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. Interspeech 2004, pp. 2765 to 2768, doi:10.21437/Interspeech.2004-668. https://www.isca-archive.org/interspeech_2004/morris04_interspeech.html
2. Hugging Face: Evaluation metrics for ASR. Audio Course, chapter 5. https://huggingface.co/learn/audio-course/en/chapter5/evaluation
3. Koenecke, A. et al. (2020): Racial disparities in automated speech recognition. PNAS, published 7 April 2020, doi:10.1073/pnas.1915768117. https://www.pnas.org/doi/10.1073/pnas.1915768117
4. pyannote.metrics documentation, section Diarization error rate. https://pyannote.github.io/pyannote-metrics/reference.html