How Automatic Speech Recognition Works (Explained Simply)
By the VidzTranscripts team · · 4 min read
Automatic speech recognition (ASR) went from frustrating to genuinely useful in just a few years. Today's systems punctuate, handle accents and work in dozens of languages. Knowing roughly how they work helps you understand their mistakes and get better transcripts. This article explains the process in plain language, using what happens to a file uploaded to VidzTranscripts as the example.
Step 1: Getting the audio ready
Videos and audio files come in many formats: MP4, MOV, MP3, M4A, WhatsApp's OPUS and more. The first step extracts the sound and converts it to one simple format: a single channel (mono) at 16,000 samples per second. That is lower quality than music needs, but it keeps everything in the range of the human voice, and it is what speech models are trained on. It is also why the size or resolution of a video makes no difference to transcription: the picture is thrown away immediately.
Step 2: Finding the speech
Next, a small, fast model called a voice activity detector scans the audio and marks where people are speaking. Silence, long pauses and pure music are skipped. This makes transcription faster, and it matters for accuracy: speech models asked to transcribe silence sometimes invent words, so removing the silence first prevents a whole class of errors.
Step 3: Turning sound into a picture
Speech models don't work on the raw sound wave. The audio is cut into slices a few hundredths of a second long, and for each slice the system measures how much energy there is at different pitches. The result is a spectrogram: a kind of image of the sound, with time running left to right and pitch from bottom to top. Vowels, consonants and even individual voices have recognisable shapes in this image.
Step 4: The model
VidzTranscripts uses Whisper, an open-source speech recognition model family released by OpenAI and trained on hundreds of thousands of hours of speech in many languages. Whisper is a transformer, the same broad kind of neural network used in modern language models, with two halves:
- The encoder reads about 30 seconds of spectrogram at a time and builds an internal representation of what was said, taking the whole window into account rather than one sound at a time.
- The decoder writes the text one small piece (a token) at a time, each time looking at the encoder's representation and at the words it has already written.
Because the decoder sees the words it has written so far, it behaves partly like a language model: it knows that "the cat sat on the" is more likely to be followed by "mat" than "bat". This is why modern ASR produces sensible sentences with capital letters and punctuation, and also why its mistakes tend to be plausible-looking words rather than gibberish.
Step 5: Choosing the words
At each step the model has probabilities for many possible next tokens. Instead of always taking the single most likely one, a technique called beam search keeps several candidate sentences going at once and picks the best complete sequence. It is slower, but it avoids getting stuck after one early wrong guess.
Step 6: Timings and subtitles
Finally, the system works out when each word was spoken. We then regroup the words into caption-sized lines: breaking at sentence ends and pauses, keeping lines to about two rows of text, and never showing one caption for more than about seven seconds. That grouping is what makes the SRT and VTT files readable as subtitles.
Why it runs on our own servers
Speech models like this can run without a graphics card, using an optimised engine and a compressed version of the model. We run them on our own servers, which means recordings are never sent to a third-party AI service, and it is how the service can be free for everyday use.
Why transcripts still have mistakes
- Rare words: names, brands and technical jargon appeared rarely in the training data, so the model substitutes a common word that sounds similar.
- Noise and echo: they blur the spectrogram, hiding the shapes the model relies on.
- Overlapping speakers: two voices at once produce a mixed picture that matches neither.
- Music: singing and instruments occupy the same pitch range as speech.
- Short clips: a few seconds of audio give the model little context, and automatic language detection has little to go on.
- Similar-sounding words that change the meaning, like "can" and "can't", because the language-model side of the decoder finds both plausible.
How accuracy is measured
Researchers measure ASR with word error rate (WER): the number of words substituted, deleted or inserted, divided by the number of words actually spoken. A WER of 5% means about one word in twenty is wrong, which is roughly the level of a careful human transcriber on clear speech. On clear, close-miked speech, modern models come close to that; on noisy phone recordings or strong accents the error rate can be several times higher. That's why the recording itself matters more than anything else you can control.