Skip to content
VidzTranscripts

How Automatic Speech Recognition Works (Explained Simply)

By the VidzTranscripts team · · 4 min read

Automatic speech recognition (ASR) went from frustrating to genuinely useful in just a few years. Today's systems punctuate, handle accents and work in dozens of languages. Knowing roughly how they work helps you understand their mistakes and get better transcripts. This article explains the process in plain language, using what happens to a file uploaded to VidzTranscripts as the example.

Step 1: Getting the audio ready

Videos and audio files come in many formats: MP4, MOV, MP3, M4A, WhatsApp's OPUS and more. The first step extracts the sound and converts it to one simple format: a single channel (mono) at 16,000 samples per second. That is lower quality than music needs, but it keeps everything in the range of the human voice, and it is what speech models are trained on. It is also why the size or resolution of a video makes no difference to transcription: the picture is thrown away immediately.

Step 2: Finding the speech

Next, a small, fast model called a voice activity detector scans the audio and marks where people are speaking. Silence, long pauses and pure music are skipped. This makes transcription faster, and it matters for accuracy: speech models asked to transcribe silence sometimes invent words, so removing the silence first prevents a whole class of errors.

Step 3: Turning sound into a picture

Speech models don't work on the raw sound wave. The audio is cut into slices a few hundredths of a second long, and for each slice the system measures how much energy there is at different pitches. The result is a spectrogram: a kind of image of the sound, with time running left to right and pitch from bottom to top. Vowels, consonants and even individual voices have recognisable shapes in this image.

Step 4: The model

VidzTranscripts uses Whisper, an open-source speech recognition model family released by OpenAI and trained on hundreds of thousands of hours of speech in many languages. Whisper is a transformer, the same broad kind of neural network used in modern language models, with two halves:

Because the decoder sees the words it has written so far, it behaves partly like a language model: it knows that "the cat sat on the" is more likely to be followed by "mat" than "bat". This is why modern ASR produces sensible sentences with capital letters and punctuation, and also why its mistakes tend to be plausible-looking words rather than gibberish.

Step 5: Choosing the words

At each step the model has probabilities for many possible next tokens. Instead of always taking the single most likely one, a technique called beam search keeps several candidate sentences going at once and picks the best complete sequence. It is slower, but it avoids getting stuck after one early wrong guess.

Step 6: Timings and subtitles

Finally, the system works out when each word was spoken. We then regroup the words into caption-sized lines: breaking at sentence ends and pauses, keeping lines to about two rows of text, and never showing one caption for more than about seven seconds. That grouping is what makes the SRT and VTT files readable as subtitles.

Why it runs on our own servers

Speech models like this can run without a graphics card, using an optimised engine and a compressed version of the model. We run them on our own servers, which means recordings are never sent to a third-party AI service, and it is how the service can be free for everyday use.

Why transcripts still have mistakes

How accuracy is measured

Researchers measure ASR with word error rate (WER): the number of words substituted, deleted or inserted, divided by the number of words actually spoken. A WER of 5% means about one word in twenty is wrong, which is roughly the level of a careful human transcriber on clear speech. On clear, close-miked speech, modern models come close to that; on noisy phone recordings or strong accents the error rate can be several times higher. That's why the recording itself matters more than anything else you can control.

Need a transcript?

Convert your video or audio to text and subtitles for free.

Transcribe a file

Popular converters