How AI Transcription Actually Works: A Non-Technical Explainer
Transcription happens in three stages: preparing the audio, recognizing the speech, and cleaning up the output. Knowing what each stage does makes it easier to guess why a transcript went wrong in a particular place.
Stage 1: Preparing the audio
- Extraction. Only the audio track is needed, so it is pulled out of the video file. Audio is a small fraction of a video's size: ten minutes at 128 kbps is under 10 MB.
- Normalization. Volume is evened out so a quiet speaker and a loud one reach the model at similar levels.
- Noise handling. Some pipelines filter background noise first. Many current models are instead trained on noisy recordings and take the audio more or less as it is. Either way, loud music under speech still causes errors.
- Windowing. Long recordings are processed in pieces. OpenAI's Whisper, for example, works on 30-second windows.
Stage 2: Recognizing the speech
Most current speech recognition models are transformer networks, the same family of architecture behind large language models, trained on transcribed speech instead of text.
How the model "hears"
Many models don't read the raw waveform. The audio is first turned into a spectrogram: a picture of frequency over time, where the horizontal axis is time, the vertical axis is frequency, and brightness is loudness. The model reads that picture and predicts which words produced it.
Because it was trained on a very large amount of real speech, the model learns patterns such as:
- choosing "their", "there" or "they're" from the surrounding words
- telling a pause at the end of a sentence from a hesitation mid-thought
- writing "gonna" as spoken or as "going to", depending on how its training transcripts were written
Word-level timing
To place subtitles, a tool needs to know when each word was spoken. Models get there in different ways, for example by aligning the predicted words to the audio frames they came from. SRT files write times to the millisecond, but that is the precision of the file format, not a promise about the alignment.
Language detection
Many systems can detect the spoken language from the opening seconds, and some cope with speakers who switch languages mid-sentence. Choosing the language yourself removes one way for things to go wrong.
Stage 3: Cleaning up the output
- Punctuation and capitalization. Older systems added these with a separate model. Many current models output punctuated text directly.
- Speaker labels (diarization). Often a separate step that groups the audio by voice and tags each stretch as Speaker 1, Speaker 2 and so on. Useful for interviews and meetings.
- Subtitle segmentation. The transcript is split into subtitle-sized pieces of one or two lines, broken at natural pauses.
Why accuracy varies
- Accents and dialects. A model does best on speech that resembles what it was trained on.
- Audio quality. Phone recordings, heavy compression and laptop microphones blur the spectrogram, so the model has less to work with.
- Domain vocabulary. General models stumble on medical terms, legal jargon, brand names and acronyms. Domain-specific models exist for fields such as medicine.
- Overlapping speech. When two people talk at once, their voices overlap in the spectrogram, and separating them is still hard.
What makes some languages harder
- No spaces between words. Chinese, Japanese and Thai are written without spaces, so the model has to decide where one word ends. Korean does use spaces, but its spacing rules are intricate, so spacing is worth checking in review.
- Tone. In Mandarin, Thai and Vietnamese, the same syllable with a different pitch contour is a different word.
- Several writing systems. Japanese mixes hiragana, katakana and kanji, so the model also has to pick a script for each word.
Practical tips for a better transcript
- Use a dedicated microphone. Even an inexpensive USB microphone close to the speaker beats a laptop's built-in one.
- Cut background noise at the source. Close windows and turn off fans before you record.
- Avoid talking over each other. In interviews, let the other person finish. Overlap is the error you can't fix later.
- Test in your own language. Before you commit to a tool, run a short sample in the language and audio conditions you actually record in.
Where Picute fits
Picute sends every job to Soniox's speech-to-text model. You choose the spoken language from the 60 it supports, can turn on speaker labels, and can see the word-level timing in the editor, where each word highlights as it plays. The stages above still apply: a clean recording gives a cleaner transcript, and names and terms still need a check.