Can diarization handle two people talking at the same time?

Not well. When two voices overlap, the audio in that stretch doesn't match either person's voice cleanly, so the system typically assigns the whole overlap to one speaker, or the words themselves come out garbled. Expect to fix overlaps by hand. Separate tracks per speaker avoid most of this, because each track contains one voice.

How can I tell whether a tool's speaker labels are good on my recordings?

Test it on a real recording where you know who said what, ideally 20 to 30 minutes with several speakers. Then count four kinds of error: wrong speaker on a turn, one person split into two labels, two people merged into one label, and speaker changes that were missed. Compare tools on the same file. A demo with two studio voices tells you little about a six-person meeting.

Speaker Diarization Explained: How AI Tells Voices Apart in Meetings and Interviews

What Diarization Is

Speaker diarization is the "who spoke when" layer of a transcript. It answers:

Where does one speaker stop and the next start?
Is the voice at 4:23 the same person as the voice at 18:07?
How many different speakers are in the recording?

Transcription gives you the words; diarization gives you the turns. They fail in different ways, so a transcript can have accurate words and wrong speaker labels. Review the labels as a separate pass.

How It Works

Most systems go through three stages.

1. Voice activity detection

The system separates speech from silence, music and background noise. Each stretch of speech becomes a candidate turn.

2. Voice embeddings

Each speech segment is turned into a voice embedding: a list of numbers that describes how the voice sounds (pitch, timbre, speaking rate, the shape of the vocal tract). Segments from the same person produce embeddings that sit close together; different people's sit further apart.

3. Clustering

The system groups similar embeddings. Each group becomes "Speaker 1", "Speaker 2" and so on, and the number of groups is its estimate of how many people spoke.

Why More Speakers Means More Errors

With two or three clearly different voices, the groups are far apart and easy to separate. As speakers are added, the groups start to overlap. Two people with similar voices may be merged into one label, and one person whose voice changes (laughing, shouting, moving away from the mic) may be split into two. With a large group, the system may also underestimate how many people there are.

The amount of speech per person matters too. Someone who says one sentence in an hour gives the system very little to match against, so short contributions are mislabeled more often.

Separate Tracks Skip the Problem

The most reliable fix is not a better model but one audio track per speaker:

speaker_alice.m4a   (only Alice's mic)
speaker_bob.m4a     (only Bob's mic)
speaker_carol.m4a   (only Carol's mic)

Each track has one voice, so there is nothing to cluster. You transcribe each track and combine the results by timestamp. This works when participants wear headphones; otherwise other voices bleed into each mic.

Ways to get separate tracks:

Zoom: local recording has an option to save a separate audio file for each participant
Riverside, Zencastr: record each remote guest on their own track
Craig (Discord bot): records each speaker separately
A multi-channel audio interface for in-person recording, one mic per person

Recording Habits That Help Single-Track Diarization

A separate mic per person. Two people on one laptop mic look like one voice to the system.
Fewer interruptions. Overlapping speech is the hardest case for any system.
Consistent setup. If someone switches from headset to laptop mic mid-call, the system may treat them as a new speaker.
Names at the start. If each person introduces themselves, the reviewer can match labels to people quickly.

Common Errors and Fixes

"Speaker 1 became Speaker 3 halfway through"

The person's voice changed enough to look new, usually after a mic change or a move. Replace Speaker 3 with Speaker 1 for that stretch.

"Two people share one label"

Their voices were too similar and got merged. The only fix is listening to the ambiguous parts and relabeling by content.

"One person has two labels"

The opposite problem. If one of the two labels appears only briefly, it is usually the same person; merge them.

"Speaker changes land a moment late"

Turn boundaries are approximate. It rarely matters for an archive; for captions, adjust the boundary in review.

When It's Worth Turning On

Use speaker labels for interviews, meetings where decisions need attribution, and podcast transcripts that readers will follow by speaker.

Skip them for single-speaker content, and for group discussions where only the gist matters.

In Picute

Speaker labels are an option you turn on when you submit a video or recording; they work on a single mixed file. Like any diarization, they need a review pass, more so with many voices. If you have separate tracks, you can instead transcribe each track as its own job and combine the SRT files by time with the free Merge SRT tool, which merges two files at a time and can keep their original timings. Each track is billed for its full length (1 credit per minute), and the merged file carries no speaker names unless you add them.