What is speaker diarization?
Speaker diarization is the process of segmenting an audio recording by speaker identity, answering the question "who spoke when?" across a multi-person conversation.
If you've ever read a transcript where every line starts with "Unknown speaker:", you've felt the frustration that speaker diarization solves. Diarization is the automatic process of dividing an audio recording into segments and labeling each segment with the identity of the speaker who produced it. The word itself comes from the Latin diarium (daily record), reflecting the original goal of creating a structured log of a spoken interaction.
How it works
A diarization system generally works in three stages.
1. Voice activity detection: The system first identifies which parts of the audio contain speech and which contain silence, noise, or music. This step removes irrelevant regions so the model doesn't waste effort on them.
2. Speaker change detection: The system scans the speech regions for moments where one speaker transitions to another. Changes in pitch, vocal timbre, and speaking rhythm all serve as cues.
3. Speaker clustering: Segments that belong to the same voice are grouped together. This is typically done by converting each segment into a compact "speaker embedding"—a mathematical fingerprint of the vocal characteristics—and then clustering embeddings that are similar to each other. Each cluster gets a label (Speaker 1, Speaker 2, and so on) rather than a name, unless the system has reference audio to match against known identities.
Modern diarization systems are usually combined with speech to text, so you end up with a transcript where each line is attributed to a specific speaker and timestamped.
Where it's used
Speaker diarization is essential in any context where multiple people are talking:
- Meeting notes: Automatically attribute action items and decisions to the person who stated them, rather than leaving them as floating quotes.
- Interview transcription: Separate interviewer questions from respondent answers in a research or journalism workflow.
- Call center analytics: Label agent versus customer turns so quality assurance tools can score each party independently.
- Legal proceedings: Court reporters and eDiscovery tools use diarization to create accurate transcripts of depositions and hearings where multiple attorneys, witnesses, and judges speak.
- Podcast and broadcast production: Editors can quickly navigate to a specific guest's segments without scrubbing through audio.
Practical limitations
Diarization accuracy degrades when speakers talk over each other (overlapping speech is hard to segment cleanly), when two speakers have very similar voices, or when the audio quality is poor. The system also outputs speaker labels like "Speaker A" rather than real names by default—name assignment requires a separate identification step.
Diarization and subtitles
When diarization output is used to generate subtitles for a video with multiple speakers, each speaker's lines can be visually distinguished—different colors, positions, or name tags. This is particularly useful for panel discussions and round tables where viewers benefit from knowing who is speaking. For a closer look at how subtitles are structured and how they differ from closed captions, see subtitles vs captions.