Speaker diarization: how AI works out who said what in a recording

Speaker diarization is the task of dividing an audio recording into segments and labeling each one with the person speaking, so the output reads “speaker A from 0:00 to 0:12, speaker B from 0:12 to 0:20.” It answers “who spoke when” without necessarily knowing anyone’s name. Transcription tools and AI note-takers rely on it to turn a block of text into a conversation with turns.

The idea matters more as always-on recorders move into daily life. Amazon’s new Bee devices, covered in our report on Bee Frontier and Eclipse, summarize conversations into to-dos and follow-up emails. A summary that cannot tell who said “I will send the invoice” is of limited use.

Sorting a recording into voices before any words are read

Diarization works on the sound, not the meaning. A useful analogy is a stranger sitting in a noisy cafe who cannot follow the language but can still tell that three different people are talking, because each voice has its own pitch, timbre and rhythm. The software does the same thing numerically.

The classic pipeline has four steps. First, voice activity detection separates speech from silence and background noise. Second, the speech is cut into short windows. Third, a neural network turns each window into an embedding, a list of numbers meant to capture the voice rather than the words, so that two windows from the same person land close together. Fourth, a clustering algorithm groups the embeddings, and each group becomes one speaker label. The number of speakers is usually not known in advance, which is part of the difficulty.

Diarization is related to, but different from, speaker verification, which checks whether a voice matches one enrolled person. A device that unlocks content only for its owner uses verification. A system that labels everyone in a meeting uses diarization, which has no enrolled list to compare against.

From telephone research to neural embeddings

Diarization grew out of speech research on telephone calls and broadcast news, and for years relied on hand-designed acoustic features. The shift to neural networks is visible in a 2017 paper by Quan Wang, Carlton Downey, Li Wan, Philip Mansfield and Ignacio Lopez Moreno, “Speaker Diarization with LSTM.” It paired LSTM-based neural embeddings, called d-vectors, with non-parametric clustering and reported a 12.0% diarization error rate on the NIST SRE 2000 CALLHOME benchmark.

A follow-up in 2018 by Aonan Zhang, Quan Wang, Zhenyao Zhu, John Paisley and Chong Wang, “Fully Supervised Speaker Diarization,” replaced the clustering step with a recurrent network called UIS-RNN, trained on recordings with time-stamped speaker labels. The authors reported a 7.6% error rate on the same benchmark and noted that their method can decode online, while most leading systems depended on offline clustering. Online decoding matters for anything that runs while a person is still talking.

Toolkits made the approach widely available. In 2019, Hervé Bredin and colleagues released pyannote.audio, an open-source library of trainable components for voice activity detection, speaker change detection, overlapped speech detection and speaker embedding that can be combined into a diarization pipeline. It was presented at ICASSP 2020.

Skipping the clustering step with end-to-end models

Clustering has two known weaknesses. It is not trained to minimize diarization errors directly, and it handles overlapping speech poorly, because a window containing two voices has to be assigned to one group. In 2019, Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Yawen Xue, Kenji Nagamatsu and Shinji Watanabe published a self-attention version of End-to-End Neural Diarization (EEND). A single network takes a multi-speaker recording and outputs, for each moment, which speakers are active, so two people can be marked as talking at once.

Self-attention lets the model compare every moment of the recording with every other moment, rather than only neighboring ones. The authors report that this approach outperformed both their earlier LSTM-based version and a state-of-the-art x-vector clustering system on simulated mixtures, real telephone calls and real dialogue recordings.

Neither family has won outright. Clustering-based systems cope with many speakers and long recordings more easily, while end-to-end models handle overlap better. Many current systems mix the two ideas.

Where it shows up in everyday products

Meeting recorders, call-center analytics, podcast and interview transcription, and clinical note tools all use diarization to attribute lines to people. Wearable recorders are the newest case. A device clipped to clothing or stuck to a phone hears the owner, the people around them, and sometimes a speakerphone, so it must separate several voices in real time or soon after.

Amazon’s announcement for Bee, as summarized in our story, describes summaries, action items and an Echo that uses voice ID to recognize the user before sharing Bee content. That is speaker recognition of the owner. The announcement as reported does not explain how Bee labels other people in a conversation, so how much of this relies on diarization is not something the public material settles.

Why overlap, noise and short turns still cause errors

The organizers of the DIHARD series of challenges, led by Neville Ryant and colleagues, built a test set specifically to measure how diarization holds up across recording conditions. The third edition, reported in 2020, covered 11 domains including meeting speech, clinical interviews, web video and telephone speech, and drew 499 valid system outputs from 30 organizations. The authors report marked improvement since the first challenge, particularly for two-person conversations, but also state that for many domains, such as web video, the problem “remains far from solved.”

The hard cases are predictable:

• Overlapping speech: people interrupt each other, and clustering assigns a mixed window to one speaker.
• Short turns: a “yes” or “right” lasts too briefly to carry a reliable voice signature.
• Noise and distance: restaurants, open offices and speakerphones degrade the embeddings that everything else depends on.
• Similar voices: family members or same-gender colleagues can be merged into one speaker.

A mislabeled turn is more than a cosmetic flaw. In a summary, it can credit a commitment or a statement to the wrong person.

Privacy questions that come with labeling voices

Voice embeddings are biometric-style data, which is why recorders that tell speakers apart raise different questions from ones that only transcribe. Where people are recorded, consent rules vary by region, and Amazon says Eclipse prompts users to get call participants’ permission when a recording starts. Whether a system stores voiceprints, discards them after each session, or never creates them for bystanders is a design choice worth checking in a product’s documentation.

Technically, the field is moving toward models that handle diarization and transcription together, and toward systems that work reliably on live audio with an unknown number of speakers. Benchmarks such as DIHARD, which test across messy real-world domains rather than clean studio recordings, are the clearest measure of whether always-on recorders can attribute speech correctly outside the lab.