A benchmark investigating how emotional transitions within speech affect deepfake detection, across bona fide and synthetic audio.
🤗 EmoShift Dataset on Hugging Face ↗
Anonymous submission · Under review
Abstract
Speech deepfake detection systems must operate on expressive, naturally varying voices. Yet typical evaluations often treat emotional state as an utterance-level attribute, leaving the effects of emotion changes within an utterance comparatively underexplored.
We introduce EmoShift, a benchmark contrasting stable-emotion (AA) and emotion-transition (AB) speech for both bona fide and synthetic recordings. Across controlled and naturalistic settings, we evaluate seven detectors and analyze how emotional dynamics alter spoof scores, false alarms, and misses at fixed operating thresholds.
Transitions can increase spoof miss rates on in-the-wild audio by up to 25.3 percentage points under thresholds calibrated using stable-emotion speech. Conversational real speech exhibits a complementary pattern, where emotion changes can elevate false-alarm rates for some detectors. These observations show why evaluating emotional trajectories matters beyond utterance-level emotion categories.
Overview Figure
Within-utterance emotional stability
Within-utterance emotional change
Bona fide + Synthetic Speech → Seven Detectors → Score Shifts and Error Rates
Conceptual diagram. Paper figure will replace this after an approved anonymous export is provided.
Dataset Details
The prepared four-corpus release contains 16,960 utterances, with equal numbers of stable and transitioning examples. The naturalistic conditions do not universally provide same-text AA/AB controls. In particular, real OMG-Emotion examples are distinct natural utterances.
| Corpus | Speech style | Bona fide | Spoof | Total |
|---|---|---|---|---|
| ESD | Controlled | 800 | 2,400 | 3,200 |
| OMG-Emotion | In-the-wild | 440 | 1,320 | 1,760 |
| IEMOCAP | Conversational | 1,600 | 1,600 | 3,200 |
| MSP-Podcast | Naturalistic podcast | 4,400 | 4,400 | 8,800 |
| Total | AA: 8,480 / AB: 8,480 | 7,240 | 9,720 | 16,960 |
Audio Examples
For each corpus, the planned examples cover bona fide AA, bona fide AB, spoof AA, and spoof AB. The examples are selected from the final benchmark. Within each condition, bona fide and spoof samples share a source ID. ESD additionally provides matched speaker and text across AA and AB; the other corpora use independent AA and AB sources.
Resources
Analysis & Results
In-the-wild synthetic speech: Emotion transitions can lead to higher spoof miss rates at stable-condition operating thresholds, up to 25.3 percentage points in the reported experiments.
Naturalistic bona fide speech: Transitions can instead increase false alarms in conversational recordings. The direction and magnitude depend on corpus and detector.
Architecture matters: The XLS-R-based systems exhibit notable sensitivity, while other detectors show different responses associated with arousal.
Exact detector-by-corpus tables and figures will be added from the anonymous manuscript. Values have not been recreated or guessed.
Additional Analysis
Stable-condition calibration may not transfer. Equal-error-rate summaries alone can obscure changes in false-positive and false-negative rates at deployed operating thresholds.
Real and synthetic audio show different failure modes. Separate analysis of bona fide and spoof score shifts is necessary to understand which class is affected.
Experimental controls matter. Same-text matching is only available where justified by the corpus design. The broader evaluation considers splicing, duration and regional affective variation.
About the Project
EmoShift studies the robustness of speech deepfake detection to emotional trajectories within utterances. The project website is anonymous for peer review. Public download links will appear once release rights and anonymous-review requirements are satisfied.