Ismail Rasim Ulgen
Author directory2026
Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages
Haopeng Xie | Ismail Rasim Ulgen | Sofia Son | Berrak Sisman | Philipp Koehn
Findings of the Association for Computational Linguistics: EMNLP 2026
Haopeng Xie | Ismail Rasim Ulgen | Sofia Son | Berrak Sisman | Philipp Koehn
Findings of the Association for Computational Linguistics: EMNLP 2026
Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.
Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models
Xiutian Zhao | Ismail Rasim Ulgen | Philipp Koehn | Björn Schuller | Berrak Sisman
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Xiutian Zhao | Ismail Rasim Ulgen | Philipp Koehn | Björn Schuller | Berrak Sisman
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Speech-generative large audio-language models (LALMs) can produce expressive speech, yet reliable emotion control remains elusive: conversions often miss the target affect and may degrade linguistic fidelity through refusals, hallucinations, or paraphrase. We present, to our knowledge, one of the first neuron-level studies of emotional voice conversion (EVC) in LALMs and provide intervention-based evidence that compact sets of emotion-sensitive neuron (ESN) are functionally involved in, and can be exploited for, steering expressive speech generation at inference time. ESNs are identified via success-filtered activation aggregation enforcing both emotion realization and content preservation. Across three LALMs (Qwen2.5-Omni-7B, MiniCPM-o 4.5, Kimi-Audio), ESN interventions yield emotion-specific gains that generalize to unseen speakers. Controllability depends on selector design, mask sparsity, filtering, and intervention strength. Our results provide an intervention-based framework for training-free emotion control in LALMs for EVC.
AffectDF: The Most Comprehensive Benchmark for Speech Deepfake Detection against Emotionally Expressive Attacks
Aurosweta Mahapatra | Xiutian Zhao | Shreeram Suresh Chandra | Zihan Zhang | Zongyang Du | Ismail Rasim Ulgen | Kong Aik Lee | Nicholas Andrews | Carlos Busso | Berrak Sisman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Aurosweta Mahapatra | Xiutian Zhao | Shreeram Suresh Chandra | Zihan Zhang | Zongyang Du | Ismail Rasim Ulgen | Kong Aik Lee | Nicholas Andrews | Carlos Busso | Berrak Sisman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Speech deepfake detection (SDD) systems achieve strong performance on conventional benchmarks; however, existing datasets provide limited coverage of emotionally expressive and recent large audio-language model (LALM)-based attacks. Existing emotional spoofing datasets are also limited in scale and attack diversity, typically covering only voice conversion (VC) or text-to-speech (TTS) attacks. We introduce AffectDF, the most comprehensive benchmark for emotionally expressive speech deepfakes, spanning TTS, VC, emotional VC, and LALM-based spoofing attacks across both acted and spontaneous emotional speech. AffectDF contains approximately 260 hours of speech generated using 21 spoofing attacks across five emotional states. We benchmark state-of-the-art SDD systems under conventional and emotional spoofing conditions. Our experiments reveal severe robustness degradation when models trained on conventional benchmarks are evaluated on AffectDF, with several systems approaching near-random performance. Surprisingly, even large-scale emotional training does not consistently improve cross-domain robustness, indicating that current SDD systems fail to learn generalized spoof representations under emotional and prosodic variability. Robustness further varies substantially across emotional states, attack families, and acted vs spontaneous emotional speech conditions. These findings expose fundamental limitations of current SDD systems and establish AffectDF as a benchmark for developing more robust spoof detection models.