本研究旨在探討並改善正常語音與耳語語音之間的跨語音型態語者辨識問題。耳語語音因缺乏聲帶振動，導致基頻與諧波結構消失，使其嵌入特徵與正常語音之間存在顯著的分布落差，進而影響語者辨識模型於跨型態條件下之穩定性與準確性。
為改善此問題，本研究結合自監督式學習（Self-Supervised Learning, SSL）語音模型所擷取之高階語音嵌入，並提出一個基於 Transformer 架構之特徵轉換模型，用以學習耳語與正常語音嵌入之間的非線性映射關係。透過均方誤差損失（Mean Squared Error, MSE）與三元組損失（Triplet Loss）之共同優化，模型可對齊跨語音型態之嵌入分布，同時保留語者之區辨特性。
實驗採用自建之 Whisper-100 語料庫，並於 Normal → Whisper 與 Whisper → Normal 兩種跨型態情境下進行評估。本研究同時從語者識別（Identification）與語者驗證（Verification）兩個面向進行分析，分別以分類準確率與等錯誤率（Equal Error Rate, EER）作為評估指標。
實驗結果顯示，所提出之特徵轉換方法能縮小正常語音與耳語語音之間的嵌入分布差異，提升跨型態語者辨識表現，並改善未見語者條件下的語者驗證能力。此外，本研究亦分析性別因素對嵌入分布與特徵轉換效果之影響，結果顯示不同性別資料在跨型態轉換與辨識表現上存在差異，說明資料分布特性對耳語語者辨識模型之學習具有重要影響。整體而言，本研究驗證了以自監督式語音嵌入結合特徵轉換策略處理跨語音型態語者辨識問題之可行性，並為後續耳語語音辨識與跨型態語音處理研究提供參考。

This study aims to investigate and improve the problem of cross-phonation speaker recognition between normal speech and whispered speech. Since whispered speech lacks vocal fold vibration, the fundamental frequency and harmonic structure are largely absent, resulting in a significant distribution gap between whispered and normal speech embeddings. This mismatch further affects the stability and accuracy of speaker recognition models under cross-phonation conditions. To address this issue, this study leverages high-level speech embeddings extracted from self-supervised learning (SSL) speech models and proposes a Transformer-based feature mapping model to learn the nonlinear mapping relationship between whispered and normal speech embeddings. By jointly optimizing Mean Squared Error (MSE) loss and Triplet Loss, the model can align the embedding distributions across phonation types while preserving speaker-discriminative characteristics. Experiments were conducted on the self-constructed Whisper-100 corpus and evaluated under two cross-phonation scenarios: Normal→Whisper and Whisper→Normal. This study analyzes the performance from both speaker identification and speaker verification perspectives, using classification accuracy and Equal Error Rate (EER) as the evaluation metrics, respectively. Experimental results show that the proposed feature transformation method can reduce the embedding distribution gap between normal and whispered speech, improve cross-phonation speaker recognition performance, and enhance speaker verification capability under unseen-speaker conditions. In addition, this study analyzes the influence of gender factors on embedding distributions and feature transformation effectiveness. The results show that gender-related differences exist in cross-phonation transformation and recognition performance, indicating that data distribution characteristics play an important role in learning whispered speech speaker recognition models. Overall, this study verifies the feasibility of using self-supervised speech embeddings combined with a feature transformation strategy to address cross-phonation speaker recognition, and provides a reference for future research on whispered speech recognition and crossphonation speech processing.

[1] Preeti Wadhwani. Conversational system market size: By component, by technology, by deployment mode, by application, growth forecast, 2025–2034. Global Market Insights Inc. (Report ID: GMI13898), May 2025.

[2] T. Itoh, K. Takeda, and F. Itakura. Acoustic analysis and recognition of whispered speech. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU ’01., pages 429–432, 2001.

[3] Zhaofeng Lin, Tanvina Patel, and Odette Scharenborg. Improving whispered speech recognition performance using pseudo-whispered based data augmentation. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023.

[4] Xing Fan and John HL Hansen. Speaker identification within whispered speech audio streams. IEEE transactions on audio, speech, and language processing, 19(5):1408–1421, 2010.

[5] Lu Yi and Man Wai Mak. Cross-domain adaptation in distance space for speaker verification. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 2238–2243, 2023.

[6] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179–1210, 2022.

[7] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc.

[8] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451–3460, October 2021.

[9] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. Beats: audio pre-training with acoustic tokenizers. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.

[10] Shikhar Bharadwaj, Samuele Cornell, Kwanghee Choi, Satoru Fukayama, Hye jin Shim, Soham Deshmukh, and Shinji Watanabe. Openbeats: A fully open-source general-purpose audio encoder. 2025 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1–5, 2025.

[11] Najim Dehak, Patrick J. Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet. Front-endfactor analysisforspeakerverification. IEEETransactionsonAudio, Speech, and Language Processing, 19(4):788–798, 2011.

[12] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879–4883, 2018.

[13] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329–5333, 2018.

[14] Ville Vestman, Dhananjaya Gowda, Md Sahidullah, Paavo Alku, and Tomi Kinnunen. Speaker recognition from whispered speech: A tutorial survey and an application of time-varying linear prediction. Speech Communication, 99:62–79, 2018.

[15] Aref Farhadipour, Homa Asadi, and Volker Dellwo. Leveraging self-supervised models for automatic whispered speech recognition. In 2024 14th International Conference on Computer and Knowledge Engineering (ICCKE), pages 188–193, 2024.

[16] Matthias Janke, Michael Wand, Till Heistermann, Tanja Schultz, and K Prahallad. Fundamental frequency generation for whisper-to-audible speech conversion. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2579–2583. IEEE, 2014.

[17] Jun Rekimoto. Wesper: Zero-shot and realtime whisper to normal voice conversion for whisper-based speech interactions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, 2023. Association for Computing Machinery.

[18] Chi Zhang and John HL Hansen. Analysis and classification of speech mode: whispered through shouted. In Interspeech, volume 7, pages 2289–2292, 2007.

[19] Xing Fan and John HL Hansen. Acoustic analysis and feature transformation from neutral to whisper for speaker identification within whispered speech audio streams. Speech communication, 55(1):119–134, 2013.

[20] Jakaria Islam Emon, Md Abu Salek, and Kazi Tamanna Alam. Whisper speaker identification: Leveraging pre-trained multilingual transformers for robust speaker embeddings. arXiv preprint arXiv:2503.10446, 2025.

[21] Dominik Wagner, Ilja Baumann, and Tobias Bocklet. Generative adversarial networks for whispered to voiced speech conversion: a comparative study. International Journal of Speech Technology, 27(4):1093–1110, 2024.

[22] Songting Liu. Zero-shot voice conversion with diffusion transformers. arXiv preprint arXiv:2411.09943, 2024.

[23] Santi Prieto, Alfonso Ortega Giménez, Iván López-Espejo, and EDUARDO LLEIDA SOLANO. Shouted speech compensation for speaker verification robust to vocal effort conditions. In Interspeech, 2020.

[24] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2015.