Abstract
In this paper, we propose a method to automatically compute a speech evaluation metric, Virtual Mean Opinion Score (vMOS) for the speech generated by Text-to-Speech (TTS) models to analyse its human-ness. In contrast to the currently used manual speech evaluation techniques, the proposed method uses an end-to-end neural network to calculate vMOS which is qualitatively similar to manually obtained Mean Opinion Score (MOS). The Generative Adversarial Network (GAN) and a binary classifier have been trained on real natural speech with known MOS. Further, the vMOS has been calculated by averaging the scores obtained by the two networks. In this work, the input to GAN’s discriminator is conditioned with the speech generated by off-the-shelf TTS models so as to get closer to the natural speech. It has been shown that the proposed model can be trained with a minimum amount of data as its objective is to generate only the evaluation score and not speech. The proposed method has been tested to evaluate the speech synthesized by state-of-the-art TTS models and it has reported the vMOS of 0.6675, 0.4945 and 0.4890 for Wavenet2, Tacotron and Deepvoice3 respectively while the vMOS for natural speech is 0.6682 on a scale from 0 to 1. These vMOS scores correspond to and are qualitatively explained by their manually calculated MOS scores.
J. Jaiswal and A. Chaubey—Denotes equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Arik, S.Ö., et al.: Deep voice: real-time neural text-to-speech. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 195–204. JMLR.org (2017)
Wang, Y., et al.: Tacotron: towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 (2017)
Ping, W.: Deep voice 3: scaling text-to-speech with convolutional sequence learning. arXiv preprint arXiv:1710.07654 (2017)
van den Oord, A., et al.: Wavenet: a generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016)
Salza, P.L., Foti, E., Nebbia, L., Oreglia, M.: MOS and pair comparison combined methods for quality evaluation of text-to-speech systems. Acta Acust. United Acust. 82(4), 650–656 (1996)
Ghate, P., Shirbahadurkar, S.D.: A survey on methods of TTS and various test for evaluating the quality of synthesized speech. Int. J. Dev. Res. 07, 15236–15239 (2017)
Ito, K.: The LJ speech dataset (2017)
Veaux, C., Yamagishi, J., MacDonald, K., et al.: CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit. University of Edinburgh, The Centre for Speech Technology Research (CSTR) (2017)
Livingstone, S.R., Russo, F.A.: The Ryerson audio-visual database of emotional speech and song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in North American English. PloS One 13(5), e0196391 (2018)
Busso, C., et al.: IEMOCAP: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42(4), 335 (2008)
Dwivedi, R.C., et al.: Acoustic parameters of speech: lack of correlation with perceptual and questionnaire-based speech evaluation in patients with oral and oropharyngeal cancer treated with primary surgery. Head Neck 38(5), 670–676 (2016)
Sebastian, R.J., Ryan, E.B.: Speech cues and social evaluation: markers of ethnicity, social class, and age. In: Recent Advances in Language, Communication, and Social Psychology, pp. 112–143. Routledge (2018)
Tyagi, V., Ganapathiraju, A., Wyss, F.I.: Method and system for selectively biased linear discriminant analysis in automatic speech recognition systems. US Patent 9,679,556, 13 June 2017
Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. In: Advances in Neural Information Processing Systems, pp. 577–585 (2015)
Gajic, B., Narayanan, S.S., Parthasarathy, S., Rose, R.C., Rosenberg, A.E.: System and method of performing user-specific automatic speech recognition. US Patent 9,058,810, 16 June 2015
Renals, S.: Automatic Speech Recognition-ASR Lecture: Multilingual Speech Recognition (2017)
Russell, M., Cook, A.: Experimental evaluation of duration modelling techniques for automatic speech recognition. In: ICASSP 1987, IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 12, pp. 2376–2379. IEEE (1987)
Moore, A.H., Parada, P.P., Naylor, P.A.: Speech enhancement for robust automatic speech recognition: evaluation using a baseline system and instrumental measures. Comput. Speech Lang. 46, 574–584 (2017)
Otani, T., Togawa, T., Nakayama, S.: Speech evaluation apparatus and speech evaluation method. US Patent App. 15/703,249, 29 March 2018
Sailor, H.B., Patil, H.A.: Fusion of magnitude and phase-based features for objective evaluation of TTS voice. In: The 9th International Symposium on Chinese Spoken Language Processing, pp. 521–525. IEEE (2014)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). arxiv:1412.6980. Comment: Published as a Conference Paper at the 3rd International Conference for Learning Representations, San Diego (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR, abs/1512.03385 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
An, S., Ling, Z., Dai, L.: Emotional statistical parametric speech synthesis using LSTM-RNNs. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1613–1616. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jaiswal, J. et al. (2020). A Generative Adversarial Network Based Ensemble Technique for Automatic Evaluation of Machine Synthesized Speech. In: Palaiahnakote, S., Sanniti di Baja, G., Wang, L., Yan, W. (eds) Pattern Recognition. ACPR 2019. Lecture Notes in Computer Science(), vol 12047. Springer, Cham. https://doi.org/10.1007/978-3-030-41299-9_45
Download citation
DOI: https://doi.org/10.1007/978-3-030-41299-9_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41298-2
Online ISBN: 978-3-030-41299-9
eBook Packages: Computer ScienceComputer Science (R0)