Abstract
The paper points to problematic and usually neglected aspects of using listening tests for TTS evaluation. It shows that simple random selection of phrases to be listened to may not cover those cases which are relevant to the evaluated TTS system. Also, it shows that a reliable phrase set cannot be chosen without a deeper knowledge of the distribution of differences in synthetic speech, which are obtained by comparing the output generated by an evaluated TTS system to what stands as a baseline system. Having such knowledge, the method able to evaluate the reliability of listening tests, as related to the estimation of possible invalidity of listening results-derived conclusion, is proposed here and demonstrated on real examples.
The research has been supported by the Technology Agency of the Czech Republic, project No. TA01030476 and by the European Regional Development Fund (ERDF), project “New Technologies for Information Society” (NTIS), European Centre of Excellence, ED1.1.00/02.0090.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ircing, P., Psutka, J., Psutka, J.V.: Using morphological information for robust language modeling in Czech ASR system. IEEE Transactions on Audio Speech and Language Processing 17, 840–847 (2009)
Pražák, A., Psutka, J.V., Hoidekr, J., Kanis, J., Müller, L., Psutka, J.: Automatic online subtitling of the Czech parliament meetings. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2006. LNCS (LNAI), vol. 4188, pp. 501–508. Springer, Heidelberg (2006)
Hanzlíček, Z., Matoušek, J.: Voice conversion based on probabilistic parameter transformation and extended inter-speaker residual prediction. In: Matoušek, V., Mautner, P. (eds.) TSD 2007. LNCS (LNAI), vol. 4629, pp. 480–487. Springer, Heidelberg (2007)
Tihelka, D., Romportl, J.: Exploring automatic similarity measures for unit selection tuning. In: INTERSPEECH 2009, Proceedings of the 10th Annual Conference of International Speech Communication Association, pp. 736–739. ISCA, Brighton (2009)
Black, A.W., Tokuda, K.: The Blizzard Challenge – 2005: Evaluating corpus-based speech synthesis on common datasets. In: Interspeech, Lisbon, Portugal, pp. 77–80 (2005)
Yamagishi, J., Zen, H., Jian Wu, Y., Toda, T., Tokuda, K.: The HTS-2008 system: Yet another evaluation of the speaker-adaptive hmm-based speech synthesis system in The 2008 Blizzard Challenge. In: Proc. Blizzard Challenge 2008 (2008)
Tihelka, D., Kala, J., Matoušek, J.: Enhancements of viterbi search for fast unit selection synthesis. In: Interspeech, pp. 174–177 (2010)
Eide, E., Aaron, A., Bakis, R., Cohen, P., Donovan, R., Hamza, W., Mathes, T., Picheny, M., Polkosky, M., Smith, M., Viswanathan, M.: Recent improvements to the IBM trainable speech synthesis system. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2003, vol. 1, pp. 708–711 (2003)
Vepa, J., King, S.: Subjective evaluation of join cost functions used in unit selection speech synthesis. In: Interspeech, Jeju Island, Korea, pp. 1181–1184 (2004)
Grůber, M., Tihelka, D.: Expressive speech synthesis for Czech limited domain dialogue system – basic experiments. In: IEEE 10th International Conference on Signal Processing (ICSP), Beijing, China, vol. 1, pp. 561–564 (2010)
Grůber, M.: Enumerating differences between various communicative functions for purposes of Czech expressive speech synthesis in limited domain. In: Interspeech, pp. 650–653. Curran Associates, Inc., Morehouse Lane (2012)
Přibil, J., Přibilová, A.: Distributed listening test program for synthetic speech evaluation, pp. 241–242 (2008)
Hanzlíček, Z., Matoušek, J., Tihelka, D.: Experiments on reducing footprint of unit selection TTS system (2013)
Bowman, A.W., Azzalini, A.: Applied Smoothing Techniques for Data Analysis. Oxford University Press (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tihelka, D., Grůber, M., Hanzlíček, Z. (2013). Robust Methodology for TTS Enhancement Evaluation. In: Habernal, I., Matoušek, V. (eds) Text, Speech, and Dialogue. TSD 2013. Lecture Notes in Computer Science(), vol 8082. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40585-3_56
Download citation
DOI: https://doi.org/10.1007/978-3-642-40585-3_56
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40584-6
Online ISBN: 978-3-642-40585-3
eBook Packages: Computer ScienceComputer Science (R0)