Abstract
In this paper, we propose and compare single- and multi-objective programming (MOP) approaches to the language model (LM) adaptation that require the optimization of a number of competing objectives. In LM adaptation, an adapted LM is found so that it is as close as possible to two independently trained LMs. The LM adaptation approach developed in this paper is based on reformulating the training objective of a maximum a posteriori (MAP) method as an MOP problem. We extract the individual at least partially conflicting objective functions, which yields a problem with four objectives for a bigram LM: The first two objectives are concerned with the best fit to the adaptation data while the remaining two objectives are concerned with the best prior information obtained from a general domain corpus. Solving this problem in an iterative manner such that each objective is optimized one after another with constraints on the rest, we obtain a target LM that is a log-linear interpolation of the component LMs. The LM weights are found such that all the (at least partially conflicting) objectives are optimized simultaneously. We compare the performance of the SOP- and MOP-based solutions. Our experimental results demonstrate that the ICO method achieves a better balance among the design objectives. Furthermore, the ICO method gives an improved system performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The divergences from unigram models as well as bigram models should be considered since backing-off is used when an unknown n-gram is observed during the recognition (test) stage.
References
Mitchell, T. M. (1997). Machine learning. New York: McGraw-Hill.
Liu, G. P., & Kadirkamanathan, V. (1995). Learning with multi objective criteria. In Proc. of IEE conference on artificial neural networks (pp. 53–58).
Braga, A. P., Takahashi, R. H. C., Costa, M. A., & Teixeira, R. A. (2006). Multi-objective algorithms for neural networks learning. In Y. Jin, (Ed.), Multi-objective machine learning (pp. 151–171). Berlin: Springer.
Suttorp, T., & Igel, C. (2006). Multi-objective optimization of support vector machines. In Y. Jin, (Ed.), Multi-objective machine learning (pp. 199–220). Berlin: Springer.
Bellegarda, J. R. (2000). Exploiting latent semantic information in statistical language modeling. Proceedings of IEEE, 88, 1279–1296.
Yaman, S., & Lee, C.-H. (2006). An iterative constrained optimization approach to classifier design. In Proc. of the IEEE conference on acoustics, speech, and signal processing. Toulouse, France.
Yaman, S., Chien, J.-T., & Lee, C.-H. (2007). Structural Bayesian language modeling and adaptation. In Proc. Interspeech. Antwerp, Belgium.
Miettinen, K. (1999). Nonlinear multiobjective optimization. Berlin: Springer.
Das, I., & Dennis, J. (1996). Closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Rice University, Houston, TX: Tech. Rep. 96–36, Department of Computational and Applied Mathematics.
Bishop, C. M. (2007). Pattern recognition and machine learning. Berlin: Springer.
Nocedal, J., & Wright, S. J. (1999). Numerical optimization. Berlin: Springer.
Paul, D. B., & Baker, J. M. (1992). The design for the wall street journal-based CSR corpus. In The proc. of internation conference of spoken language processing, Banff, Alberta, Canada, September.
Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural language processing. Cambridge: MIT.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yaman, S., Lee, CH. A Comparison of Single- and Multi-Objective Programming Approaches to Problems with Multiple Design Objectives. J Sign Process Syst 61, 39–50 (2010). https://doi.org/10.1007/s11265-008-0295-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11265-008-0295-2