Abstract
Unlike fixed combining rules, the trainable combiner is applicable to ensembles of diverse base classifier architectures with incomparable outputs. The trainable combiner, however, requires the additional step of deriving a second-stage training dataset from the base classifier outputs. Although several strategies have been devised, it is thus far unclear which is superior for a given situation. In this paper we investigate three principal training techniques, namely the re-use of the training dataset for both stages, an independent validation set, and the stacked generalization. On experiments with several datasets we have observed that the stacked generalization outperforms the other techniques in most situations, with the exception of very small sample sizes, in which the re-using strategy behaves better. We illustrate that the stacked generalization introduces additional noise to the second-stage training dataset, and should therefore be bundled with simple combiners that are insensitive to the noise. We propose an extension of the stacked generalization approach which significantly improves the combiner robustness.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Dietrich, C., Palm, G., Schwenker, F.: Decision templates for the classification of bioacustic time series. Information Fusion 4, 101–109 (2003)
Duin, R.P.W.: The combining classifiers: to train or not to train? In: Proc. of 16th Int. Conf. on Pattern Recognition (Quebec City), vol. II, pp. 765–770 (2002)
Duin, R.P.W., Juszczak, P., de Ridder, D., Paclík, P., Pekalska, E., Tax, D.M.J.: PR-Tools 4.0, a Matlab toolbox for pattern recognition. Technical report, ICT Group, TU Delft, The Netherlands (January 2004), http://www.prtools.org
Duin, R.P.W., Tax, D.M.J.: Experiments with classifier combining rules (invited paper). In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 16–29. Springer, Heidelberg (2000)
Kuncheva, L.I., Bezdek, J.C., Duin, R.P.W.: Decision templates for multiple classifier fusion: an experimental comparison. Pattern Recog. 34(2), 299–314 (2001)
Kuncheva, L.I.: Combining Pattern Classifiers. Wiley & Sons, Chichester (2004)
LeBlanc, M., Tibshirani, R.: Combining estimates in regression and classification. Journal of the American Statistical Association 91(436) (1996)
Raudys, S., Janeliunas, A.: Reduction of the boasting bias of linear experts. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, p. 242. Springer, Heidelberg (2002)
Roli, F., Raudys, S., Marcialis, G.L.: An experimental comparison of fixed and trained fusion rules for crisp classifier outputs. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, pp. 232–241. Springer, Heidelberg (2002)
Skurichina, M., Paclik, P., Duin, R.P.W., de Veld, D.C.G., Sterenborg, H.J.C.M., Witjes, M.J.H., Roodenburg, J.L.N.: Selection/Extraction of spectral regions for autofluorescence spectra measured in the oral cavity. In: Fred, A., Caelli, T.M., Duin, R.P.W., Campilho, A.C., de Ridder, D. (eds.) SSPR&SPR 2004. LNCS, vol. 3138, pp. 1096–1104. Springer, Heidelberg (2004)
Suen, C.Y., Lam, L.: Multiple classifier combination methodologies for different output levels. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, p. 52. Springer, Heidelberg (2000)
Ting, K.M., Witten, I.H.: Issues in stacked generalization. Journal of Artificial Intelligence Research 10, 271–289 (1999)
Wolpert, D.H.: Stacked generalization. Neural Networks 5, 241–259 (1992)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Paclík, P., Landgrebe, T.C.W., Tax, D.M.J., Duin, R.P.W. (2005). On Deriving the Second-Stage Training Set for Trainable Combiners. In: Oza, N.C., Polikar, R., Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2005. Lecture Notes in Computer Science, vol 3541. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11494683_14
Download citation
DOI: https://doi.org/10.1007/11494683_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26306-7
Online ISBN: 978-3-540-31578-0
eBook Packages: Computer ScienceComputer Science (R0)