Sharing Classifiers among Ensembles from Related Problem Domains
Pages 522 - 529
Abstract
A classification ensemble is a group of classifiers that all solve the same prediction problem in different ways. It is well-known that combining the predictions of classifiers within the same problem domain using techniques like bagging or boosting often improves the performance. This research shows that sharing classifiers among different but closely related problem domains can also be helpful. In addition, a semi-definite programming based ensemble pruning method is implemented in order to optimize the selection of a subset of classifiers for each problem domain. Computational results on a catalog dataset indicate that the ensembles resulting from sharing classifiers among different product categories generally have larger AUCs than those ensembles trained only on their own categories. The pruning algorithm not only prevents the occasional decrease of effectiveness caused by conflicting concepts among the problem domains, but also provides a better understanding of the problem domains and their relationships.
References
[1]
E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 36(1-2):105-139, 1999.
[2]
L. Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
[3]
L. Breiman. Arcing classifiers. Annals of Statistics, 26:801- 849, 1998.
[4]
L. Breiman. Random forests. Machine Learning, 45(1):5- 32, 2001.
[5]
S. Burer and R. Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming (Series B), 95:329- 357, 2003.
[6]
R. Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.
[7]
R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes. Ensemble selection from libraries of models. In Proc. of the 21st International Conference on Machine Learning, 2004.
[8]
P. K. Chan, W. Fan, A. Prodromidis, and S. J. Stolfo. Distributed data mining in credit card fraud detection. IEEE Intelligent Systems, November/December:67-74, 1999.
[9]
W. Fan, S. J. Stolfo, and J. Zhang. The application of adaboost for distributed, scalable and on-line learning. In Proc. of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 362-366. ACM Press, 1999.
[10]
W. Fan, H. Wang, and P. Yu. Mining extremely skewed trading anomalies. In Proc. of the 9th International Conference on Extending Database Technology, pages 801-810, 2004.
[11]
Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In International Conference on Machine Learning, pages 148-156, 1996.
[12]
M. Geomans and D. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of ACM, 42:1115- 1145, 1995.
[13]
Q. Han, Y. Ye, and J. Zhang. An improved rounding method and semidefinite programming relaxation for graph partition. Mathematical Programming, pages 509-535, 2002.
[14]
L. K. Hansen and P. Salamon. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell., 12(10):993-1001, 1990.
[15]
A. Krogh and J. Vedelsby. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 231-238. MIT Press, 1995.
[16]
A. Lazarevic and Z. Obradovic. The distributed boosting algorithm. In KDD '01: Proc. of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 311-316. ACM Press, 2001.
[17]
D. Margineantu and T. Dietterich. Pruning adaptive boosting. In 14th International Conference on Machine Learning, pages 211-218, 1997.
[18]
A. McCallum. Multi-label text classification with a mixture model trained by EM, 1999. AAAI Workshop on Text Learning.
[19]
R. J. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufman, San Manteo, CA, 1993.
[20]
A. Sharkey. On combining artificial neural nets. Connection Science, 8:299-313, 1996.
[21]
W. N. Street and Y. Kim. A streaming ensemble algorithm (SEA) for large-scale classification. In Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-01), pages 377-382, 2001.
[22]
G. Weiss and F. Provost. Learning when training data are costly: The effect of class distribution on tree induction. Journal of Artificial Intelligence Research, 19:315- 354, 2003.
[23]
Wikipedia. Herfindahl index, 2004.
[24]
D. H. Wolpert. Stacked generalization. Neural Networks, 5:241-259, 1992.
[25]
Y. Zhang, W. N. Street, and S. Burer. Ensemble pruning via semi-definite programming, 2005. Under review.
Index Terms
- Sharing Classifiers among Ensembles from Related Problem Domains
Recommendations
Ensembles of Region Based Classifiers
CIT '07: Proceedings of the 7th IEEE International Conference on Computer and Information TechnologyIn machine learning, ensemble classifiers have been introduced for more accurate pattern classification than single classifiers. We propose a new ensemble learning method that employs a set of region based classifiers. Since the distribution of data can ...
Ensembles of biased classifiers
ICML '05: Proceedings of the 22nd international conference on Machine learningWe propose a novel ensemble learning algorithm called Triskel, which has two interesting features. First, Triskel learns an ensemble of classifiers, each biased to have high precision on instances from a single class (as opposed to, for example, ...
Ensembles as a sequence of classifiers
IJCAI'97: Proceedings of the Fifteenth international joint conference on Artifical intelligence - Volume 2An ensemble is a classifier created by combining the predictions of multiple component classifiers. We present a new method for combining classifiers into an ensemble based on a simple estimation of each classifier's competence. The classifiers are ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2005
837 pages
ISBN:0769522785
Publisher
IEEE Computer Society
United States
Publication History
Published: 27 November 2005
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Jan 2025