Abstract
We present a new hierarchical strategy for fine-grained categorization. Standard, fully automated systems report a single estimate of the category, or perhaps a ranked list, but have non-neglible error rates for most realistic scenarios, which limits their utility. Instead, we propose a semi-automated system which outputs a it confidence set (CS)—a variable-length list of categories which contains the true one with high probability (e.g., a 99 % CS). Performance is then measured by the expected size of the CS, reflecting the effort required for final identification by the user. The implementation is based on a hierarchical clustering of the full set of categories. This tree of subsets provides a graded family of candidate CS’s containing visually similar categories. There is also a learned discriminant score for deciding between each subset and all others combined. Selection of the CS is based on the joint score likelihood under a Bayesian network model. We apply this method to determining the species of a plant from an image of a leaf against either a uniform or natural background. Extensive experiments are reported. We obtain superior results relative to existing methods for point estimates for scanned leaves and report the first useful results for natural images at the expense of asking the user to initialize the process by identifying specific landmarks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Angelova, A. & Zhu, S. (2013). Efficient object detection and segmentation for fine-grained recognition. In: CVPR.
Belhumeur, P.N., Chen, D., Feiner, S., Jacobs, D.W., Kress, W.J., Ling, H., Lopez, I.C., Ramamoorthi, R., Sheorey, S., White, S., Zhang, L. (2008). Searching the world’s herbaria: A system for visual identification of plant species. In: ECCV (4), pp 116–129.
Bourdev, L.D. & Malik, J. (2009). Poselets: Body part detectors trained using 3d human pose annotations. In: ICCV, pp 1365–1372.
Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In: ECCV (4), pp 438–451.
Burl, M.C. & Perona, P. (1998). Using hierarchical shape models to spot keywords in cursive handwriting data. In: CVPR, pp 535–540.
Caballero, C. & Aranda, M.C. (2010). Plant species identification using leaf image retrieval. In: CIVR, pp 327–334.
Casanova, D., Florindo, J.B. & Bruno, O.M. (2011). Ifsc/usp at imageclef 2011: Plant identication task. In: CLEF (Notebook Papers/Labs/Workshop).
Casanova, D., Florindo, J.B., Gonçalves, W.N. & Bruno, O.M. (2012) Ifsc/usp at imageclef 2012: Plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).
Cook, N. R. (2005). Confidence Intervals and Sets. : John Wiley and Sons Ltd.
Cope, J. S., Corney, D. P. A., Clark, J. Y., Remagnino, P., & Wilkin, P. (2012). Plant species identification using digital morphometrics: A review. Expert Syst Appl, 39(8), 7562–7573.
del Coz, J. J., Díez, J., & Bahamonde, A. (2009). Learning nondeterministic classifiers. Journal of Machine Learning Research, 10, 2273–2293.
Deng, J., Berg, A.C., Li, K., Li, F.F. (2010). What does classifying more than 10, 000 image categories tell us? In: ECCV (5), pp 71–84.
Deng, J., Krause, J., Berg, A.C. & Li, F.F. (2012). Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In: CVPR, pp 3450–3457.
Deng, J., Krause, J. & Li, F.F. (2013). Fine-grained crowdsourcing for fine-grained recognition. In: CVPR, pp 580–587.
Du, J.X., Huang, D., Wang, X. & Gu, X. (2005). Shape recognition based on radial basis probabilistic neural network and application to plant species identification. In: ISNN (2), pp 281–285.
Duan, K., Parikh, D., Crandall, D.J. & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In: CVPR, pp 3474–3481.
El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11, 1605–1641.
Ellis, B. (2009). Manual of leaf architecture. Cornell paperbacks, Published in association with the New York Botanical Garden.
Elpel, T. (2004). Botany in a Day: The Patterns Method of Plant Identification. Thomas J. Elpel’s herbal field guide to plant families of North America. : Hops Press.
Fan, X. & Geman, D. (2004). Hierarchical object indexing and sequential learning. In: ICPR (3), pp 65–68.
Farrell, R., Oza, O., Zhang, N., Morariu, V.I., Darrell, T., Davis, L.S. (2011a). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.
Farrell, R., Oza, O., Zhang, Z., Morariu, V., Darrell, T. & Davis, L. (2011b). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.
Felzenszwalb, P.F. & Schwartz, J.D. (2007). Hierarchical matching of deformable shapes. In: CVPR.
Ferecatu, M. (2005). Image retrieval with active relevance feedback using both visual and keyword-based descriptors. PhD thesis, Université de Versailles SaintQuentin-en-Yvelines.
Fergus, R., Bernal, H., Weiss, Y. & Torralba, A. (2010). Semantic label sharing for learning with many categories. In: ECCV (1), pp 762–775.
Fernández, A., & Gómez, S. (2008). Solving non-uniqueness in agglomerative hierarchical clustering using multidendrograms. J Classification, 25(1), 43–65.
Goëau, H., Bonnet, P., Joly, A., Boujemaa, N., Barthelemy, D., Molino, J.F., Birnbaum, P., Mouysset, E. & Picard, M. (2011). The clef 2011 plant images classification task. In: CLEF (Notebook Papers/Labs/Workshop).
Goëau, H., Bonnet, P., Joly, A., Yahiaoui, I., Barthelemy, D., Boujemaa, N. & Molino, J.F. (2012). The imageclef 2012 plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).
Grall-Maës, E., & Beauseroy, P. (2009). Optimal decision rule with class-selective rejection and performance constraints. IEEE Trans Pattern Anal Mach Intell, 31(11), 2073–2082.
Gu, X., Du, J.X. & Wang, X. (2005). Leaf recognition based on the combination of wavelet transform and gaussian interpolation. In: ICIC (1), pp 253–262.
Gupta, S. S. (1965). On some multiple decision (selection and ranking) rules. Technometrics, 7(2), 225–245.
Ha, T. M. (1997). The optimum class-selective rejection rule. IEEE Trans Pattern Anal Mach Intell, 19(6), 608–615.
Horiuchi, T. (1998). Class-selective rejection rule to minimize the maximum distance between selected classes. Pattern Recognition, 31(10), 1579–1588.
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Comput Surv, 31(3), 264–323.
Jr, C. N. S., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Min Knowl Discov, 22(1–2), 31–72.
Kumar, N., Belhumeur, P.N., Biswas, A., Jacobs, D.W., Kress, W.J., Lopez, I.C. & Soares, J.V.B. (2012). Leafsnap: A computer vision system for automatic plant species identification. In: ECCV (2), pp 502–516.
Larios, N., Deng, H., Zhang, W., Sarpola, M., Yuen, J., Paasch, R., et al. (2008). Automated insect identification through concatenated histograms of local appearance features: feature vector generation and region detection for deformable objects. Mach Vis Appl, 19(2), 105–123.
Lazebnik, S., Schmid, C. & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: CVPR (2), pp 2169–2178.
Lee, P. (1989). Bayesian statistics: an introduction. No. v. 2 in A Charles Griffin Book, Oxford University Press, http://books.google.fr/books?id=_hXvAAAAMAAJ
Li, F.F. & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In: CVPR (2), pp 524–531.
Ling, H., & Jacobs, D. W. (2007). Shape classification using the inner-distance. IEEE Trans Pattern Anal Mach Intell, 29(2), 286–299.
Liu, J., Kanazawa, A., Jacobs, D.W., Belhumeur, P.N. (2012). Dog breed classification using part localization. In: ECCV (1), pp 172–185.
Manh, A. G., Rabatel, G., Assemat, L., & Aldon, M. J. (2001). Weed leaf image segmentation by deformable templates. Journal of agricultural engineering research, 80(2), 139–146.
Martínez-Muñoz, G., Delgado, N.L., Mortensen, E.N., Zhang, W., Yamamuro, A., Paasch, R., Payet, N., Lytle, D.A., Shapiro, L.G., Todorovic, S., Moldenke, A. & Dietterich, T.G. (2009). Dictionary-free categorization of very similar objects via stacked evidence trees. In: CVPR, pp 549–556.
Mouine, S., Yahiaoui, I., Verroust-Blondet, A. (2013). A shape-based approach for leaf classification using multiscaletriangular representation. In: ICMR, pp 127–134.
Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London Series A, Mathematical and Physical Sciences 236(767):pp. 333–380, http://www.jstor.org/stable/91337
Nilsback, M.E. & Zisserman, A. (2006). A visual vocabulary for flower classification. In: CVPR (2), pp 1447–1454.
Otsu, N. (1979). A Threshold Selection Method from Gray-level Histograms. Man and Cybernetics: IEEE Transactions on Systems.
Rejeb Sfar, A., Boujemaa, N. & Geman, D. (2013a). Identification of plants from multiple images and botanical idkeys. In: ICMR, pp 191–198.
Rejeb Sfar, A., Boujemaa, N., Geman, D. (2013b). Vantage feature frames for fine-grained categorization. In: CVPR, pp 835–842.
Söderkvist, O. (2001). Computer vision classification of leaves from swedish trees. Master’s thesis, Linköping University, SE-581 83 Linköping, Sweden, liTH-ISY-EX-3132.
Teng, C.H., Kuo, Y.T. & Chen, Y.S. (2009). Leaf segmentation, its 3d position estimation and leaf classification from a few images with very close viewpoints. In: ICIAR, pp 937–946.
Tversky, B. & Hemenway, K. (1984). Objects, parts, and categories. Experimental Psychology: General.
Wah, C., Branson, S., Perona, P. & Belongie, S. (2011). Multiclass recognition and part localization with humans in the loop. In: ICCV, pp 2524–2531.
Wang, J., Yang, J., Yu, K., Lv, F., Huang, T. & Gong, Y. (2010). Locality-constrained linear coding for image classification. In: CVPR, pp 3360–3367.
Wang, X., Du, J.X. & Zhang, G.J. (2005). Recognition of leaf images based on shape features using a hypersphere classifier. In: ICIC (1), pp 87–96.
Wang, X. F., Huang, D. S., Du, J. X., Xu, H., & Heutte, L. (2008). Classification of plant leaf images with complicated background. Applied Mathematics and Computation, 205(2), 916–926.
Ward, J, Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244.
Wu, J. & Rehg, J.M. (2008). Where am i: Place instance and category recognition using spatial pact. In: CVPR.
Wu, S.G., Bao, F.S., Xu, E.Y., Wang, Y., Chang, Y.F. & Xiang, Q.L. (2007). A leaf recognition algorithm for plant classification using probabilistic neural network. CoRR abs/0707.4289.
Yang, S., Bo, L., Wang, J. & Shapiro, L.G. (2012). Unsupervised template learning for fine-grained object recognition. In: NIPS, pp 3131–3139.
Yao, B., Bradski, G.R. & Li, F.F. (2012). A codebook-free and annotation-free approach for fine-grained image categorization. In: CVPR, pp 3466–3473.
Yuan, M., & Wegkamp, M. H. (2010). Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11, 111–130.
Zhang, N., Farrell, R. & Darrell, T. (2012). Pose pooling kernels for sub-category recognition. In: CVPR, pp 3665–3672.
Zweig, A. & Weinshall, D. (2007). Exploiting object hierarchy: Combining models from different category levels. In: ICCV, pp 1–8.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Derek Hoiem.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Rejeb Sfar, A., Boujemaa, N. & Geman, D. Confidence Sets for Fine-Grained Categorization and Plant Species Identification. Int J Comput Vis 111, 255–275 (2015). https://doi.org/10.1007/s11263-014-0743-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-014-0743-3