Abstract
Deep recurrent models applied in Image Caption, which link up computer vision and natural language processing, have achieved excellent results enabling automatically generating natural sentences describing an image. However, the mismatch of sample distribution between training data and the open world may leads to tons of hiding-in-dark Unknown Unknowns (UUs). And such errors may greatly harm the correctness of generated captions. In this paper, we present a framework targeting on UUs reduction and model optimization based on recurrently training with small amounts of external data detected under assistance of crowd commonsense. We demonstrate and analyze our method with currently state-of-the-art image-to-text model. Aiming at reducing the number of UUs in generated captions, we obtain over 12% of UUs reduction and reinforcement of model cognition on these scenes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lakkaraju, H., Kamar, E., Caruana, R., et al.: Identifying unknown unknowns in the open world: representations and policies for guided exploration. In: AAAI 2017, pp. 2124–2132 (2017)
Vinyals, O., Toshev, A., Bengio, S., et al.: Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 652–663 (2016)
Devlin, J., Gupta, S., Girshick, R., et al.: Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467 (2015)
Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CIDEr: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Krishna, R.A., Hata, K., Chen, S., et al.: Embracing error to enable rapid crowdsourcing. In: CHI Conference on Human Factors in Computing Systems, pp. 3167–3179. ACM (2016)
Horvitz, E.: Artificial intelligence in the open world. Presidential Address. AAAI (2008). http://bit.ly/2gCN7t9
Attenberg, J., Ipeirotis, P., Provost, F.: Beat the machine: challenging humans to find a predictive model’s unknown unknowns. J. Data Inf. Qual. 6(1), 1–17 (2015)
Davis, E., Marcus, G.: Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58(9), 92–103 (2015)
Cambria, E., Olsher, D., Rajagopal, D.: SenticNet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis. In: AAAI, pp. 1515–1521 (2014)
Donahue, J., Hendricks, L.A., Guadarrama, S., et al.: Long-term recurrent convolutional networks for visual recognition and description. AB initto calculation of the structures and properties of molecules, pp. 85–91. Elsevier (2014)
Fang, H., Platt, J.C., Zitnick, C.L., et al.: From captions to visual concepts and back. In: Computer Vision and Pattern Recognition, pp. 1473–1482. IEEE (2015)
Karpathy, A., Li, F.F.: Deep visual-semantic alignments for generating image descriptions. In: Computer Vision and Pattern Recognition, pp. 3128–3137. IEEE (2015)
Acknowledgement
This research is funded by the National Key Technology Support Program (No. 2015BAH01F02), the National Nature Science Foundation of China (No. 61602179) and the Natural Science Foundation of Shanghai (No. 17ZR1444900).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ni, M., Yang, J., Lin, X., He, L. (2017). Reducing Unknown Unknowns with Guidance in Image Caption. In: Lintas, A., Rovetta, S., Verschure, P., Villa, A. (eds) Artificial Neural Networks and Machine Learning – ICANN 2017. ICANN 2017. Lecture Notes in Computer Science(), vol 10614. Springer, Cham. https://doi.org/10.1007/978-3-319-68612-7_62
Download citation
DOI: https://doi.org/10.1007/978-3-319-68612-7_62
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-68611-0
Online ISBN: 978-3-319-68612-7
eBook Packages: Computer ScienceComputer Science (R0)