Unsupervised disambiguation of image captions
Abstract
Given a set of images with related captions, our goal is to show how visual features can improve the accuracy of unsupervised word sense disambiguation when the textual context is very small, as this sort of data is common in news and social media. We extend previous work in unsupervised text-only disambiguation with methods that integrate text and images. We construct a corpus by using Amazon Mechanical Turk to caption sense-tagged images gathered from ImageNet. Using a Yarowsky-inspired algorithm, we show that gains can be made over text-only disambiguation, as well as multimodal approaches such as Latent Dirichlet Allocation.
References
[1]
Kobus Barnard and Matthew Johnson. 2005. Word sense disambiguation with pictures. In Artificial Intelligence, volume 167, pages 13--130.
[2]
Kobus Barnard, Matthew Johnson, and David Forsyth. 2003. Word sense disambiguation with pictures. In Workshop on Learning Word Meaning from Non-Linguistic Data, Edmonton, Canada.
[3]
David M. Blei, Andrew Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. In JMLR, volume 3, pages 993--1022.
[4]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition.
[5]
Pinar Duygulu, Kobus Barnard, Nando de Freitas, and David Forsyth. 2002. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In European Conference on Computer Vision, Copenhagen, Denmark.
[6]
Christiane Fellbaum. 1998. Wordnet: An electronic lexical database. In Bradford Books.
[7]
Yansong Feng and Mirella Lapata. 2010. Topic models for image annotation and text illustration. In Annual Conference of the North American Chapter of the ACL, pages 831--839, Los Angeles, California.
[8]
Michael Jamieson, Afsaneh Fazly, Suzanne Stevenson, Sven Dickinson, and Sven Wachsmuth. 2009. Using language to learn structured appearance models for image annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(1): 148--164.
[9]
Chee Wee Leong and Rada Mihalcea. 2011. Measuring the semantic relatedness between words and images. In International Conference on Semantic Computing, Oxford, UK.
[10]
Nicolas Loeff, Cecilia Ovesdotter Alm, and David Forsyth. 2006. Discriminating image senses by clustering with multimodal features. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 547--554, Sydney, Australia.
[11]
David Lowe. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91--110.
[12]
George Miller, Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. A semantic concordance. In Proceedings of the 3rd DARPA Workshop on Human Language Technology, pages 303--308.
[13]
Siddharth Patwardhan, Satanjeev Banerjee, and Ted Pedersen. 2007. UMND1: Unsupervised word sense disambiguation using contextual semantic relatedness. In Proceedings of SemEval-2007, pages 390--393, Prague, Czech Republic.
[14]
Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. Task 17: English lexical sample, SRL and all words. In Proceedings of SemEval-2007, pages 87--92, Prague, Czech Republic.
[15]
Kate Saenko and Trevor Darrell. 2008. Unsupervised learning of visual sense models for polysemous words. In Proceedings of Neural Information Processing Systems, Vancouver, Canada.
[16]
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting of the ACL, pages 189--196, Cambridge, Massachusetts.
- Unsupervised disambiguation of image captions
Recommendations
An unsupervised method for word sense disambiguation
AbstractWord sense disambiguation (WSD) finds the actual meaning of a word according to its context. This paper presents a novel WSD method to find the correct sense of a word present in a sentence. The proposed method uses both the WordNet ...
Unsupervised Word-Sense Disambiguation Using Bilingual Comparable Corpora
An unsupervised method for word-sense disambiguation using bilingual comparable corpora was developed. First, it extracts word associations, i.e., statistically significant pairs of associated words, from the corpus of each language. Then, it aligns ...
Unsupervised word sense disambiguation using bilingual comparable corpora
COLING '02: Proceedings of the 19th international conference on Computational linguistics - Volume 1An unsupervised method for word sense disambiguation using a bilingual comparable corpus was developed. First, it extracts statistically significant pairs of related words from the corpus of each language. Then, aligning pairs of related words ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Publisher
Association for Computational Linguistics
United States
Publication History
Published: 07 June 2012
Qualifiers
- Research-article
Acceptance Rates
Overall Acceptance Rate 8 of 31 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 119Total Downloads
- Downloads (Last 12 months)39
- Downloads (Last 6 weeks)4
Reflects downloads up to 18 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in