Guess What? A Game for Affective Annotation of Video Using Crowd Sourcing

1. von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 319–326. ACM, New York (2004)

   Google Scholar 

2. Baron-Cohen, S., Golan, O., Wheelwright, S., Hill, J.J.: Mind reading: the interactive guide to emotions (2004), http://www.jkp.com/mindreading

3. Bavelas, J., McGee, D., Phillips, B., Routledge, R.: Microanalysis of communication in psychotherapy. Human Systems 11(1), 47–66 (2000)

   Google Scholar 

4. Bernstein, M., Chi, E., Chilton, L., Hartmann, B., Kittur, A., Miller, R.: Crowdsourcing and Human Computation: Systems, Studies and Platforms (May 2011), http://crowdresearch.org/chi2011-workshop/ (last accessed April 15, 2011)

5. Burke, M.A., Young, P.: Norms, Customs, and Conventions. In: Benhabib, J., Bisin, A., Jackson, M. (eds.) Handbook for Social Economics. Elsevier, Amsterdam (2010)

   Google Scholar 

6. Calvo, R., D’Mello, S.: Affect Detection: An Interdisciplinary Review of Models, Methods, and their Applications. IEEE Transactions on Affective Computing 1(1), 18–37 (2010)

   Article  Google Scholar 

7. Ganjisaffar, Y., Javanmardi, S., Lopes, C.: Leveraging crowdsourcing heuristics to improve search in wikipedia. In: Proceedings of the 5th International Symposium on Wikis and Open Collaboration, WikiSym 2009, pp. 27:1–27:2. ACM, New York (2009)

   Google Scholar 

8. Golan, O., Baron-Cohen, S.: Systemizing empathy: Teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia. Development and Psychopathology 18(02), 591–617 (2006)

   Article  Google Scholar 

9. Hayes, A.: SPSS, SAS, and Mplus Macros and Code (2011), http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html (last accessed April 15, 2011)

10. Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Communication Methods and Measures (2007)

    Google Scholar 

11. Hsueh, P.Y., Melville, P., Sindhwani, V.: Data quality from crowdsourcing: a study of annotation selection criteria. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, HLT 2009, pp. 27–35. Association for Computational Linguistics (2009)

    Google Scholar 

12. Kazai, G., Lease, M.: TREC 2011 Crowdsourcing Track (November 2011), https://sites.google.com/site/treccrowd2011 (last accessed April 15, 2011)

13. Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. Sage Publications, Thousand Oaks (2004)

    Google Scholar 

14. McCreadie, R., Macdonald, C., Ounis, I.: Crowdsourcing a news query classification dataset. In: Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pp. 31–38 (2010)

    Google Scholar 

15. Nowak, S., Rüger, S.: How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: Proceedings of the International Conference on Multimedia Information Retrieval, pp. 557–566. ACM, New York (2010)

    Chapter  Google Scholar 

16. O’Connor, M.F.: Automatic Understanding of Social Scenes. Master’s thesis, University of Cambridge (2011)

    Google Scholar 

17. Pantic, M., Rothkrantz, L.: Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91(9), 1370–1390 (2003)

    Article  Google Scholar 

18. Philippot, P., Feldman, R., Coats, E.: The social context of nonverbal behavior. Cambridge Univ. Pr., Cambridge (1999)

    Google Scholar 

19. Riek, L.D., Robinson, P.: Challenges and opportunities in building socially intelligent machines. IEEE Signal Processing (2011)

    Google Scholar 

20. Scherer, K., Banziger, T., Roesch, E.: A blueprint for affective computing: A sourcebook. Oxford University Press, Oxford (2010)

    Google Scholar 

21. Sheng, V., Provost, F., Ipeirotis, P.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614–622. ACM, New York (2008)

    Chapter  Google Scholar 

22. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.: Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254–263. Association for Computational Linguistics (2008)

    Google Scholar 

23. Tarasov, A., Delany, S., Cullen, C.: Using crowdsourcing for labelling emotional speech assets. In: W3C workshop on Emotion ML (2010)

    Google Scholar 

24. Yan, R., Yang, J., Hauptmann, A.: Automatically labeling video data using multi-class active learning. In: 9th IEEE International Conference on Computer Vision, pp. 516–523. IEEE, Los Alamitos (2003)

    Chapter  Google Scholar 

Download references