[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Inferring Personality Traits from Attentive Regions of User Liked Images Via Weakly Supervised Dual Convolutional Network

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In social media, users usually unconsciously their preferences on images, which can be considered as the personal cues for inferring their personality traits. Existing methods map the holistic image features into personality traits. However, users’ attention on their liked images is typically localized, which should be taken into account in modeling personality traits. In this paper, we propose an end-to-end weakly supervised dual convolutional network (WSDCN) for personality prediction, which consists of a classification network and a regression network. The classification network captures personality class-specific attentive image regions while only requiring the image-level personality class labels. The regression network is used for predicting personality traits. Firstly, the users’ Big-Five (BF) traits are converted into ten personality class labels for their liked images. Secondly, the Multi-Personality Class Activation Map (MPCAM) is generated based on the classification network and utilized as the localized activation to produce local deep features, which are then combined with the holistic deep features for the regression task. Finally, the user liked images and the associated personality traits are used to train the end-to-end WSDCN model. The proposed method is able to predict the BF personality traits simultaneously by training the WSDCN network only once. Experimental results on the annotated PsychoFlickr database show that the proposed method is superior to the state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Matthews G, Deary I, Whiteman M (2009) Personality traits. Cambridge University Press, Cambridge

    Google Scholar 

  2. Furnham A, Jackson CJ, Miller T (1999) Personality, learning style and work performance. Personal Individ Differ 27(6):1113–1122

    Google Scholar 

  3. Guntuku SC, Roy S, Lin W (2015) Personality modeling based image recommendation. In: Proceedings of the international conference on multimedia modeling, pp 171–182

  4. Guntuku SC, Yaden DB, Kern ML, Ungar LH, Eichstaedt JC (2017) Detecting depression and mental illness on social media: an integrative review. Curr Opin Behav Sci 18:43–49

    Google Scholar 

  5. House VN (2011) Personal photography, digital technologies and the uses of the visual. Vis Stud 26(2):125–134

    Google Scholar 

  6. Zhao S, Gao Y, Ding G, Chua TS (2018) Real-time multimedia social event detection in microblog. IEEE Trans Cybern 48(11):3218–3231

    Google Scholar 

  7. Deng C, Chen Z, Liu X, Gao X, Tao D (2018) Triplet-based deep hashing network for cross-modal retrieval. IEEE Trans Image Process 27(8):3893–3903

    MathSciNet  MATH  Google Scholar 

  8. Li C, Deng C, Li N, Liu W, Gao X, Tao D (2018) Self-supervised adversarial hashing networks for cross-modal retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4242–4251

  9. Yang E, Deng C, Li C, Liu W, Li J, Tao D (2018) Shared predictive cross-modal deep quantization. IEEE Trans Neural Netw 99:1–12

    Google Scholar 

  10. Joshi D, Datta R, Fedorovskaya E, Luong Q (2011) Aesthetics and emotions in images. IEEE Signal Proc Mag 28(5):94–115

    Google Scholar 

  11. Zhao S, Yao H, Gao Y, Ding G, Chua TS (2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9(4):526–540

    Google Scholar 

  12. Zhu H, Li L, Zhao S, Jiang H (2018) Evaluating attributed personality traits from scene perception probability. Pattern Recognit Lett 116:121–126

    Google Scholar 

  13. Zhao S, Ding G, Han J, Gao Y (2018) Personality-aware personalized emotion recognition from physiological signals. In: Proceedings of the international joint conferences on artificial intelligence, pp 1660–1667

  14. Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the ACM international conference on multimedia, pp 83–92

  15. Zhao S, Gao Y, Jiang X, Yao H, Chua TS , Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the ACM international conference on multimedia, pp 47–56

  16. Peng KC, Chen T, Sadovnik A, Gallagher AC (2015) A mixed bag of emotions: model, predict, and transfer emotion distributions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 860–868

  17. You Q, Luo J, Jin H, Yang J (2016) Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: Proceedings of the AAAI conference on artificial intelligence, pp 308–314

  18. Cristani M, Vinciarelli A, Segalin C, Perina A (2013) Unveiling the multimedia unconscious: implicit cognitive processes and multimedia content analysis. In: Proceedings of the ACM international conference on multimedia, pp 213–222

  19. Segalin C, Cristani M, Perina A, Vinciarelli A (2017) The pictures we like are our image: continuous mapping of favorite pictures into self-assessed and attributed personality traits. IEEE Trans Affect Comput 8(2):268–285

    Google Scholar 

  20. Guntuku SC, Zhou JT, Roy S, Lin WS, Tsang IW (2018) Who likes what, and why? Insights into personality modeling based on image ‘likes’. IEEE Trans Affect Comput 9(1):130–143

    Google Scholar 

  21. Vinciarelli A, Mohammadi G (2014) A survey of personality computing. IEEE Trans Affect Comput 5(3):273–291

    Google Scholar 

  22. Goldberg LR (1993) The structure of phenotypic personality traits. Am Psychol 48(1):26–34

    Google Scholar 

  23. Goldberg LR (1990) An alternative “description of personality”: the big-five factor structure. J Pers Soc Psychol 59(6):1216

    Google Scholar 

  24. Rammstedt B, John O (2007) Measuring personality in one minute or less: a 10-item short version of the big five inventory in English and German. J Res Pers 41(1):203–212

    Google Scholar 

  25. Jenkins R (2014) Social identity. Routledge 6(1):1396

    Google Scholar 

  26. Zen G, Lepri B, Ricci E, Lanz O (2010) Space speaks: towards socially and personality aware visual surveillance. In: Proceedings of the ACM international workshop on multimodal pervasive video analysis, pp 37–42

  27. Pianesi F, Mana N, Cappelletti A, Lepri B, Zancanaro M (2008) Multimodal recognition of personality traits in social interactions. In: Proceedings of the international conference on multimodal interfaces, pp 53–60

  28. Wei X, Zhang C, Zhang H, Wu J (2018) Deep bimodal regression of apparent personality traits from short video sequences. IEEE Trans Affect Comput 9(3):303–315

    Google Scholar 

  29. Segalin C, Dong SC, Cristani M (2017) Social profiling through image understanding: personality inference using convolutional neural networks. Comput Vis and Image Und 156:34–50

    Google Scholar 

  30. Kosinski M, Stillwell D, Graepel T (2013) Private traits and attributes are predictable from digital records of human behavior. Proc Natl Acad Sci 110(15):5802–5805

    Google Scholar 

  31. Tibshirani R (2011) Regression shrinkage and selection via the lasso: a retrospective. J Roy Stat Soc 73(3):273–282

    MathSciNet  MATH  Google Scholar 

  32. Peng KC, Sadovnik A, Gallagher A, Chen T (2016) Where do emotions come from? Predicting the emotion stimuli map. In: Proceedings of the IEEE international conference on image processing, pp 614–618

  33. You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions. In: Proceedings of the AAAI conference on artificial intelligence, pp 231–237

  34. Yang J, She D, Lai YK, Rosin P, Yang MH (2018) Weakly supervised coupled networks for visual sentiment analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 231–237

  35. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the international conference on neural information processing systems, pp 1097–1105

  36. Deng C, Liu X, Li C, Tao D (2018) Active multi-kernel domain adaptation for hyperspectral image classification. Pattern Recognit 77:306–315

    Google Scholar 

  37. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

  38. Diba A, Sharma V, Pazandeh A, Pirsiavash H, Gool LV (2017) Weakly supervised cascaded convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5131–5139

  39. Durand T, Mordan T, Thome N, Cord M (2017) Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5957–5966

  40. Zhao S, Ding G, Gao Y, Zhao X, Tang Y, Han J (2018) Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2018.2818685

  41. Yang J, She D, Sun M (2017) Joint image emotion classification and distribution learning via deep convolutional neural network. In: Proceedings of the international joint conference on artificial intelligence, pp 3266–3272

  42. Zhao S, Yao H, Gao Y, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multi-task shared sparse regression. IEEE Trans Multimedia 19(3):632–645

    Google Scholar 

  43. Zhao S, Zhao X, Ding G, Keutzer, K (2018) EmotionGAN: unsupervised domain adaptation for learning discrete probability distributions of image emotions. In: Proceedings of ACM multimedia conference on multimedia conference, pp 1319–1327

  44. Zhao S, Ding G, Gao Y, Han J (2017) Approximating discrete probability distribution of image emotions by multi-modal features fusion. Transfer 1000(1)

  45. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Li FF (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):1–42

    MathSciNet  Google Scholar 

  46. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Comput Sci

  47. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  48. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C (2016) Tensorflow: large-scale machine learning on heterogeneous distributed systems, arXiv preprint. arXiv:1603.04467

Download references

Acknowledgements

This work was supported by Outstanding Innovation Scholarship for Doctoral Candidate of “Double First Rate” Construction Disciplines of CUMT.

Author information

Authors and Affiliations

  1. School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116, China

    Hancheng Zhu & Leida Li

  2. School of Management, China University of Mining and Technology, Xuzhou, 221116, China

    Hongyan Jiang

  3. Tencent Media Lab, Tencent, Shenzhen, 518000, China

    Allen Tan

Authors
  1. Hancheng Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leida Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongyan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Allen Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leida Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, H., Li, L., Jiang, H. et al. Inferring Personality Traits from Attentive Regions of User Liked Images Via Weakly Supervised Dual Convolutional Network. Neural Process Lett 51, 2105–2121 (2020). https://doi.org/10.1007/s11063-019-09987-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-019-09987-7

Keywords

Search

Navigation