More Web Proxy on the site http://driver.im/

research-article

Open access

Human Attributes Prediction under Privacy-preserving Conditions

Authors:

Mohan KankanhalliAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 4698 - 4706

https://doi.org/10.1145/3474085.3475687

Published: 17 October 2021 Publication History

Abstract

Human attributes prediction in visual media is a well-researched topic with a major focus on human faces. However, face images are often of high privacy concern as they can reveal an individual's identity. How to balance this trade-off between privacy and utility is a key problem among researchers and practitioners. In this study, we make one of the first attempts to investigate the human attributes (emotion, age, and gender) prediction under the different de-identification (eyes, lower-face, face, and head obfuscation) privacy scenarios. We first constructed the Diversity in People and Context Dataset (DPaC). We then performed a human study with eye-tracking on how humans recognize facial attributes without the presence of face and context. Results show that in an image, situational context is informative of a target's attributes. Motivated by our human study, we proposed a multi-tasking deep learning model - Context-Guided Human Attributes Prediction (CHAPNet), for human attributes prediction under privacy-preserving conditions. Extensive experiments on DPaC and three commonly used benchmark datasets demonstrate the superiority of CHAPNet in leveraging the situational context for a better interpretation of a target's attributes without the full presence of the target's face. Our research demonstrates the feasibility of visual analytics under de-identification for privacy.

Supplementary Material

ZIP File (mfp2883aux.zip)

This supplementary material provides additional details and examples to complement the main paper. We organize the appendix according to the sections of the main paper for ease of reference.

Download
6.66 MB

References

[1]

Nihad A Abdalrady and Saleh Aly. 2020. Fusion of Multiple Simple Convolutional Neural Networks for Gender Classification. In 2020 International Conference on Innovative Trends in Communication and Computer Engineering (ITCE). IEEE, 251--256.

[2]

Lior Abramson, Rotem Petranker, Inbal Marom, and Hillel Aviezer. 2020. Social interaction context shapes emotion recognition through body language, not facial expressions. Emotion (2020).

[3]

Irwin Altman. 1975. The environment and social behavior: privacy, personal space, territory, and crowding. (1975).

[4]

Tadas Baltru?aitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10.

[5]

Lisa Feldman Barrett. 2012. Emotions are real. Emotion 12, 3 (2012), 413.

[6]

Irving Biederman. 1987. Recognition-by-components: a theory of human image understanding. Psychological review 94, 2 (1987), 115.

[7]

Abeba Birhane and Vinay Uday Prabhu. 2021. Large image datasets: A pyrrhic win for computer vision?. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1536--1546.

[8]

Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2016. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data? (2016).

[9]

Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91.

[10]

Zoya Bylinskii, Phillip Isola, Constance Bainbridge, Antonio Torralba, and Aude Oliva. 2015. Intrinsic and extrinsic effects on image memorability. Vision research 116 (2015), 165--178.

[11]

Antoni B Chan, Zhang-Sheng John Liang, and Nuno Vasconcelos. 2008. Privacy preserving crowd monitoring: Counting people without people models or tracking. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--7.

[12]

Zhimin Chen and David Whitney. 2019. Tracking the affective state of unseen persons. Proceedings of the National Academy of Sciences 116, 15 (2019), 7559--7564.

[13]

Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical reviewof fair machine learning. arXiv preprint arXiv:1808.00023 (2018).

[14]

Ji Dai, Behrouz Saghafi, Jonathan Wu, Janusz Konrad, and Prakash Ishwar. 2015. Towards privacy-preserving recognition of human activities. In 2015 IEEE international conference on image processing (ICIP). IEEE, 4238--4242.

Digital Library

[15]

Afshin Dehghan, Enrique G Ortiz, Guang Shu, and Syed Zain Masood. 2017. Dager: Deep age, gender and emotion recognition using convolutional neural network. arXiv preprint arXiv:1702.04280 (2017).

[16]

Jia Deng,Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.

[17]

Yuan Dong, Yinan Liu, and Shiguo Lian. 2016. Automatic age estimation based on deep learning algorithm. Neurocomputing 187 (2016), 4--10.

Digital Library

[18]

Andrew C Gallagher and Tsuhan Chen. 2009. Understanding images of groups of people. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 256--263.

[19]

Qinquan Gao, Hanxin Zeng, Gen Li, and Tong Tong. 2021. Graph Reasoning- Based Emotion Recognition Network. IEEE Access 9 (2021), 6488--6497.

[20]

Katharine H Greenaway, Elise K Kalokerinos, and Lisa A Williams. 2018. Context is everything (in emotion research). Social and Personality Psychology Compass 12, 6 (2018), e12393.

[21]

Edward T Hall. 1963. A system for the notation of proxemic behavior 1. American anthropologist 65, 5 (1963), 1003--1026.

[22]

Le Hou, Chen-Ping Yu, and Dimitris Samaras. 2016. Squared earth mover's distance-based loss for training deep neural networks. arXiv preprint arXiv:1611.05916 (2016).

[23]

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in'Real-Life'Images: detection, alignment, and recognition.

[24]

Jacob Israelashvili, Ran R Hassin, and Hillel Aviezer. 2019. When emotions run high: A critical role for context in the unfolding of dynamic, real-life facial affect. Emotion 19, 3 (2019), 558.

[25]

Kimmo Karkkainen and Jungseock Joo. 2021. FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1548--1558.

[26]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4401--4410.

[27]

Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. 2019. Context based emotion recognition using emotic dataset. IEEE transactions on pattern analysis and machine intelligence 42, 11 (2019), 2755--2766.

[28]

Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. 2019. Context-aware emotion recognition networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10143--10152.

[29]

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2018. Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15, 2018 (2018), 11.

[30]

Mohamed Loey, Gunasekaran Manogaran, Mohamed Hamed N Taha, and Nour Eldeen M Khalifa. 2021. A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic. Measurement 167 (2021), 108288.

[31]

Nikos K Logothetis and David L Sheinberg. 1996. Visual object recognition. Annual review of neuroscience 19, 1 (1996), 577--621.

[32]

Jordi Mansanet, Alberto Albiol, and Roberto Paredes. 2016. Local deep neural networks for gender recognition. Pattern Recognition Letters 70 (2016), 80--86.

Digital Library

[33]

Aleix M Martinez. 2019. Context may reveal how you feel. Proceedings of the National Academy of Sciences 116, 15 (2019), 7169--7171.

[34]

Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. 2019. Pose-guided feature alignment for occluded person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 542--551.

[35]

Trisha Mittal, Pooja Guhan, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege's Principle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14234--14243.

[36]

Nicolas Pinto, N Majaj, Youssef Barhomi, E Solomon, and JJ DiCarlo. 2010. Human versus machine: comparing visual object recognition systems on a level playing field. Cosyne Abstracts (2010).

[37]

Mary C Potter. 1976. Short-term conceptual memory for pictures. Journal of experimental psychology: human learning and memory 2, 5 (1976), 509.

[38]

Pau Rodríguez, Guillem Cucurull, Josep M Gonfaus, F Xavier Roca, and Jordi Gonzàlez. 2017. Age and gender recognition in the wild with deep attention. Pattern Recognition 72 (2017), 563--571.

Digital Library

[39]

Rasmus Rothe, Radu Timofte, and Luc Van Gool. 2015. Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE international conference on computer vision workshops. 10--15.

Digital Library

[40]

Michael Ryoo, Brandon Rothrock, Charles Fleming, and Hyun Jong Yang. 2017. Privacy-preserving human activity recognition from extreme low resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.

Digital Library

[41]

Caifeng Shan. 2010. Learning local features for age estimation on real-life faces. In Proceedings of the 1st ACM international workshop on Multimodal pervasive video analysis. 23--28.

Digital Library

[42]

Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536 (2017).

[43]

Simon Thorpe, Denis Fize, and Catherine Marlot. 1996. Speed of processing in the human visual system. nature 381, 6582 (1996), 520--522.

[44]

Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).

[45]

Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. 2020. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 24--25.

[46]

Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. 2018. Towards privacy-preserving visual recognition via adversarial training: A pilot study. In Proceedings of the European Conference on Computer Vision (ECCV). 606--624.

Digital Library

[47]

Tian Xu, Jennifer White, Sinan Kalkan, and Hatice Gunes. 2020. Investigating bias and fairness in facial expression recognition. In European Conference on Computer Vision. Springer, 506--523.

Digital Library

[48]

Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2021. A Study of Face Obfuscation in ImageNet. arXiv preprint arXiv:2103.06191 (2021).

[49]

Juha Ylioinas, Abdenour Hadid, and Matti Pietikäinen. 2012. Age classification in unconstrained conditions using LBP variants. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, 1257--1260.

[50]

Minghui Zhang, Yumeng Liang, and Huadong Ma. 2019. Context-aware affective graph reasoning for emotion recognition. In 2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 151--156.

[51]

Peng Zhang, Tony Thomas, Sabu Emmanuel, and Mohan S Kankanhalli. 2010. Privacy preserving video surveillance using pedestrian tracking mechanism. In Proceedings of the 2nd ACM workshop on Multimedia in forensics, security and intelligence. 31--36.

Digital Library

[52]

Zhifei Zhang, Yang Song, and Hairong Qi. 2017. Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5810--5818.

[53]

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40, 6 (2017), 1452--1464.

Cited By

Kansal KWong YKankanhalli M(2024)Implications of Privacy Regulations on Video Surveillance SystemsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3706108Online publication date: 28-Nov-2024
https://doi.org/10.1145/3706108
Xu DFan SKankanhalli MEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Combating Misinformation in the Era of Generative AI ModelsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612704(9291-9298)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612704
Qing LWen HChen HJin RCheng YPeng Y(2023)DVC-Net: a new dual-view context-aware network for emotion recognition in the wildNeural Computing and Applications10.1007/s00521-023-09040-836:2(653-665)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1007/s00521-023-09040-8
Show More Cited By

Index Terms

Human Attributes Prediction under Privacy-preserving Conditions
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
2. Security and privacy
  1. Human and societal aspects of security and privacy
    1. Privacy protections

Recommendations

Privacy Protection on Multiple Sensitive Attributes
Information and Communications Security
Abstract
In recent years, a privacy model called k-anonymity has gained popularity in the microdata releasing. As the microdata may contain multiple sensitive attributes about an individual, the protection of multiple sensitive attributes has become an ...
Hiding personalised anonymity of attributes using privacy preserving data mining

Privacy preserving data mining PPDM is a new direction in the area of data mining, where privacy preserving techniques have been applied to maintain the data privacy. Example through the process of data mining the sensitive data of an individual can be ...
Learnable Privacy-Preserving Anonymization for Pedestrian Images
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

This paper studies a novel privacy-preserving anonymization problem for pedestrian images, which preserves personal identity information (PII) for authorized models and prevents PII from being recognized by third parties. Conventional anonymization ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
673
Total Downloads

Downloads (Last 12 months)126
Downloads (Last 6 weeks)18

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kansal KWong YKankanhalli M(2024)Implications of Privacy Regulations on Video Surveillance SystemsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3706108Online publication date: 28-Nov-2024
https://doi.org/10.1145/3706108
Xu DFan SKankanhalli MEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Combating Misinformation in the Era of Generative AI ModelsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612704(9291-9298)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612704
Qing LWen HChen HJin RCheng YPeng Y(2023)DVC-Net: a new dual-view context-aware network for emotion recognition in the wildNeural Computing and Applications10.1007/s00521-023-09040-836:2(653-665)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1007/s00521-023-09040-8
Tang DZhou SJiang HChen HLiu Y(2022)Gender-Adversarial Networks for Face Privacy PreservingIEEE Internet of Things Journal10.1109/JIOT.2022.31558789:18(17568-17576)Online publication date: 15-Sep-2022
https://doi.org/10.1109/JIOT.2022.3155878
Li JBhat ABarmaki R(2022)Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions2022 26th International Conference on Pattern Recognition (ICPR)10.1109/ICPR56361.2022.9956680(762-769)Online publication date: 21-Aug-2022
https://doi.org/10.1109/ICPR56361.2022.9956680

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents