Deepfake detection using convolutional vision transformers and convolutional neural networks
Authors: 
Ahmed Hatem Soudy, Omnia Sayed, Hala Tag-Elser, Rewaa Ragab, Sohaila Mohsen, Tarek Mostafa, Amr A. Abohany, Salwa O. SlimAuthors Info & Claims
Pages 19759 - 19775
Abstract
Deepfake technology has rapidly advanced in recent years, creating highly realistic fake videos that can be difficult to distinguish from real ones. The rise of social media platforms and online forums has exacerbated the challenges of detecting misinformation and malicious content. This study leverages many papers on artificial intelligence techniques to address deepfake detection. This research proposes a deep learning (DL)-based method for detecting deepfakes. The system comprises three components: preprocessing, detection, and prediction. Preprocessing includes frame extraction, face detection, alignment, and feature cropping. Convolutional neural networks (CNNs) are employed in the eye and nose feature detection phase. A CNN combined with a vision transformer is also used for face detection. The prediction component employs a majority voting approach, merging results from the three models applied to different features, leading to three individual predictions. The model is trained on various face images using FaceForensics++ and DFDC datasets. Multiple performance metrics, including accuracy, precision, F1, and recall, are used to assess the proposed model’s performance. The experimental results indicate the potential and strengths of the proposed CNN that achieved enhanced performance with an accuracy of 97%, while the CViT-based model achieved 85% using the FaceForences++ dataset and demonstrated significant improvements in deepfake detection compared to recent studies, affirming the potential of the suggested framework for detecting deepfakes on social media. This study contributes to a broader understanding of CNN-based DL methods for deepfake detection.
References
[1]
Al-Hussein M, Venkataraman S, Jawahar C (2020) Deepfake detection for video: an open source challenge. arXiv preprint arXiv:2006.06058
[2]
Altwaijry N and Al-Turaiki I Arabic handwriting recognition system using convolutional neural network Neural Comput Appl 2021 33 7 2249-2261
[3]
Arif MS, Mukheimer A, and Asif D Enhancing the early detection of chronic kidney disease: a robust machine learning model Big Data Cogn Comput 2023 7 3 144
[4]
Asif D, Bibi M, Arif MS, and Mukheimer A Enhancing heart disease prediction through ensemble learning techniques with hyperparameter optimization Algorithms 2023 16 6 308
[5]
Desai M and Shah M An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (mlp) and convolutional neural network (cnn) Clin eHealth 2021 4 1-11
[6]
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
[7]
Dua M, Shakshi Singla R, Raj S, and Jangra A Deep cnn models-based ensemble approach to driver drowsiness detection Neural Comput Appl 2021 33 3155-3168
[8]
Hasan MJ et al (2020) Understanding the influence of epochs and learning rate on deep learning-based sentiment analysis. In: Proceedings of the international conference on artificial intelligence and applications, pp 553–561
[9]
Ismail A, Elpeltagy M, Zaki SM, and Eldahshan K A new deep learning-based methodology for video deepfake detection using xgboost Sensors 2021 21 16 5413
[10]
Jena B, Nayak GK, and Saxena S Convolutional neural network and its pretrained models for image classification and object detection: a survey Concurr Comput Pract Exp 2022 34 6
[11]
Jing TW, Murugesan RK (2021) Protecting data privacy and prevent fake news and deepfakes in social media via blockchain technology. In: Advances in cyber security: second international conference, ACeS 2020, Penang, Malaysia, December 8–9, 2020, Revised Selected Papers 2. Springer, Berlin, pp 674–684
[12]
Karandikar A, Deshpande V, Singh S, Nagbhidkar S, and Agrawal S Deepfake video detection using convolutional neural network Int J Adv Trends Comput Sci Eng 2020 9 2 1311-1315
[13]
Karandikar A, Thakare Y, Sah O, Sah R, Nafde S, and Kumar S Detection of deepfake video using residual neural network and long short-term memory Int J Next-Gener Comput 2023
[14]
Kfir O, Kfir A, Friedman A (2019) Generative adversarial attacks in social media. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & sata mining, pp 2558–2566
[15]
Kumar A, Bhavsar A, Verma R (2020) Detecting deepfakes with metric learning. In: 2020 8th international workshop on biometrics and forensics (IWBF). IEEE, pp 1–6
[16]
Li Y, Chang MC, Lyu S (2018) In ictu oculi: exposing ai created fake videos by detecting eye blinking. In: 2018 IEEE International workshop on information forensics and security (WIFS). IEEE, pp 1–7
[17]
Li Y, Yang X, Sun P, Qi H, Lyu S (2020) Celeb-df: a large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3207–3216
[18]
Mitra A, Mohanty SP, Corcoran P, and Kougianos E A machine learning based approach for deepfake detection in social media through key video frame extraction SN Comput Sci 2021 2 1-18
[19]
Mogan JN, Lee CP, Lim KM, Ali M, and Alqahtani A Gait-cnn-vit: multi-model gait recognition with convolutional neural networks and vision transformer Sensors 2023 23 8 3809
[20]
Nawaz Y, Arif MS, Shatanawi W, and Nazeer A An explicit fourth-order compact numerical scheme for heat transfer of boundary layer flow Energies 2021 14 12 3396
[21]
Nawaz Y, Arif MS, and Abodayeh K An explicit-implicit numerical scheme for time fractional boundary layer flows Int J Numer Methods Fluids 2022 94 7 920-940
[22]
Nawaz Y, Arif MS, and Abodayeh K A third-order two-stage numerical scheme for fractional stokes problems: a comparative computational study J Comput Nonlinear Dyn 2022 17 10
[23]
Nguyen TT, Nguyen QVH, Nguyen DT, Nguyen DT, Huynh-The T, Nahavandi S, Nguyen TT, Pham QV, and Nguyen CM Deep learning for deepfakes creation and detection: a survey Comput Vis Image Underst 2022 223
[24]
Rafique R, Gantassi R, Amin R, Frnda J, Mustapha A, and Alshehri AH Deep fake detection and classification using error-level analysis and deep learning Sci Rep 2023 13 1 7422
[25]
Sabir E, Cheng J, Jaiswal A, AbdAlmageed W, Masi I, and Natarajan P Recurrent convolutional strategies for face manipulation detection in videos Interfaces (GUI) 2019 3 1 80-87
[26]
Singh KA, Vats O, Shankar M (2022) Deepfake detection using deep learning methods: a systematic and comprehensive review. Wiley Interdiscip Rev Data Min Knowl Discov 14(3):e1520.
[27]
Taeb M and Chi H Comparison of deepfake detection techniques through deep learning J Cybersecur Priv 2022 2 1 89-106
[28]
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in Neural Information Processing Systems 30 (NIPS 2017)
[29]
Wang Y, Zhang Z, Feng L, Ma Y, and Du Q A new attention-based cnn approach for crop mapping using time series sentinel-2 images Comput Electron Agric 2021 184
[30]
Wodajo D, Atnafu S (2021) Deepfake video detection using convolutional vision transformer. arXiv preprint arXiv:2102.11126
[31]
Xie J, Hua J, Chen S, Wu P, Gao P, Sun D, Lyu Z, Lyu S, Xue X, and Lu J Hypersformer: a transformer-based end-to-end hyperspectral image classification method for crop classification Remote Sens 2023 15 14 3491
[32]
Yamashita R, Nishio M, Do RKG, and Togashi K Convolutional neural networks: an overview and application in radiology Insights Imaging 2018 9 611-629
[33]
Yang Z, Liang J, Xu Y, Zhang XY, and He R Masked relation learning for deepfake detection IEEE Trans Inf Forensics Secur 2023 18 1696-1708
[34]
Yu P, Xia Z, Fei J, and Lu Y A survey on deepfake video detection IET Biometrics 2021 10 6 607-624
[35]
Zhang T Deepfake generation and detection, a survey Multimed Tools Appl 2022 81 5 6259-6276
[36]
Zhao M, Qin W, Yu W, Li X, and Gao B Adversarial examples: attacks on deepfake detection models IEEE Trans Multimed 2020 23 8 1295-1305
[37]
Zhou XY, Jiang WJ, and Yu ZH Deepfakes detection using deep learning methods: a survey Int J Imaging Robot 2020 14 3 196-211
Index Terms
- Deepfake detection using convolutional vision transformers and convolutional neural networks
Index terms have been assigned to the content through auto-classification.
Recommendations
Food Detection and Recognition Using Convolutional Neural Network
MM '14: Proceedings of the 22nd ACM international conference on MultimediaIn this paper, we apply a convolutional neural network (CNN) to the tasks of detecting and recognizing food images. Because of the wide diversity of types of food, image recognition of food items is generally very difficult. However, deep learning has ...
Genetic algorithm‐optimised structure of convolutional neural network for face recognition applications
Proposing a proper method for face recognition is still a challenging subject in biometric and computer vision applications. Although some reliable systems were introduced under relatively controlled conditions, their recognition rate is not satisfactory ...
Research on improved wavelet convolutional wavelet neural networks
AbstractConvolutional neural network (CNN) is recognized as state of the art of deep learning algorithm, which has a good ability on the image classification and recognition. The problems of CNN are as follows: the precision, accuracy and efficiency of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 08 August 2024
Accepted: 01 July 2024
Received: 19 August 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
- Helwan University
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Jan 2025