Pytorch implementation of the paper:
Towards Robust Multimodal Sentiment Analysis with Incomplete Data
This is a reorganized code, if you find any bugs please contact me. Thanks.
MOSI/MOSEI/CH-SIMS Download: Please see MMSA
The basic training environment for the results in the paper is Pytorch 2.2.1, Python 3.11.7 with NVIDIA Tesla A40.
You can quickly run the code with the following command:
bash train.sh
After the training is completed, the checkpoints corresponding to the three random seeds (1111,1112,1113) can be used for evaluation. For example, evaluate the the model's binary classification accuracy in MOSI:
CUDA_VISIBLE_DEVICES=0 python robust_evaluation.py --config_file configs/eval_mosi.yaml --key_eval Has0_acc_2
- This work builds upon our previous works ALMT, which was published in EMNLP 2023.
- Due to the regression metrics (such as MAE and Corr) and classification metrics (such as acc2 and F1) focus on different aspects of model performance. A model that achieves the lowest error in sentiment intensity prediction does not necessarily perform best in classification tasks. To comprehensively demonstrate the capabilities of the models, all the results of all models in the comparisons are selected as the best-performing checkpoint for each type of metric. This means that the classification metrics (such as acc2 and F1) and regression metrics (such as MAE and Corr) correspond to different epochs of the same training process. If you wish to compare the performance of models across different metrics at the same epoch, we recommend you rerun this code.
- In Table 9, the Acc-5 of the CENET at the r=0.7 is incorrectly reported as
59.86%. The correct value should be 23.57%. This error impacts the overall robustness evaluation in Table 2, where the Acc-5 of CENET is revised from37.25%to 33.62%. The mistake occurred during manual filling in the values for multiple tables. This correction does not alter the performance of proposed LNLN, nor does it affect the original analysis and conclusions of the paper. We sincerely apologize for the oversight and thank the readers for identifying this issue.
Please cite our paper if you find our work useful for your research:
@inproceedings{zhang-etal-2024-lnln,
title = "Towards Robust Multimodal Sentiment Analysis with Incomplete Data",
author = "Zhang, Haoyu and
Wang, Wenbin and
Yu, Tianshu",
booktitle = "The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)",
year = "2024"
}