[arXiv] [Project Page] [Blogs] [中文博客]

This repository is built to explore the potential and extensiability of transformers for multimodal learning. We utilize the advantages of Transformers to deal with length-variant sequence. Then we proposes the Data-to-Sequence tokenization following a meta-scheme, then we apply it to 12 modalities including text, image, point cloud, audio, video, infrared, hyper-spectral, X-Ray, tabular, graph, time-series, and Inertial Measurement Unit (IMU) data.

After obtaining the token sequence, we employ a modality-shared encoder to extract representation across different modalities. With task-specific heads, Meta-Transformer can hanle various tasks on the different modalities, such as: classification, detection, and segmentation.

• 2023.7.21: Paper is released at arxiv, and code will be gradually released.
• 2023.7.8: Github Repository Initialization.

• Meta-Transformer with Large Language Models.
• Multimodal Joint Training with Meta-Transformer.
• Support More Modalities and More Tasks.

Welcome to contribute to our project!

To contact us, never hestitate to send an email to yiyuanzhang.ai@gmail.com ,kaixionggong@gmail.com, zhangkaipeng@pjlab.org.cn, or xyyue@ie.cuhk.edu.hk!

If the code and paper help your research, please kindly cite:

@article{zhang2023metatransformer,
        title={Meta-Transformer: A Unified Framework for Multimodal Learning}, 
        author={Zhang, Yiyuan and Gong, Kaixiong and Zhang, Kaipeng and Li, Hongsheng and Qiao, Yu and Ouyang, Wanli and Yue, Xiangyu},
        year={2023},
        journal={arXiv preprint arXiv:2307.10802},
  }

This project is released under the Apache 2.0 license.

This code is developed based on excellent open-sourced projects including MMClassification, MMDetection, MMsegmentation, OpenPoints, Time-Series-Library, Graphomer, SpectralFormer, and ViT-Adapter.