ImJoy: an open-source computational platform for the deep learning era

To the Editor — Deep learning (DL) methods achieve breakthrough performances in analyzing biomedical data across countless tasks, including medical diagnostics, DNA sequence analysis, augmented microscopy and drug design. Combined with increasing data repositories in genomics, imaging and other fields, such successes underlay a growing demand to adapt DL methods to new datasets and questions¹. However, the dissemination of DL approaches faces considerable hurdles. Most published DL studies^2,3,4,5 require users to retrain models on their own data to obtain the best performance and/or avoid erroneous results. Although trained models are frequently available through web applications or ImageJ plugins, retraining is typically only possible via scripts or command lines, rather than graphical user interfaces (GUIs). In addition, the complexities of setting up the required hardware and software environments often constitute forbidding obstacles⁶. Furthermore, the large datasets and computational resources typical of current DL successes pose challenges to traditional desktop-oriented software that tightly couple GUI and computation. Cloud services can partly alleviate these difficulties, but raise privacy and confidentiality issues that can be prohibitive for medical data⁷. Meanwhile, deploying scientific software to mobile platforms can make them accessible to billions of people⁸, enabling large-scale biomedical research and citizen science. These opportunities and challenges call for new computational frameworks.

By leveraging various web libraries, such as three.js and D3, ImJoy allows users to build rich and interactive applications (Fig. 1a). For example, the ImageAnnotator plugin allows annotation of images, a pre-requisite for training segmentation methods, and can run on touchscreen devices (for example, smartphones). The HPA-UMAP plugin visualizes protein localization features computed by DL from Human Protein Atlas images⁹. Such interactive visualizations are instrumental to exploring massive databases and interacting with cloud services.