A Survey of Multilingual Neural Machine Translation A Survey of Multilingual Neural Machine Translation

There are a wide range of architectural choices in the design of MNMT models. The choices are primarily defined by the degree of parameter sharing among various supported languages.

Minimal Parameter Sharing. Firat et al. [48] proposed a model comprised of separate embeddings, encoders and decoders for each language that all shared a single attention mechanism. Additionally there are two shared components: a layer for all encoders for initializing the initial decoder state by using the final encoder state and an affine layer for all decoders to project the final decoder state before computing softmax. However, the focus is on the shared attention layer as it has to bear most of the burden of connecting all source and target languages. Figure 4(a) depicts a simplified view of this model. Different from the attention score calculated for a single language pair in Equation (2), the attention score in Firat et al. [48] is calculated from multiple encoders and decoders as:

Complete Parameter Sharing. Johnson et al. [70] proposed a highly compact model where all languages share the same embeddings, encoder, decoder, and attention mechanism. Typically, a common vocabulary across all languages is first generated using a subword-level encoding mechanism such as BPE, WPM, or SPM. Thereafter, all corpora are concatenated⁴ and the input sequences are prefixed with a special token (called the language tag) to indicate the target language (see Figure 4(b)). This enables the decoder to correctly generate the target language despite all target languages sharing the same decoder parameters. Note that the embedding and softmax layers are shared across all languages and Ha et al. [58] proposed a similar model, but they maintained separate vocabularies for each language. While this might help in faster inference due to smaller softmax layers, the possibility of cognate sharing is lower, especially for linguistically close languages sharing a common script. According to us, Johnson et al. [70] should be particularly useful for related languages, especially those that use the same script, because they have high degree of lexical and syntactic similarity [123]. However, Ha et al. [58] can be used for unrelated languages. In practice, we recommend experimenting with both Ha et al. [58] and Reference [70] and then choosing the appropriate model. We believe that an empirical analysis of these architectures in a variety of language and corpora settings will be helpful. A few techniques through which lexical similarity can be further leveraged are:

• representing all languages in a common script using script conversion [38, 93] or transliteration (Nakov and Ng [103] for multilingual SMT);
  
• using a common subword-vocabulary across all languages, e.g., character [93] and BPE [106];
  
• representing words by both character encoding and a latent embedding space shared by all languages [155].
  

Concerning the choice of non-embedding or softmax layers, Rikters et al. [121] and Lakew et al. [89] have compared RNN, CNN, and the self-attention-based architectures for MNMT. They show that self-attention-based architectures outperform the other architectures in many cases. The most common hypothesis is that self-attention enables random access to all words in a sentence, which leads to better word and sentence representations.

The complete sharing approach treats the NMT system as a black box to train a multilingual system. We would like to emphasize that this type of model has maximum simplicity and has minimal parameter size, usually around 30M (similar to bilingual models), as all languages share the same parameters; and achieves comparable/better results w.r.t. bilingual systems. Work on massively multilingual NMT [2, 5, 11] pushes the idea of completely shared models for all language pairs to its limits. Aharoni et al. [2], Arivazhagan et al. [5] trained a single model for 59 and 103 languages and explore a wide range of model configurations focusing on data selection,⁵ corpora balancing, vocabulary, deep stacking, training, and decoding approaches. While massively multilingual models have a strong positive impact on low-resource languages, they tend to benefit translation into English a lot more than from English. Furthermore, the gains in translation quality tend to taper off when using more than 50 languages. These works focus on translation performance between language pairs for which there was no training data but we will discuss this separately in Section 5. However, a massively multilingual system also runs into representation bottlenecks [2, 135], where not all translation directions show improved performance despite a massive amount of data being fed to a model with a massive number of parameters. Future research should focus on better addressing the bottleneck issues, issues with deeply stacked models, and designing new modular layers that can handle a wide variety and a large number of languages. We strongly recommend readers to read the paper by Arivazhagan et al. [5], which itself is a survey on multilingual NMT on web-scale data. From the perspective of understanding the working of such models, a toolkit for visualization and inspection of multilingual models should be extremely valuable and Reference [83] is a good start in this direction.

Controlled Parameter Sharing. In between the extremities of parameter sharing exemplified by the above-mentioned models lies an array of choices. The degree of parameter sharing can be controlled at various layers of the MNMT system. A major factor driving the degree of parameter sharing is the divergence between the languages involved [123] and the desired simplicity as well as the flexibility of modeling.

Sharing encoders among multiple languages is very effective and is widely used [93, 123]. Keeping decoders separate is important, because the burden of generation is mainly on the decoder. Therefore, the job of the encoder is relatively simpler, which means that sharing an encoder between many languages leads to better parameter utilization. However, the decoder and its attention mechanism should be as robust as possible. Blackwood et al. [13] explored target language, source language, and pair-specific attention parameters. They showed that target language-specific attention performs better than other attention-sharing configurations, thus highlighting that designing a strong decoder is extremely important. For self-attention-based NMT models, Sachan and Neubig [123] explored various parameter-sharing strategies. They showed that sharing the decoder self-attention and encoder-decoder cross-attention parameters is useful for linguistically dissimilar languages. By sharing self- and cross-attention mechanisms in the decoder, the decoder most likely learns target language representations that are better aligned with source language representations. Wang et al. [157] further proposed a mechanism to generate a universal representation instead of separate encoders and decoders to maximize parameter sharing. They also used language-sensitive embedding, attention, and discriminator for different languages. This helps control the amount of sharing in an indirect fashion. Bapna and Firat [11] also extend a fully shared model with language-pair-specific adaptor layers that are fine-tuned for those pairs. After training a fully shared model, additional adaptor layers are inserted into the model and only those layers are fine-tuned, which requires significantly less computation cost. This does lead to an increase in the number of parameters but it is modest compared to a minimally shared system in which we would have to introduce many more parameters due to the addition of more encoders and decoders.

Fixing sharing configurations prior to training is ill-advised, because sharing one set of parameters might be optimal for one language pair but not another. To this end, Zaremoodi et al. [164] proposed a routing network to dynamically control parameter sharing where the parts to be shared depend on the parallel corpora used for training. However, Platanios et al. [115] learned the degree of parameter sharing from the training data. This is achieved by defining the language-specific model parameters as a function of global parameters and language embeddings. A base set of parameters is transformed into another set of parameters for a specific language using linear projections. If $\theta$ is the set of base parameters, then the parameters for a specific language pair $src_{i}$ and $tgt_{j}$ can be obtained via a linear transformation $F(\theta)$. This linear transformation involves learning some additional parameters for projection, but this involves far fewer parameters compared to modeling complete sets of parameters for each new language and thus this approach is quite attractive. It will be interesting to determine whether using non-linear projections is better than linear ones. This approach reduces the number of language-specific parameters (only language embeddings), while still allowing each language to have its own unique parameters for different network layers. In fact, the number of parameters is only a small multiple of the compact model (the multiplication factor accounts for the language embedding size) [70], but the language embeddings can directly impact the model parameters instead of the weak influence that language tags have.

Designing the right sharing strategy is important to maintaining a balance between model compactness, translation accuracy, simplicity, and flexibility. There are not many works that address the representation bottlenecks in these types of models. However, we should note that work by Huang et al. [65] focuses on efficient training mechanisms for extremely large multilingual models with over 128 layers so they can learn high-quality representations. Although, they do not actually analyze the learned representations it could be beneficial to analyze such models to understand how representation bottlenecks can be overcome. It will be interesting to see more approaches that use the training data itself to enable a model to increase or decrease its own complexity or capacity. Reinforcement learning and genetic algorithms applied to neural architecture search (NAS) [169] can be one of the ways to achieve this. We also refer the readers to work on conditional computation [12] in which parts of the model to be computed are determined by the data.

A central task in MNMT is alignment of representations of words and sentences across languages so divergence between languages can be bridged, enabling the model to handle many languages. This involves the study and understanding of the representations learned by multilingual models and using this understanding to further improve modeling choices. The remainder of this subsection discusses these issues related to multilingual representations.

Vocabulary. An important but not substantially addressed aspect of MNMT (NMT in general) is vocabulary. In the case of MNMT models that share vocabularies, the most common way to create a vocabulary is to sample an equal number of words for each language to ensure balanced representation. These vocabularies can then be used to learn sub-word vocabularies that can eliminate the out-of-vocabulary issue for NMT. However, this does not account for the nature of the languages involved and the skew in their sizes. Recent works attempt to address this using the concept of temperature based vocabulary sampling [2] and this is known to have a positive impact on MNMT models. This work was conducted in a massively multilingual setting, and we believe that further studies and innovation are required.

The Nature of Multilingual Representations. Since MNMT systems share network components across languages, they induce a relationship among representations of sentences across languages. Understanding the nature of these multilingual representations can help get insights into the working of multilingual models. Some works that visualize multilingual model embeddings suggest that the encoder learns similar representations for similar sentences across languages [36, 70]. Since these visualizations are done in very low dimensions (2–3 dimensions), they might not convey the right picture about the language-invariance of multilingual representations. Kudugunta et al. [83] do a systematic study of representations generated from a massively multilingual system using SVCCA [118], a framework for comparing representations across different languages, models, and layers. Their study brings out the following observations on the nature of multilingual embeddings from a compact MNMT system:

• While encoder representations for similar sentences are similar across languages, there is a fine-grained clustering based on language similarity. This explains why transfer learning works better with related languages (as discussed in Section 4).
  
• The boundary between the encoder and decoder is blurry and the source language representations depend on the target language and viceversa.
  
• Representation similarity varies across layers. The invariance increases in higher layers on the encoder side. However, the invariance decreases in higher layers on the decoder side. This is expected, since the decoder is sensitive to the target language to be generated. The decoder has to achieve the right balance between language-agnostic and language-aware representations.
  

Language-invariant representations seem like a nice abstraction, drawing comparisons to interlingual representations [70]. It has been a major goal of most MNMT research, as discussed in the subsequent sections. They have been shown to be beneficial for building compact models and transfer learning. Given these empirical observations from multiple sources, we think that language-invariant representations in appropriate parts of the multilingual model is a desirable property.

Encoder Representation. There are two issues that might make encoder representations language-dependent. Parallel sentences, from different source languages, can have a different number of tokens. Hence, the decoder's attention mechanism sees a variable number of encoder representations for equivalent sentences across languages. To overcome this, an attention bridge network generates a fixed number of contextual representations that are input to the attention network [94, 149]. By minimizing the diversity of representations, the decoder's task is simplified and it becomes better at language generation. The choice of a single encoder for all languages is also promoted by Hokamp et al. [64], who opt for language-specific decoders. Murthy et al. [101] pointed out that the sentence representations generated by the encoder are dependent on the word order of the language and are, hence, language-specific. They focused on reordering input sentences to reduce the divergence caused due to different word orders to improve the quality of transfer learning.

Decoder Representation. The divergence in decoder representation needs to be addressed when multiple target languages are involved. This is a challenging scenario, because a decoder should generate representations that help it generate meaningful and fluent sentences in each target language. Therefore, a balance between learning language-invariant representations and being able to generate language-specific translations is crucial. If several parallel corpora for different language pairs are simply concatenated and then fed to a standard NMT model, then the NMT model might end up generating a mixed language translation as a result of vocabulary leakage due to language-invariant representations. The language tag trick has been very effective in preventing vocabulary leakage [70], because it enables the decoder to clearly distinguish between different languages.⁶ Further, Blackwood et al. [13] added the language tag to the beginning as well as end of sequence to enhance its effect on the sentence representations learned by a left-to-right encoder. This shows that dedicating a few parameters to learn language tokens can help a decoder maintain a balance between language-agnostic and language-distinct features. Hokamp et al. [64] showed that more often than not, using separate decoders and attention mechanisms gives better results as compared to a shared decoder and attention mechanism. This work implies that the best way to handle language divergence would be to use a shared encoder for source languages and different decoders for target languages. We expect that the balance between language-agnostic and language-specific representations should depend on the language pairs. Prasanna [117], Tan et al. [141] are some of the works that cluster languages into language families and train separate MNMT models per family. Language families can be decided by using linguistic knowledge⁷ [117] or by using embedding similarities where the embeddings are obtained from a multilingual word2vec model [141]. Comparing language-family-specific models and mixed-language-family models shows that the former models outperform the latter models. In the future, when training a model on a large number of languages, researchers could consider different decoders for different language families and each decoder in turn can use the language token trick to generate specific languages in the language family assigned to the decoder.

Impact of Language Tag. There are some works that explore the effects of the language tag trick on the shared decoder, its language-agnostic (or lack thereof) representations, and the final translation quality. Wang et al. [156] explored multiple methods for supporting multiple target languages: (a) target language tag at beginning of the decoder, (b) target language dependent positional embeddings, and (c) divide hidden units of each decoder layer into shared and language-dependent ones. Each of these methods provides gains over Johnson et al. [70], and combining all methods gave the best results. Hokamp et al. [64] showed that in a shared decoder setting, using a task-specific (language pair to be translated) embedding works better than using language tokens. We expect that this is because learning task-specific embeddings needs more parameters and helps the decoder learn better features to distinguish between tasks. In the related sub-area of pre-training using MASS [138], mBERT/XLM [32] it was shown that using language or task tags is sufficient for distinguishing between the languages used for pre-training large models. Given that massive pre-training is relatively new, further exploration is much needed.

In the above discussion, we have not considered the possibility of task interference and negative language transfer. Previous research has shown that certain languages help improve translation quality when grouped together [36, 170]. This implies the possibility of language pairs that may interfere with each other (task interference in the multi-task setting) and cause unhelpful knowledge to be transferred (negative transfer). Deeper research on this is pending.

MNMT model training is a critical issue that requires sophisticated methods. The core of all methods is the minimization of the negative log-likelihood of the translation for all language pairs. Conceptually, the negative log-likelihood of the MNMT model is the average of the negative log-likelihoods of multiple bilingual models. As we have mentioned before, minimizing the negative log-likelihood can be done using one or more gradient-descent-based optimization algorithms such as SGD, ADAM, ADAGRAD, and so on [122]. There are two main types of training approaches: single stage or parallel or joint training and sequential or multi-stage training. Depending on the use-case, multi-stage training can be used for model compression (knowledge distillation) or fine-tuning, addition of data and/or languages (incremental training).

Single Stage Parallel/Joint Training. We simply pre-process and concatenate the parallel corpora for all language pairs and then feed them to the model batch-wise. For models with separate encoders and decoders, each batch consists of sentence pairs for a specific language pair [48]; whereas for fully shared models, a single batch can contain sentence pairs from multiple language pairs [70, 93]. As some language pairs might have more data than other languages, the model may be biased to translate these pairs better. To avoid this, sentence pairs from different language pairs are sampled to maintain a healthy balance. The most common way of maintaining balance is to oversample smaller datasets to match the sizes of the largest datasets, but Arivazhagan et al. [5] worked on temperature-based sampling and showed its cost-benefit analysis.

Knowledge Distillation. Knowledge distillation was originally proposed by Hinton et al. [63] for tasks that do not involve generating sequences such as image classification. The underlying idea is to train a large model with many layers and then distill its knowledge into a small model with fewer layers by training the small model on the softmax generated by the large model instead of the actual labels. This approach does not work well for sequence generation and, hence, Kim and Rush [73] proposed sequence distillation where they simply translate the training data using a pre-trained model. They then use this pseudo-data to train smaller models. These smaller models are able to learn faster and better because of the reduced burden for decision making. Following this, Tan et al. [142] trained bilingual models for all language pairs involved and then these bilingual models are used as teacher models to train a single student model for all language pairs. The student model is trained using a linear interpolation of the standard likelihood loss as well as distillation loss that captures the distance between the output distributions of the student and teacher models. The distillation loss is applied for a language pair only if the teacher model shows better translation accuracy than the student model on the validation set. This approach shows better results than joint training of a black-box model, but training time increases significantly, because bilingual models also have to be trained.

Incremental Training. These approaches aim to decrease the cost of incorporating new languages or data in multilingual models by avoiding expensive retraining. This is because not all language pairs may be available when the initial MNMT model is trained. It will be expensive to re-train the multilingual model from scratch when a language pair has to be added. A practical concern with training MNMT in an incremental fashion is dealing with vocabulary. Some works alter a pre-trained model's capacity to accommodate new languages. Lakew et al. [90] updated the vocabulary of the parent model with the low-resource language pair's vocabulary before transferring parameters. Embeddings of words that are common between the low- and high-resource languages are left untouched and randomly initialized embeddings may be used for as yet unseen words. A simpler solution would be to consider a universal romanized script so incorporating a new language can be a matter of resuming training or fine-tuning. Escolano et al. [45] focused on first training bilingual models and then gradually increasing their capacity to include more languages. To address representation bottlenecks, Bapna and Firat [11] proposed expanding the capacities of pre-trained MNMT models (especially those trained on massive amounts of multilingual data) using tiny feed-forward components that they call adaptors. For each language pair in a pre-trained (multilingual) model, they add adaptors at each layer and fine-tune them on parallel corpora for that direction. These modular adaptor layers can be considered as experts that specialize for specific language pairs and can be added incrementally. They showed that this approach can help boost the performance of massively multilingual models trained by Arivazhagan et al. [5] and Aharoni et al. [2]. Note that this model can be used to incorporate new data, but new language pairs cannot be added. For multiway models, it is critical for incremental training to deal with catastrophic forgetting where previously learned knowledge is forgotten by including an additional language pair or training on additional data. While previously mentioned work on adaptor layers [11] is an effective solution, further research will be beneficial.

A major criticism of all these approaches is that the MNMT models are trained in the same way as a regular NMT model. Most researchers tend to treat all language pairs equally, with the exception of oversampling smaller corpora to match the sizes of the larger corpora, and tend to ignore the fact that NMT might be able to handle some language pairs better than others. There are works that propose to focus on scaling learning rates or gradients differently for high-resource and low-resource language pairs [67]. However, Kiperwasser and Ballesteros [74] proposed a multi-task learning model for learning syntax and translation, where they showed different effects of their model for high-resource and low-resource language pairs. Furthermore, different language pairs might be learned at different rates and the optimal stopping criteria is also heavily under-researched. Currently, model selection is based on an automatic evaluation metric score on the dev set composed of multiple language pairs. As the metric is an average score for multiple language pairs, the chosen model might not be optimal for individual pairs. Therefore, we suppose that the MNMT models in the vast majority of papers are sub-optimally trained even if they improve the translation quality for low-resource languages. In hindsight, this particular aspect of MNMT training deserves more attention.

Although multiway MNMT modeling has been thoroughly explored, there are still a number of open questions, especially the representation bottleneck and parameter-sharing protocols. The next section will pay special attention to MNMT in resource-constrained scenarios.

Most studies have explored transfer learning on the source-side: The high-resource and low-resource language pairs share the same target language. The simplest approach is jointly training both language pairs [70]. However, the final model may not be optimally tuned for the child language pair, whose performance we are interested in.

A better approach is to fine-tune the parent model with data from the child language pair. Zoph et al. [170] showed that such transfer learning can benefit low-resource language pairs. First, they trained a parent model on a high-resource language pair. The child model is initialized with the parent's parameters wherever possible and trained on the small parallel corpus for the low-resource pair. They also studied the effect of fine-tuning only a subset of the child model's parameters (source and target embeddings, RNN layers, and attention). They observed that fine-tuning all parameters except the input and output embeddings was optimal in one setting. However, further experiments are required for drawing strong conclusions.

Training the parent model to optimality may not be the best objective for child tasks. It may be preferable that parent model parameters are amenable to fast adaptation/fine-tuning on child tasks. Learning such a parent model is referred to as meta-learning and can generalize well to child tasks. Gu et al. [55] used the model-agnostic meta-learning (MAML) framework [47] to learn appropriate parameter initialization from the parent pair(s) by taking the child pair into consideration. The meta-learning-based approach significantly outperforms simple fine-tuning. They also show that having more parent language pairs (jointly trained) also improves performance on the child language pairs.

Transfer learning on the target-side has been more challenging than transfer learning on the source-side. Distinct target languages require target-language-specific representations, while transfer learning prefers target-language-invariant representations. The success of transfer learning relies on achieving the right balance between these factors. Johnson et al. [70] showed that joint training does not provide any significant benefit. Fine-tuning is beneficial in very low-resource scenarios [37], but gains may be limited due to catastrophic forgetting. Dabre et al. [37] showed that a multi-stage fine-tuning process is beneficial when multiple target languages are involved. They do not focus on language divergence during their multilingual multi-stage tuning but show that the size of helping data matters. From their work, it will be worthwhile to consider involving multiple medium-sized (few hundreds of thousands of lines) helping corpora involving a variety of languages. This multilingual multi-stage transfer learning scenario requires further research.

Zoph et al. [170] randomly initialized the word embeddings of the child source language, because those could not be transferred from the parent. However, this approach does not map the embeddings of similar words across the source languages a priori. Gu et al. [54] improved on this simple initialization by mapping pre-trained monolingual word embeddings of the parent and child sources to a common vector space. This mapping is learned via the orthogonal Procrustes method [125] using bilingual dictionaries between the sources and the target language [61]. Kim et al. [71] proposed a variant of this approach where the parent model is first trained and monolingual word-embeddings of the child source are mapped to the parent source's embeddings prior to fine-tuning. While Gu et al. [54] require the child and parent sources to be mapped while training the parent model, the mapping in Kim et al. [71]’s model can be trained after the parent model has been trained.

It is not always possible to have parent and child languages from the same language family and, hence, blindly fine-tuning a parent model on the child language data might not take into account the syntactic divergence between them. Although it is important to address this issue there are surprisingly few works that address it. Murthy et al. [101] showed that reducing the word order divergence between source languages by reordering the parent sentences to match child word order is beneficial in extremely low-resource scenarios. Since reordering is part of the pre-processing pipeline, it is referred to as pre-ordering. Pre-ordering has been widely used in phrase-based MT [31] and explored in NMT [165] to overcome source-target syntactic divergence. Pre-ordering can be done via hand-written rules [87, 120] or learnt rules/models [151, 162]. We expect that future work will focus on syntactic divergence between languages via NMT models that handle reordering implicitly rather than rely on pre-ordering. Kim et al. [71] took a different approach to mitigate syntactic divergence. They trained the parent encoder with noisy source data introduced via probabilistic insertion and deletion of words as well as permutation of word pairs. This ensures that the encoder is not over-optimized for the parent source language syntax. Gu et al. [54] proposed to achieve better transfer of syntax-sensitive contextual representations from parents using a mixture of language experts network.

Results from the above-mentioned approaches show that, in many cases, transfer learning benefits the child language pair even if the parent and child languages are not related. These benefits are also seen even when the languages have different writing systems. These are interesting findings and can be attributed to the ability of the NMT models to learn cross-lingual representations, as discussed in previous sections. It is natural to ask if language relatedness can impact the effectiveness of lexical and syntactic transfer. Learning cross-lingual embeddings, used for lexical transfer, is difficult for distant languages [114, 137].

Zoph et al. [170] and Dabre et al. [40] empirically showed that a related parent language benefits the child language more than an unrelated parent. Maimaiti et al. [95] further showed that using multiple highly related high-resource language pairs and applying fine-tuning in multiple rounds can improve translation performance more, compared to only using one high-resource language pair for transfer learning. Kocmi and Bojar [76] presented a contradictory result in the case of Estonian (related Finnish vs. unrelated Czech/Russian parent) and suggest that size of the parent is more important. Another factor that may have aided transfer using Russian/Czech is their contact relationship with Estonian, since Russian and Czech are spoken in geographically nearby regions to Estonia. Most of the literature suggests that language relatedness is beneficial. It is probably easier to overcome language divergence when the languages are related, but a deeper study of how to uncover different factors is warranted.

Further, language relatedness can be explicitly utilized to improve transfer learning. Language relatedness is typically exploited by using shared subword BPE vocabulary and BPE embeddings between the parent and child languages [106]. Maimaiti et al. [95] used a unified transliteration scheme at the character level. This utilizes the lexical similarity between the languages and shows significant improvements in translation quality. Neubig and Hu [105] used “similar language regularization” to prevent overfitting when rapidly adapting a pre-trained, massively multilingual NMT model (universal model) for low-resource languages. While fine-tuning the universal model for a low-resource pair, overfitting is avoided by using a subset of the training data for a related high-resource pair along with the low-resource pair. Chaudhary et al. [19] used this approach to translate 1,095 languages to English. Further, not all parallel data from the parent task may be useful in improving the child task. Wang and Neubig [154] proposed selection of sentence pairs from the parent task based on the similarity of the parent's source sentences to the child's source sentences. The significant gains from simple methods described point to the value of utilizing language relatedness. Further methods should be explored to create language-invariant representations specifically designed for related languages.

While low-resource translation is hard in itself, an even more extreme scenario is where no direct data exists between language pairs of interest. The next section discusses literature related to this scenario.

The simplest approach to pivot translation is building independent source-pivot (S-P) and pivot-target (P-T) MT systems. At test time, the source sentence cascades through the S-P and P-T systems to generate the target sentence. Note that this pivot method is agnostic to the underlying translation technology and can be applied to SMT [147], RBMT [160], or NMT [91] systems. Also, the pivoting can be done using an MNMT system, too [56, 91] (as opposed to pivoting via bilingual systems). This simple process has two limitations due to its pipeline characteristic: (a) translation errors compound in a pipeline; and (b) decoding time is doubled, since inference has to be run twice. Sometimes, more than one pivot may be required to translate between the source and pivot language. Increased pipeline length exacerbates the above-mentioned problems. The quality of the source-pivot translation is a bottleneck to the system. A variant of this approach extracts n-best translations from the S-P system. For each pivot translation, the P-T system can generate m-best translations. The $n\times m$ translation candidates can be re-ranked using scores from both systems and external features to generate the final translation. This approach improves the robustness of the pipeline to translation errors.

Multilingual NMT models offer an appealing possibility. Even if the MNMT system has not been trained for the unseen language pair, Johnson et al. [70] showed that the system is able to generate reasonable target-language translations for the source sentence. Along with the source sentence, the desired output language's language tag is provided as input. This is sufficient for the system to generate output in the target language. Note that the MNMT system was exposed to zero bilingual resources between the source and target languages during training and encountered the unseen language pair only at test-time. Hence, this translation scenario is referred to as zero-shot translation.

The appeal of zero-shot translation is two-fold:

• Translation between any arbitrary language pair can be done in a single decoding step, without any pivoting, irrespective of the number of implicit pivots required to bridge the source and target language in the multilingual training corpus. The MNMT system can be conceived as an implicit pivoting system.
  
• Given a multilingual parallel corpus spanning N languages, only a single multilingual NMT model is required to translate between $N\times (N-1)$ languages.
  

Challenges of Zero-shot Translation. It has been argued that training an MNMT system incorporating multiple languages could benefit zero-shot translation due to better interlingual representations and elimination of cascading errors [70]. The simple zero-shot system described above, though promising, belies these expectations and its performance is generally lower than the pivot translation system [4, 56, 70, 113].

Some researchers have analyzed zero-shot translation to understand its underperformance. The following reasons have been suggested as limitations of zero-shot translation:

Spurious correlations between input and output language. During training, the network is not exposed to the unseen pairs. In its quest to capture all correlations in the training data, the model learns associations between the input representations and the target language for the observed language pairs. At test time, irrespective of the target language tag, the network will tend to output a language it has already observed with the source language in the training data [56]. Arivazhagan et al. [4] showed that translation quality is closer to pivot systems if evaluation is restricted to sentences where the correct target language is generated.

Language variant encoder representations. The encoder representations generated by the model for equivalent source and pivot languages are not similar. Due to this discrepancy, the output generated by the decoder will be different for the source and pivot representations [4, 83].

To address these limitations, the following methods have been proposed for improving zero-shot translation:

Minimize divergence between encoder representations. During training, additional objectives ensure that the source and pivot encoder representations are similar. Arivazhagan et al. [4] suggested an unsupervised approach to align the source and pivot vector spaces by minimizing a domain adversarial loss [50]—a discriminator is trained to distinguish between different encoder languages using representations from an adversarial encoder. Since S-P parallel corpora is also available, supervised loss terms that penalize divergence in source and pivot representations for each sentence pair have also be explored. Different loss functions such as cosine distance [4], Euclidean distance [113], and correlation distance [124] have been shown to be beneficial in reducing the source/pivot divergence. Ji et al. [69] proposed to use pre-trained cross-lingual encoders trained using multilingual MLM, XLM, and BRLM objectives to obtain language-invariant encoder representations. Sen et al. [129] used denoising autoencoding and back-translation to obtain language-invariant encoder representations.

Encourage output agreement. Al-Shedivat and Parikh [3] incorporated additional terms in the training objective to encourage source and pivot representations of parallel sentences to generate similar output sentences (synthetic) in an auxiliary language (possibly an unseen pair). This also avoids spurious correlations, since the network learns to generate unseen pairs. Similar considerations motivated Pham et al. [113] to add a pivot auto-encoding task in addition to the source-pivot translation task. They incorporate additional loss terms that encourage the attention-context vectors as well as decoder output representations to agree while generating the same pivot output. Xu et al. [163] considered different translation paths among multiple languages in unsupervised NMT by designing training objectives for these paths to achieve the same goal.

Effect of corpus size and number of languages. Aharoni et al. [2] suggested that the zero-shot performance of multilingual NMT system increases with the number of languages incorporated in the model. It is not clear if the approaches mentioned above to address zero-shot NMT limitations can scale to a large number of languages. Arivazhagan et al. [4] showed that cosine distance-based alignment can be scaled to a small set of languages. Some studies suggest that zero-shot translation works reasonably well only when the multilingual parallel corpora is large [91, 97].

Addressing wrong language generation. To address the problem of generation of words in the wrong language, Ha et al. [59] proposed to filter the output of the softmax, forcing the model to translate into the desired language. This method is rather effective despite its simplicity.

The zero-shot approaches discussed above can complement multiway NMT systems described in Section 3 to support translation between the language pairs the model has not observed during training. Further work is needed to establish if these methods can scale to massively multilingual models.

The zero-shot approaches discussed in the previous section seek to avoid any training specific to unseen language pairs. Their goal is to enable training massively multilingual NMT systems that can perform reasonably well in zero-shot directions, without incurring any substantial overhead during training for all unseen language heads. When there is a case for optimizing the translation quality of an unseen pair, the training process can also consider objectives specific to the language pair of interest or tune the system specifically for the language pair of interest. Such approaches are referred to as zero-resource approaches [49]. Their training objectives and regimen customized for the unseen language pair of interest distinguish them from zero-shot approaches. Note that zero-resource approaches do not use any true source-target parallel corpus. The following approaches have been explored for zero-resource translation.

Synthetic Corpus Generation. The pivot side of the P-T parallel corpus is back-translated to the source language. The back-translation can be achieved either through zero-shot translation or pivot translation, creating a synthetic S-T parallel corpus. An S-T translation model can be trained on this synthetic parallel corpus. Adding the synthetic corpus to the multilingual corpus helps alleviate the spurious correlation problem. Some works have shown that this approach can outperform the pivot translation approach [49, 56, 91]. The source is synthetic, hence, there is a difference between the training and test scenarios. Further, synthetic parallel corpus can be generated from monolingual pivot data, too [35].

Iterative Approaches. The S-T and T-S systems can be trained iteratively such that the two directions reinforce each other [91]. Sestorain et al. [132] jointly trained both the models incorporating language modelling and reconstruction objectives via reinforcement learning. The LM objective ensures grammatical correctness of translated sentences, while the reconstruction objective ensures translation quality. The major shortcoming of iterative approaches is that they usually do not yield improvements after the first 2–3 iterations and are extremely time-consuming.

Teacher-student Training. Chen et al. [21] assumed that the source and pivot sentences of the S-P parallel corpus will generate similar probability distributions for translating into a third language (target). They build the S-T model (“student”) without any parallel S-T corpus by training the model to follow a P-T model (“teacher”). They propose two approaches: sentence-level mode approximation and word-level KL divergence. Both approaches outperform the pivot baseline, with the latter showing better performance. A shortcoming of the first method is reliance on S-T parallel corpus with synthetic target, whereas the second approach learns to follow a soft distribution.

Combining Pre-trained Encoders and Decoders. Kim et al. [72] combined S-P encoder with P-T decoder to create the S-T model. They improved the simple initialization using some fine-tuning objectives and/or source-pivot adaptation to ensure that source and pivot representations are aligned. Whether this can be done via a fully shared MNMT model or not is unclear.

Most the methods discussed in this section are either just competitive with or slightly better than simple pivot translation. This opens opportunities in understanding the challenges in unseen language translation and exploring solutions.

Although rule-based systems (RBMT) do not attract research interest, we have included this short discussion for completeness with a goal to understanding the connections of MNMT with previous approaches. RBMT systems using an interlingua were explored widely in the past. The interlingua is a symbolic semantic, language-independent representation for natural language text [133]. Two commonly used interlinguas are UNL [146] and AMR [9]. Different interlinguas have been proposed in various systems, such as KANT [109], UNL, UNITRAN [43], and DLT [158]. Language-specific analyzers converted language input to interlingua representation, while language-specific decoders converted the interlingua representation into another language. To achieve an unambiguous semantic representation, a lot of linguistic analysis had to be performed and many linguistic resources were required. Hence, in practice, most interlingua systems were limited to research systems or translation in specific domains and could not scale to many languages.

Phrase-based SMT (PBSMT) systems [81], a very successful MT paradigm, were also bilingual for the most part. Compared to RBMT, PBSMT requires less linguistic resources and instead requires parallel corpora. However, like RBMT, they work with symbolic, discrete representations, making multilingual representation difficult. Moreover, the central unit in PBSMT is the phrase, an ordered sequence of words (not in the linguistic sense). Given its arbitrary structure, it is not clear how to build a common symbolic representation for phrases across languages. Nevertheless, some shallow forms of multilingualism have been explored in the context of: (a) pivot-based SMT, (b) multi-source PBSMT, and (c) SMT involving related languages.

Pivoting. Most commonly used solutions are: chaining source-pivot and pivot-target systems at decoding [147], training a source-target system using synthetic data generated using target-pivot and pivot-source systems [53], and phrase-table triangulation pivoting source-pivot and pivot-target phrase tables [147, 159].

Multi-source. Typical approaches are: re-ranking outputs from independent source-target systems [110], composing a new output from independent source-target outputs [98], and translating a combined input representation of multiple sources using lattice networks over multiple phrase tables [126].

Related Languages. For multilingual translation with multiple related source languages, the typical approaches involved script unification by mapping to a common script such as Devanagari [10] or transliteration [103]. Lexical similarity was utilized using subword-level translation models [85, 86, 144, 150]. Combining subword-level representation and pivoting for translation among related languages has been explored [62, 88, 144]. Most of the above-mentioned multilingual systems involved either decoding-time operations, chaining black-box systems, or composing new phrase-tables from existing ones.

Comparison with MNMT. While symbolic representations constrain a unified multilingual representation, distributed universal language representation using real-valued vector spaces makes multilingualism easier to implement in NMT. As no language-specific feature engineering is required for NMT, it is possible to easily scale to multiple languages. Neural networks provide flexibility in experimenting with a wide variety of architectures, while advances in optimization techniques and availability of deep learning toolkits make prototyping faster.