DeepTorrent: a deep learning-based approach for predicting DNA N4-methylcytosine sites

Feature encoding schemes

In this study, we used seven different DNA sequence encoding schemes provided by the iLearn package [32] to encode the DNA sequences: one-hot encoding, composition of k-spaced nucleic acid pairs (CKSNAP), NCP, EIIPs of nucleotides, enhanced nucleic acid composition (ENAC), accumulated nucleotide frequency (ANF) and position-specific trinucleotide propensity based on single-stranded characteristics (PSTNPss). Here, we considered four types of nucleic acids (‘A’, ‘C’, ‘G’ and ‘T’) and the unknown character ‘-’ in DNA sequences. These seven encoding schemes are introduced in the following subsections.

Composition of k-spaced nucleic acid pairs

CKSNAP, also known as nucleotide pair spectrum encoding, have been previously used for 6mA site prediction [18]. CKSNAP transform a DNA sequence into a numerical vector by calculating the occurrence frequency of all possible k-spaced nucleotide pairs along the DNA sequence. A k-spaced nucleotide pair denotes that there are k spaces between these two nucleotides. For example, in the sequence ‘AXXGXXXT’, ‘AG’ is a two-spaced nucleotide pair, and ‘GT’ is a three-spaced nucleotide pair. Let denote the k-spaced nucleotide pair. The frequency of can be defined as:

(1)

where represents the count of along the DNA sequence, l is the length of the DNA sequence and is the space between nucleotide pairs. Thus, denotes the number of all -spaced nucleotide pairs along a DNA sequence with length . Therefore, the DNA sequence was encoded into a 5 5 +1-dimensional vector. In this work, we set = 5.

Accumulated nucleotide frequency

The ANF describes the density information of each nucleotide in a DNA sequence [33]. It is an effective feature encoding scheme for 4mC site prediction [22]. We used the ANF feature descriptors for DNA sequences. Let denote a DNA sequence, and let represent the th prefix sequence of . Here, l is the length of and is the th nucleotide of . The ANF value of is defined as follows:

(2)

(3)

where is the length of the subsequence . For example, the sequence ‘GTTGACT’ is encoded as (1, 0.5, 0.67, 0.5, 0.2, 0.17, 0.43).

Position-specific trinucleotide propensity based on single-stranded characteristics

The PSTNPss depicts the position-specific trinucleotide statistical propensity based on the single-stranded characteristics of DNA [35]. There are 4³ = 64 trinucleotides derived from the alphabet [A, C, G, T], such as ‘AAA’, ‘AAC’, … ‘TTT’, etc. Supposing that , the 64 trinucleotides strings can be sorted by the defined partial order Let L be the sorted trinucleotide list. For a 41 bp long sequence, the trinucleotide positional specificity can be represented by a 6439 matrix F. . The value of the ith row and jth column in F is defined as follows:

(4)

where and denote the frequency of the ith trinucleotide in L at the jth position appearing in the positive and negative samples, respectively. If a trinucleotide contains an unknown nucleotide ‘-’, the value of the trinucleotide will be set to zero. Thus, let S = s₁s₂… s_js_j + 1s_j + 2…s₄₁ denote a 41 bp long DNA sequence, such that can be encoded as a 39-dimensional numerical vector , where is calculated as follows:

(5)

Four different feature encoding combinations

We further grouped the seven encoding schemes described above into four feature encoding combinations. These four combinations and their dimensions are shown in Table 5. For example, ‘ANF + NCP + EIIP + One-hot’ combines ANF, NCP, EIIP and one-hot encoding schemes and encodes a 41 bp long DNA sequence as a (419)-dimensional vector.

Table 5.

Four combinations of encoding schemes

Combination ID	Encoding scheme	Dimension
1	ENAC	415
2	ANF + NCP + EIIP + One-hot	41 9
3	CKSNAP+NCP + EIIP + One-hot	15017
4	PSTNPss + NCP + EIIP + One-hot	3925

Open in a new tab

Convolution neural network

There are three convolutional layers in the DeepTorrent framework. In the first convolutional layer, three convolution blocks with different convolution kernel sizes are used to extract the features from the encoding matrix in parallel. The kernel sizes of these three convolutional blocks are 1, 3 and 5, respectively. There are 32 filters in these three convolutional blocks, the L2 regularization value is 0.002 and ReLU is used as the activation function.

Using the first convolutional layer’s three convolution blocks, we obtain three different kinds of feature representations. For an input mn encoding matrix, each type of feature is represented by an m32 matrix following the first convolutional layer. Subsequently, a merging layer with a dropout value of 0.5 is used to merge the three m32 matrices into an m96 matrix, which had a higher dimension and more meaningful feature representation. Visualization of different kernel sizes is provided as an example in Supplementary Figure S1.

The second convolutional layer, used as an inception module in [36], also contains three convolutional blocks. The first convolutional block with the kernel size 1 was first used to extract higher abstraction feature representations from the concatenated feature maps extracted by the first convolutional layer. Then, the output feature representations are used as the input to the second convolutional block with a kernel size of three and as the input to the third convolutional block with a kernel size of five. The number of filters in these three convolutional blocks is 136, the L2 regularization value is 0.002 and ReLU is used as the activation function. Thus, three m matrices are generated by the second layer of convolutional operations. Subsequently, a merging layer is used to merge three m136 matrices into an m408 matrix. The dropout value is set as 0.5 in the merging layer.

Similarly, the third convolutional layer also applies the inception module, following the same operation as that of the second convolutional layer to obtain higher-dimensional abstract features. The kernel size of the three convolutional blocks in the third convolutional layer is 1, 3 and 5, respectively. The number of filters in these three convolutional blocks is 48, the L2 regularization value is 0.002 and ReLU is used as the activation function. Like the first two convolutional layers, the third convolutional layer is also followed by a merged layer with a dropout value of 0.5.

Attention layer

The attention mechanism adaptively focuses on important positions and relevant parts and ignores irrelevant parts [37, 38]. It has been widely used in a variety of DL applications, including visual processing [38], kinase-specific phosphorylation site prediction [37], speech recognition [39] and natural language processing [40]. In this study, we were motivated by the prospect of implementing the attention layer in DeepTorrent. The attention layer selects important features from two dimensions: the feature dimension and the sequence dimension. This means that the output matrix from the CNN layer and its transposed matrix (‘transposition’ in Figure 1) are fed into the attention layer. In this manner, the two different feature representations can be selected by the attention layer.

Bidirectional long short-term memory

BLSTM is a special type of recurrent neural network (RNN), consisting of two reversed unidirectional LSTM networks [41]. BLSTM can capture the interdependencies across the sequence and integrate both forward and backward information in the sequence [42]. According to a recent study, a good strategy for utilizing the merits of both CNN and RNN is to use the CNN as the pre-processing step for the RNN [43]. In addition, the combination of LSTM with CNN has been used to predict the subcellular localization of proteins from sequence information [44] and to quantify the function of DNA sequences [45]. In DeepTorrent, the CNN layer is connected to two additional BLSTMs, each of which processes the sequence dimension and feature dimension outputs of the CNN. In this way, we obtain four feature representations, among the two feature representations from the attention layer and the two feature representations from BLSTM. Subsequently, four feature representations are combined into a more powerful feature representation by the merging layer.

Fully connected layers and output layer

We used multiple combinations of the four encoding schemes as the input to train the DeepTorrent models in parallel. With three convolutional layers, an attention layer and two BLSTM networks, each feature matrix generated by each encoding combination is converted into four feature vectors. As a result, 16 feature vectors are generated in total. Then, a merging layer is used to merge these 16 feature vectors into a combined feature vector.

Subsequently, the combined feature vector is fed into a fully connected network. The first layer of the fully connected network contains 64 units. Again, the activation function is ReLU, and the dropout value is 0.5. The second fully connected layer has eight units and uses ReLU activation. The final output layer is equipped with softmax loss as the classifier to generate the prediction results.

Parameter optimization

We employed Bayesian optimization [46] to optimize the hyperparameters of the convolutional networks of DeepTorrent. The Bayesian optimization method first models a learning algorithm’s generalization performance as a sample from a Gaussian process and then automatically finds better hyperparameters to optimize the performance of the learning algorithm. Bayesian optimization has been shown to select better hyperparameters than human expert-level optimization for CNNs [46]. In DeepTorrent, the predefined value range for each hyperparameter before optimization and its optimal value after optimization are provided in Supplementary Table S1.

Performance evaluation on the Lin_2017 training dataset

To benchmark the performance of DeepTorrent, we performed a test using all encoding inputs derived from all possible combinations of the four encoding schemes listed in Table 5. Let m, n, p and t (i.e. 1, 2, 3 and 4, respectively) represent the feature combination IDs (as shown in the first column of Table 5). Accordingly, m–n denotes a di-encoding composition containing the encoding scheme combinations m and n; m–n–p denotes a tri-encoding composition that consists of the encoding scheme combinations m, n and p and m–n–p–t denotes a tetra-encoding composition. There were 15 possible encoding compositions: four individual encoding compositions, six di-encoding compositions, four tri-encoding compositions and one tetra-encoding composition. Taking m–n–p–t as an example, DeepTorrent worked as follows: first, the encoding scheme combinations m, n, p and t were input into DeepTorrent in parallel; second, for each encoding scheme combination, the corresponding abstract feature representation was extracted by DeepTorrent; finally, four kinds of abstract feature were concatenated into the feature vector (the merged layer in Figure 1) as the feature descriptors of the predictor.

We integrated six species-specific datasets into a large dataset and used approximately 90% of the entries in the datasets as the training dataset and the remaining entries as the validation dataset. The performance results of the DeepTorrent models trained using each encoding composition are provided in Supplementary Table S2. In addition, we plotted the ROC curves of DeepTorrent trained using each encoding, as shown in Supplementary Figure S2. We then identified the encoding compositions with the best performance among the individual encoding compositions, di-encoding compositions, tri-encoding compositions and tetra-encoding compositions (Supplementary Table S2 and Figure S2). The results indicate that the best encoding compositions were 4, 3-4, 2-3-4 and 1-2-3-4. Performance comparison of the four selected encoding compositions is shown in Figure 2A. The results show that DeepTorrent achieved the best overall performance based on the model trained using the 1-2-3-4 tetra-encoding composition.

In terms of species-specific 4mC prediction, the dataset for each species was relatively small. As is well known, the use of small datasets for training DL models can cause overfitting [56]. Therefore, we used a larger dataset by combining all six species to train a base network to avoid overfitting. For each of the six species, we used the corresponding training dataset to retrain the species-specific model. The procedure was as follows: we first copied n layers of the base network as the first n layers of the species-specific model and then froze the first n layers of the species-specific model and merely fine-tuned the hyperparameters of other layers to retrain the species-specific model using species-specific training datasets. This strategy has been shown to be effective at improving the performance of classification problems with limited sample datasets in recent studies on image classification [57] and kinase-specific phosphorylation site prediction [37]. Figure 2B shows the training loss and accuracy of the base model with respect to the training epochs on both the training and validation data. During the process of training, the early stopping strategy was adopted, and we monitor accuracy changes when testing for when to stop training. The training process would be interrupted when the validation accuracy was no longer improved after 20 epochs. It is clear that the base model displayed similar training loss and validation loss for training epochs and achieved higher training accuracy (0.8241) and validation accuracy (0.7406) after about 170 epochs. This indicates that the base model provides a strong foundation for tuning the species-specific model based on transfer learning.

To illustrate how DeepTorrent learned effective feature representations, we used the t-SNE plot [58] to visualize feature representations of two dimensions that were automatically learned by DeepTorrent. The original feature representation is shown in Figure 3A. As can be seen, it is difficult to visually distinguish each species with the original feature representations. Based on the feature representations (Figure 3B) learned after the attention layer of the model, we could identify the principal components. Moreover, the feature representations (Figure 3C) after the second fully connected layer could be used to better identify and separate each species. These results suggest that DeepTorrent is able to learn good feature representations effectively.

Performance comparison with the existing methods on the Lin_2017 independent test dataset

To compare the performance of DeepTorrent with existing 4mC prediction methods, we performed comparisons with existing methods using independent dataset. For the independent test of 4mCPred [23], the positive and negative training datasets of each species’ benchmark dataset were randomly divided into 15 subsets of approximately equal size, of which 14 subsets were selected as the training datasets and the remaining one was used to test the model. For a fair and objective comparison, we used the same training datasets and the independent test datasets for each of the six species. Supplementary Figure S3 shows the ROC curves of each species-specific model with deep transfer learning. In addition, we performed an independent test of the species-specific models without transfer learning using the same training and independent test datasets and plotted the ROC curves in Supplementary Figure S4, for a comparison with models trained with transfer learning. As can be seen from Supplementary Figure S4, each species-specific model trained with transfer learning consistently achieved higher AUC values than its counterpart without transfer learning, e.g. 0.893 versus 0.832 on C. elegans, 0.911 versus 0.847 on D. melanogaster, 0.815 versus 0.783 on A. thaliana, 0.935 versus 0.796 on E. coli, 0.939 versus 0.856 on G. subterraneus and 0.929 versus 0.875 on G. pickeringii. The performance comparison results of species-specific models with and without transfer learning in terms of all evaluation metrics are provided in Supplementary Table S3. As can be seen, the species-specific models trained with transfer learning outperformed those trained without transfer learning in terms of all major evaluation metrics.

We performed an independent test to compare the performance of DeepTorrent to that of two state-of the-art methods: 4mCPred [23] and iDNA4mC [22]. The performance results for the six species are provided in Supplementary Table S4. As can be seen, compared to 4mCPred and iDNA4mC, DeepTorrent achieved the best performance in terms of all performance metrics (viz. Sn, Sp, Acc and MCC) for four species (viz. C. elegans, D. melanogaster, A. thaliana and G. pickeringii). For E. coli and G. subterraneus, DeepTorrent achieved the best performance in terms of Sp, Acc and MCC. In terms of the two major comprehensive metrics (Acc and MCC), DeepTorrent achieved the best performance on all six species. These results demonstrate that DeepTorrent is a powerful 4mC site predictor capable of accurately identifying potential m4C sites.

Ten-fold cross-validation test on the Lin_2017 training dataset

To evaluate the performance of DeepTorrent, we performed 10-fold cross-validation tests to compare DeepTorrent with several existing methods, including iDNA4mC [22], 4mCPred [23], 4mcPred-SVM [24], Meta-4mCPred [25] and 4mCCNN [27]. The performance results of these methods for the six species are provided in Supplementary Table S5. As shown in Supplementary Table S5, compared with these five methods, DeepTorrent achieved the best performance in terms of Acc and Sp across all the six species. We have also plotted the MCC values for the six methods with the six species, with the results shown in Figure 4. As can be seen, DeepTorrent achieved the highest MCC values for five out of the six species (with the exception of A. thaliana).

Cross-species validation

To examine the potential relationships between two different species, we further conducted cross-species validation using data from one species to train the DeepTorrent model and then applied the trained model to predict 4mC sites in other species. The rationale for this is that transfer learning can transfer the source domain knowledge (species-specific training data) to the target domain (another species) [23]. The cross-species performance results of DeepTorrent, iDNA4mC [22] and 4mCPred [23] are listed in Supplementary Tables S6–S8, respectively. Note that the results in Tables S7 and S8 were originally collected from [23]. In addition, we also conducted cross-species validation using the 4mcPred-SVM webserver [24]. The corresponding results of 4mcPred-SVM are listed in Supplementary Table S9. As can be seen from Supplementary Tables S6–S9, in cases where two species differ in terms of the source domain and target domain, 4mCPred achieved the strongest transfer learning capability from A. thaliana to D. melanogaster and G. subterraneus to G. pickeringii, whereas DeepTorrent showed the strongest ability to transfer knowledge in all other cases.

Ten-fold cross-validation test on the Li_2020 training dataset

Similar to the performance test described in Performanceevaluation on the Lin_2017 dataset, we first trained a base network on a large number of datasets integrated with the training datasets of the six species shown in Table 3. We then retrained the species-specific model for each of the six species using the corresponding species-specific training dataset. After that, we performed 10-fold cross-validation test and evaluated the species-specific performance of DeepTorrent model. The results are shown in Figure 5A and ROC curves are plotted in Figure 5B. The detailed performance results are provided in Supplementary Table S10. As can be seen, DeepTorrent achieved an AUC value of higher than 0.86 across all the six species and an average AUC value of 0.94 and Acc value of 0.87, respectively. These results show that DeepTorrent provides a reasonable predictive performance for species-specific 4mC prediction.

Performance comparison with other existing methods on the Li_2020 independent test dataset

To further evaluate the predictive capability of DeepTorrent, we performed the independent test and compared its performance with other existing methods using the additional independent test datasets. The performance results of all the compared methods are visualized in Figure 6, and ROC curves plotted in Figure 7. The detailed results are provided in Supplementary Table S11. The performance results of the other predictors were calculated using the corresponding web servers of these methods. As the web servers of iDNA4mC [22] and 4mCCNN [27] were unavailable, the performance results of these two methods were not included in the performance comparison. As can be seen from Figures 6 and 7 and Supplementary Table S11, DeepTorrent outperformed the other methods on all seven metrics with five species, except for E. coli, for which DeepTorrent outperformed the other methods in terms of six (of seven) performance metrics.