Graph Deep Factors for Probabilistic Time-series Forecasting

We propose a general and extensible deep hybrid graph-based probabilistic forecasting framework called GraphDF that is capable of learning complex non-linear time-series patterns globally using the graph time-series data to improve both computational efficiency and forecasting accuracy while learning individual probabilistic models for individual time-series based on their own time-series and the collection of time-series from the immediate neighborhood of the node in the graph. The GraphDF framework is data-driven, fast, scalable for real-time demand forecasting, and highly data efficient.

The state-of-the-art deep probabilistic forecasting methods focus on learning a global model that considers all time-series jointly or a local model learned from each individual time-series independently. In this work, we propose a deep graph-based probabilistic forecasting model that lies in between these two extremes. In particular, we propose a relational global model that learns complex non-linear time-series patterns globally using the structure of the graph to improve both computational efficiency and forecasting performance. Similarly, instead of modeling every time-series independently, we learn a relational local model that not only considers its individual time-series but the time-series of nodes that are connected to an individual node in the graph.

Furthermore, the proposed GraphDF framework applies to a significantly larger class of problems, which includes prior work as a special case. In particular, GraphDF naturally generalizes many existing models including those based purely on local and global models, or a combination of both. This is due to its flexibility to interpolate between purely non-relational models (either local, global, or both) and relational models that leverage the graph structure encoding the dependencies between the different time-series. The experiments demonstrate the effectiveness of the proposed deep graph-based probabilistic forecasting model in terms of its forecasting performance, runtime, and scalability.

Finally, we extend GraphDF to meet the incremental online scheme and derive the IOGraphDF model, which converges over a timespan to yield approximately accurate predictions as GraphDF, but takes a much shorter time to train and update.

We first introduce the deep graph-based probabilistic forecasting problem. Notably, this is the first hybrid deep graph-based probabilistic forecasting framework. The framework is comprised of a graph relational global component (described in Section 3.3) that learns the complex non-linear time-series patterns in the large collection of graph-based time-series data and a relational local component (Section 3.4) that handles uncertainty by learning a probabilistic forecasting model for every individual node in the graph that not only considers the time-series of the individual node, but also the time-series of nodes directly connected in the graph. This has the advantage of improving both forecasting accuracy and data efficiency.

The proposed framework solves the following graph-based time-series forecasting problem. Let $ G=(V,E,\mathcal {X}, \mathcal {Z}) $ denote the graph model where V is the set of nodes, E is the set of edges, and $ \mathcal {X}=\lbrace \boldsymbol {\mathrm{X}}^{(i)}\rbrace _{i=1}^{N} $ is the set of covariate time-series associated with the N nodes in G where $ \boldsymbol {\mathrm{X}}^{(i)} \in \mathbb {R}^{D \times T} $ is the covariate time-series data associated with node i. Hence, each node is associated with D different covariate time-series. Furthermore, $ \mathcal {Z}=\lbrace \boldsymbol {\mathrm{z}}^{(i)}\rbrace ^{N}_{i=1} $ is the set of time-series associated with the N nodes in G. The N nodes can be connected in an arbitrary fashion that reflects the dependence between nodes. Two nodes i and j that contain an edge $ (i,j) \in E $ in the graph G encodes an explicit dependency between the time-series data of node i and j. Intuitively, using these explicit dependencies encoded in G can lead to more accurate forecasts as shown in Figure 1. Furthermore, let $ \boldsymbol {\mathrm{z}}_{1:T}^{(i)} $ denote a univariate time-series for node i in the graph where $ \boldsymbol {\mathrm{z}}_{1:T}^{(i)} = [z_{1}^{(i)} \, \cdots \, z_{T}^{(i)}] \in \mathbb {R}^{T} $ and $ z_{t}^{(i)} \in \mathbb {R} $. In addition, each node i in the graph G also has D covariate time-series, $ \boldsymbol {\mathrm{X}}^{(i)} \in \mathbb {R}^{D \times T} $ where $ \boldsymbol {\mathrm{X}}^{(i)}_{:,t} \in \mathbb {R}^{D} $ (or $ \boldsymbol {\mathrm{x}}^{(i)}_{t} \in \mathbb {R}^{D} $) represents the D covariate values at time step t for node i. We also denote $ \boldsymbol {\mathrm{A}}\in \mathbb {R}^{N \times N} $ as the sparse adjacency matrix of the graph G where $ N=|V| $ is the number of nodes. If $ (i,j) \in E $, then $ A_{ij} $ denotes the weight of the edge (dependency) between node i and j, and $ A_{ij}=0 $ when $ (i,j) \not\in E $ otherwise.

We denote the unknown model parameters as $ \mathbf {\Phi } $. Our goal is to learn a generative probabilistic forecasting model described by $ \mathbf {\Phi } $ that gives the (joint) distribution on target values in the future horizon $ \tau $:

Graph Construction. For each dataset, we derive a graph where each node represents a machine with one or more time-series associated with it, and each edge represents the similarity between the node time-series i and j. The constructed graph encodes the dependency information between nodes. In this work, we estimate the edge weights using the radial basis function (RBF) kernel with the previous time-series observations as $ K(\boldsymbol {\mathrm{z}}_i,\boldsymbol {\mathrm{z}}_j) = \exp (-\frac{\Vert \boldsymbol {\mathrm{z}}_i-\boldsymbol {\mathrm{z}}_j\Vert ^2}{2\ell ^2}) $, where $ \ell $ is the length scale of the kernel.

The GraphDF framework aims at learning a parametric distribution to predict future values. In GraphDF, each node i and its time-series $ z^{(i)}_{t}, \forall t=1, 2, \ldots $ can be connected to other nodes and their time-series in an arbitrary fashion, which is encoded in the graph G. These connections represent explicit dependencies or correlations between the time-series of the nodes. Furthermore, we also assume that each node i and their time-series $ \boldsymbol {\mathrm{z}}^{(i)}_{1:t} $ are governed by two key components including (1) a relational global model (Section 3.3), and (2) a relational local random effect model (Section 3.4). As such, GraphDF is a hybrid forecasting framework. Both the relational global component and relational local component of our framework leverage the graph via the specific underlying model used for each component.

In the relational global component of GraphDF, we assume there are K latent relational global factors that determine the fixed effect to each node and their time-series. Specifically, the relational global model consists of an approach that leverages the adjacency matrix $ \boldsymbol {\mathrm{A}} $ of the graph G and $ \lbrace \boldsymbol {\mathrm{X}}_{:,1:t}^{(j)}, \boldsymbol {\mathrm{z}}_{1:t-1}^{(j)} \rbrace $ for learning the K relational global factors that capture the relational non-linear time-series patterns in the graph-based time-series data,

The GraphDF framework is defined in Equations (2)–(6). All the functions $ s_k(\cdot), b^{(i)}(\cdot), v^{(i)}(\cdot) $ take past observations and covariates $ \lbrace \boldsymbol {\mathrm{z}}_{1 : t-1}^{(j)}, \boldsymbol {\mathrm{X}}_{:,1 : t}^{(j)}\!\rbrace _{\!j=1}^{\!N} $, as well as the graph structure in the form of adjacency matrix $ \boldsymbol {\mathrm{A}} $ as inputs. We define $ \boldsymbol {\mathrm{w}}_i = [w_{i,1} \cdots w_{i,k} \cdots w_{i,K}] \in \mathbb {R}^{K} $ as the K-dimension embedding for time-series $ \boldsymbol {\mathrm{z}}^{(i)} $ where $ w_{i,k} \in \mathbb {R} $ is the weight of the k-th factor for node i. An overview of the GraphDF framework is depicted in Figure 2.

The relational global model learns K relational global factors from all time-series by a graph-based model. These relational global factors are considered as the driving latent factors. After the relational global factors are derived from the model, they are then used in a linear combination with weights given by embeddings for each time-series $ \mathbf {w}_i $, as shown in Equation (3).

The (stochastic) relational local component handles uncertainty by learning a probabilistic forecasting model for every individual node in the graph G that not only considers the time-series of the individual node but also the time-series of nodes directly connected. This has the advantage of improving both forecasting accuracy and data efficiency.

The random effects in the relational local model represent the local fluctuations of the individual node time-series. The relational local random effect for each node time-series $ b^{(i)} $ is sampled from the relational local model $ \mathcal {R}_{i} $, as shown in Equation (4). For $ \mathcal {R}_i $, we choose the Gaussian distribution as the likelihood function for sampling, but other parametric distributions such as Student-t or Gamma distributions are also possible. Compared to the relational global component of GraphDF from Section 3.3 that uses the entire graph G along with all the node multi-dimensional time-series to learn K global factors that capture the most important non-linear time-series patterns in the graph-based time-series data, the relational local component focuses on modeling an individual node $ i \in V $ and therefore leverages only the time-series of node i and the set of highly correlated time-series from its immediate local neighborhood $ \Gamma _i $. Hence, $ \lbrace \boldsymbol {\mathrm{z}}^{(j)}, \boldsymbol {\mathrm{X}}^{(j)}\rbrace , j \in \Gamma _i $. Intuitively, the relational local component of GraphDF achieves better data efficiency by leveraging the highly correlated neighboring time-series along with its own time-series. This allows GraphDF to make more accurate forecasts further in the future with less training data. We now introduce probabilistic GCRN and probabilistic DCRNN model that can be used as the stochastic relational local component in GraphDF.

3.4.2 Estimating Uncertainties via Probabilistic DCRNN. We also describe a probabilistic DCRNN for the relational local component of GraphDF. For a given node i, its relational local random effect is derived with respect to its past observations, covariates and those of its neighbors, denoted by $ \boldsymbol {\mathrm{Y}}_t^{(i)} \in \mathbb {R}^{(|C|+1) \times P} $ as defined in Equation (25). The diffusion convolution models the relational local random effect among nodes. The GRU structure is adapted with the diffusion convolution to allow the random effects to be forwarded in time.

\begin{align} \boldsymbol {\mathrm{R}}_t^{(i)} &= \sigma \big (\mathbf {\Theta }_{R}^{(i)} {\,\star _{\mathcal {G}}\,}{} \left[\boldsymbol {\mathrm{Y}}_t^{(i)},\boldsymbol {\mathrm{H}}_t^{(i)}\right] + \boldsymbol {\mathrm{b}}_R^{(i)} \big), \end{align} (28)

\begin{align} \boldsymbol {\mathrm{U}}_t^{(i)} &= \sigma \big (\mathbf {\Theta }_{U}^{(i)} {\,\star _{\mathcal {G}}\,}{} \left[\boldsymbol {\mathrm{Y}}_t^{(i)},\boldsymbol {\mathrm{H}}_t^{(i)}\right] + \boldsymbol {\mathrm{b}}_U^{(i)} \big), \end{align} (29)

\begin{align} \boldsymbol {\mathrm{C}}_t^{(i)} &= \tanh \big (\mathbf {\Theta }_{C}^{(i)} {\,\star _{\mathcal {G}}\,}{} \left[\boldsymbol {\mathrm{Y}}_t^{(i)},\boldsymbol {\mathrm{H}}_t^{(i)}\right] + \boldsymbol {\mathrm{b}}_C^{(i)} \big), \end{align} (30)

\begin{align} \boldsymbol {\mathrm{H}}_t^{(i)} &= \boldsymbol {\mathrm{U}}_t^{(i)}\odot \boldsymbol {\mathrm{H}}_{t-1}^{(i)} + \left(1 - \boldsymbol {\mathrm{U}}_t^{(i)}\right)\odot \boldsymbol {\mathrm{C}}_{t}^{(i)}, \end{align} (31)

where $ \mathbf {\Theta }_{R}^{(i)}, \mathbf {\Theta }_{U}^{(i)}, \mathbf {\Theta }_{C}^{(i)} \in \mathbb {R}^{P\times R} $ denote the parameters corresponding to different filters, $ \boldsymbol {\mathrm{H}}_t^{(i)} \in \mathbb {R}^{(|C|+1)\times R} $ is the hidden state for node i and its neighbors $ \Gamma _i $, R is the number of hidden units in the relational local model. $ \boldsymbol {\mathrm{b}}_{I}^{(i)}, \boldsymbol {\mathrm{b}}_{F}^{(i)}, \boldsymbol {\mathrm{b}}_{C}^{(i)}, \boldsymbol {\mathrm{b}}_{O}^{(i)} \in \mathbb {R}^{R} $ are bias vector parameters. The graph convolution in equations above is performed with the submatrix $ \boldsymbol {\mathrm{L}}^{(i)} $ taken from the Laplacian matrix $ \boldsymbol {\mathrm{L}} $ of the graph G that explicitly models the important and meaningful dependencies between the multi-dimensional time-series data of each node. The matrix $ \boldsymbol {\mathrm{L}}^{(i)} $ consists of rows and columns corresponding to node i and its neighbors $ \Gamma _i $. With the hidden state $ \boldsymbol {\mathrm{H}}_t^{(i)} $, the relational local random effect $ b_t^{(i)}(\cdot) $ is calculated similarly to Equations ( 26) and ( 27).

To train a GraphDF model, we estimate the parameters $ \mathbf {\Phi } $, which represent all trainable parameters ($ \boldsymbol {\mathrm{W}} $, etc.) in the relational global and relational local model, as well as the parameters in the embeddings. We leverage the maximum likelihood estimation,

In this section, we define a few of the GraphDF model variants investigated in Section 5.

• GraphDF-GG: This is the default model in our GraphDF framework, where we use a graph model to learn the K relational global factors (Section 3.3) and the probabilistic local graph component from Section 3.4.1 as the relational local model.
  
• GraphDF-GR: This model variant from the GraphDF framework uses the GCRN from Section 3.3 to learn the K relational global factors from the graph-based time-series data and leverages a simple RNN for modeling the local random effects of each node.
  
• GraphDF-RG: This model variant from the GraphDF framework uses a simple RNN to learn the K global factors and fixed effects of the nodes and for the relational random effects of the nodes, we leverage the probabilistic graph component from Section 3.4.1 as the relational local model.
  

The GraphDF framework is flexible with many interchangeable components. Importantly, the relational global component (Section 3.3) of GraphDF is completely interchangeable. In particular, this component uses the graph-based time-series data to learn the K global factors and fixed effects of the nodes. Similarly, one can also leverage any arbitrary relational local model (Section 3.4) for obtaining the relational local random effects of the nodes.

We used two real-world datasets in our experiments; Google trace data and Adobe trace data. Table 1 shows the statistics and properties (e.g., edge density and average degree).

Google Trace. The Google trace dataset records the activities of a cluster of $ 12,\!580 $ machines for 29 days since 19:00 EDT on May 1, 2011. The CPU and memory usage for each task are recorded every 5 minutes. The usage of tasks is aggregated to the usage of associated machines, resulting time-series of length $ 8,\!354 $.

Adobe Workload Trace. The Adobe trace dataset records the CPU and memory usage of $ 3,\!270 $ nodes in the period from October 31 to December 5 in 2018. The timescale is 30 minutes, resulting time-series of length $ 1,\!687 $.

For the opportunistic workload scheduling case study in Section 6, we need to train a model fast within a few minutes and then forecast a single as well as multiple timesteps ahead, which are then used to make a decision on whether the current resources are enough or if we should instead scale up or down. Therefore, models must be able to be trained fast within a few minutes. To ensure the models are trained fast within minutes, we use six observations in the time-series data for training across all experiments. Furthermore, as in most time-series forecasting problems, the future CPU usage of machines is highly dependent on the most recent observations than those in the distant past. We set the number of embedding dimensions as $ K=10 $ in $ \mathbf {w}_i \in \mathbb {R}^{K} $ and use time feature series as covariates. We set the embedding dimension to $ K=10 $ in $ \mathbf {w}_i \in \mathbb {R}^{K} $ and used $ D=5 $ covariates for each time-series. Similar to DF [75], the time features (e.g., minute of hour, hour of day) are used as covariates. We derive a fixed graph using RBF on the past observations.

The three models described in Section 3.6 are evaluated against four state-of-the-art probabilistic forecasting methods including Deep Factors, DeepAR [31], MQRNN [76], and NBEATS [57]:

• Deep Factors is a generative approach that combines a global model and a local model. To ensure a fair comparison, we modified DF to solve the same problem formulated in Equation (1), and thus the DF version used for comparison uses the same inputs as GraphDF. Unless otherwise mentioned, we use the same experimental setup as mentioned in the DF article. In particular, as suggested by the authors, we use the Gaussian likelihood in terms of the random effects in the deep factors model. We use 10 global factors with an LSTM cell of 1-layer and 50 hidden units in its global component, and 1-layer and five hidden units RNN in the local component. We also use suggested hyperparameters for other compared baselines.
  
• DeepAR is an RNN-based global model, we use an LSTM layer with 50 hidden units in DeepAR.
  
• MQRNN is a sequence model with quantile regression and NBEATS is an interpretable pure deep learning model. For MQRNN, we use a GRU bidirectional layer with 50 hidden units as encoder and a modified forking layer in the decoder.
  
• For N-BEATS, we use an ensemble modification of the model and take the median value from 10 bagging bases as results.
  

All methods are implemented using MXNet Gluon [2, 16]. The Adam optimization method is used with a default initial learning rate of 0.001 to train all models. The training epochs are selected by grid search in $ \lbrace 100, 200, \ldots , 1000\rbrace $. An early stopping strategy is leveraged if weight losses do not decrease for 10 continuous epochs. We used a learning rate decay factor of 0.5, minimum learning rate of $ 5*10^{-5} $, Xavier as the weight initializer, and trained for 500 epochs on Adobe data and 100 epochs for the Google dataset. Some hyperparameters are specific to our method: In GraphDF, we set the order $ L=2 $ in Equation (10). A small number of the order indicates that the model makes forecasts based more on neighboring nodes than those more distant. For other methods, we use default hyperparameters given by the Gluonts implementation if not otherwise mentioned.

To evaluate the probabilistic forecasts, we use the quantile loss defined as follows: given a quantile $ \rho \in (0, 1) $, a target value $ \boldsymbol {\mathrm{z}}_t $ and $ \rho $-quantile prediction $ \widehat{\boldsymbol {\mathrm{z}}}_t(\rho) $, the $ \rho $-quantile loss is

To answer the research questions, we investigate the proposed GraphDF framework with various forecast horizons including $ \tau = \lbrace 1, 3, 4, 5\rbrace $.

The results for single and multi-step ahead forecasting are provided in Table 2 and Table 3–5, respectively, where the best result for every dataset and forecast horizon are highlighted in bold. We run 10 trials for each model and report the average for $ \rho =\lbrace 0.1, 0.5, 0.9\rbrace $, denoted as the P10QL, P50QL, and P90QL, respectively. In all cases, we observe that the GraphDF models outperform the state-of-the-art method across all datasets and forecast horizons. Furthermore, we observe that in most cases, the GraphDF-GG variant that uses GCRN for both the relational global and relational local components outperforms the other variants. The second best GraphDF model is GraphDF-GR followed by Graph-RG.

To understand the overall performance and variance of the models, we show boxplots for each model in Figure 3. Strikingly, we observe that the GraphDF models provide more accurate forecasts with significantly lower variance.

Training and inference runtime performance results are shown in Tables 6 and 7, respectively. All forecasting models are trained using only six previous values for each time-series in the collection (Table 6). As expected, the relational global model is significantly faster and more scalable than the global model used in Deep Factors. In particular, we see that the runtime of our GraphDF model that uses GCRN for the relational global model with RNN as the local model is significantly faster than Deep Factors that uses the same local model, but differs in the global model used. This is due to the fact that in the state-of-the-art DF model, all time-series are considered equivalently and jointly when learning the K global factors. This can be thought of as a fully connected graph where each time-series is connected to every other time-series. In comparison, the relational global components of GraphDF leverage the graph that encodes explicit dependencies between the different time-series, and therefore, does not need to leverage all pairwise time-series, but only a smaller fraction of those that are actually related.

In terms of inference, all models are fast taking only a few seconds as shown in Table 7. For inference, we report the time (in seconds) to infer the next six values in each time-series in the collection. In all cases, the GraphDF model variants are significantly faster than DF across both Google and Adobe workload datasets.

To evaluate the scalability of GraphDF, we vary the training set size (i.e., the number of previous data points per time-series to use) from $ \lbrace 2, 4, 8, 16, 32\rbrace $. Figure 4 shows that GraphDF scales nearly linear as the training set size increases from 2 to 32. For instance, GraphDF takes around 15 seconds to train using only two data points per time-series and 30 seconds using four data points per time-series, and so on. We also observe that for the Adobe data, GraphDF is always about 3 times faster compared to DF across all training set sizes.

To evaluate the performance of IOGraphDF, we design experiments to answer the following research questions:

• Does IOGraphDF yield more accurate predictions than GraphDF?
  
• Does IOGraphDF outperform GraphDF regarding training and prediction runtime?
  
• Does IOGraphDF perform better in the opportunistic workload scheduling task?
  

Experimental Setting. To compare the performance and runtime between IOGraphDF and GraphDF model, we simulate a streaming procedure containing 100 timesteps, where new values are received by both models at each time step. At each time step, new values arrive, a new GraphDF instance is initialized and then trained with the just arrived values. By contrast, IOGraphDF model only needs to be initialized once at the beginning, and the same IOGraphDF instance is incrementally updated from the previous time step with newly arrived data. Incoming values are assumed to be more dependent on near observations than those in the distant past, therefore, we maintain a shifting window size of 9 to include the most recent observations relative to t. When t increments, the oldest values are removed from the window and newly arrived values are appended to the window. The window values are split sets for data training and fitting procedure. Using the Google dataset, for GraphDF model, the number of training epochs is set as 100, while for IOGraphDF model, the number of local iterations (the training times upon each shifting window) only needs to be set as a smaller number 50, because after the model is initialized, the existed model preserve learned information from previous time steps. In the inference stage, values in $ \tau =3 $ steps ahead are predicted and evaluated with ground truth.

Experimental Results. Overall, we observe from Figure 5 (left) and Table 8 that IOGraphDF is significantly faster, with very comparable loss (as shown in Figure 5 (right) and Table 8). Hence, IOGraphDF sacrifices a small amount of accuracy for a significant speedup.

We further conducted experiments on 100 timesteps using the best variants GraphDF-GG and IOGraphDF-GG. As observed in Figure 5 (Right), the p50QL (i.e., MAPE) of GraphDF (in blue) and IOGraphDF (yellow dashed line) for each 100 streaming timestep is plotted. In the first few timesteps, IOGraphDF has much higher errors than GraphDF; however, IOGraphDF performance converges quickly to that of GraphDF's, as two lines in Figure 5 are mostly overlapping after streaming timestep 20.

In both offline and online models, the expected runtime on training is positively related to the number of values on input and output. In the offline GraphDF model, each time new values arrived, a new model has to be created and trained using the new values, and only recent values are taken as input, hence, this causes the loss of earlier observations as possibly useful information. In our proposed IOGraphDF, the input is only modeled upon a fixed amount of recent values. Since the same model instance is used and kept updated over time, IOGraphDF has the advantage of leveraging earlier observations for forecasting, as learned by the model parameters. As a consequence, we set the local iterations, the number of times data values are used to update IOGraphDF model, a small number 50, which results a much faster IOGraphDF while still preserving a high accuracy. The runtime comparison between GraphDF and IOGraphDF over 100 timesteps is shown in Figure 5 (Left). A similar result can be observed from the experiment using Adobe dataset using the same warm start period (20 steps i.e., 10 hours), as shown in Figure 6. We also summarize the average and deviation runtime in Table 9.

Following the setup in Section 5.2, we design experiments to investigate the performance of IOGraphDF model variants over multi-step ahead prediction, and compare them against previous results from GraphDF variants. Since IOGraphDF models take advantages of incremental training, for a fair comparison, we report the result for IOGraphDF models after a warm start period (set as 20 timesteps). The values in the warm start period are used to train the IOGraphDF models, but not used for loss comparison. The values after the warm period are then used to evaluate and compare the prediction loss with the GraphDF variants. The result is shown in Table 10. We observe that on the Google dataset, IOGraphDF variants reach very close result to the GraphDF counterparts, while IOGraphDF models only require significantly less training runtime than GraphDF models, as shown in Table 8. For the Adobe dataset, IOGraphDF models not only require less runtime but also obtain more accurate predictions with smaller losses in most cases (highlighted in the column IOGraphDF-GG).

We further investigate the effects of hyperparameters upon the IOGraphDF model with extensive experiments. The length of the warm start period and the number of relational global factors K are selected from the range of $ \lbrace 5, 10, 20, 50\rbrace $. We report p50QL for forecasting 3-step ahead on Google dataset with combinations of these hyperparameters and the result is shown in Table 11. From the result table, we observe (1) When the number of relational global factor K is fixed, i.e., for each column of Table 11, the longer the warm start period is, the better performance IOGraphDF achieves. (2) When the number of warm start period is small, the performance is drastically improved as the number of relational global factors K increases, as shown in the first two rows of Table 11, and the best performance is achieved when $ K=50 $. We suggest this can be due to the insufficient training of the model since the number of warm start period is small. (3) When the number of warm start period is large, the performance improves little or even worsens as the number of relational global factors K increases, as shown in the last two rows of Table 11. This can be caused due to the excessive amount of model parameters which leads to overfitting.