WAG-NAT: Window Attention and Generator Based Non-Autoregressive Transformer for Time Series Forecasting

Time series forecasting plays a crucial part in many real-world applications. Recent studies have proven the power of Transformer to model long-range dependency for time series forecasting tasks. Nevertheless, the quadratic computational complexity of self-attention is the major obstacle to application. Previous studies focus on structural adjustments of the attention mechanism to achieve more efficient computation. In contrast, local attention has better performance than full attention in feature extraction and computation simplification due to the sparsity of the attention mechanism. Besides, in practice, the speed of inference is more significant, which is also a key factor. In response to these, we develop a novel non-autoregressive Transformer model based on window attention and generator, namely WAG-NAT. The generator allows one-step-forward inference. The window attention module contains a window self-attention layer to capture local patterns and a window interaction layer to fuse information among different windows. Experimental results show that WAG-NAT has a distinct improvement in prediction accuracy compared with RNNs, CNNs, and other previous Transformer-based models across various benchmarks. Our implementation is available at https://github.com/cybisolated/WAG-NAT.