A Resolution Extrapolation Method for Transformers Oriented Towards Scientific Computing

Solving partial differential equations (PDEs) can accurately simulate the distribution, propagation, and thermal properties of electromagnetic fields in cables, thereby ensuring the efficient and safe operation of power systems. Although existing PDE solvers based on the Transformer can simulate multi-physical systems and achieve unification in multi-physics field modeling, they lack the capability to extrapolate resolution and fail to unify the resolution of physical fields. This paper focuses on the application of Transformer-based neural networks in multi-physics field simulation computations. By integrating both the spatial and temporal characteristics of physical field data, we propose a novel Spatial-Temporal Positional Encoding (STPE) method, enabling the model to learn the temporal and spatial differential relationships inherent in PDEs. With this positional encoding, we observe a "diagonalization"phenomenon in the model's attention scores. However, as the model extrapolates to higher resolutions, this "diagonalization"weakens, and the model's prediction accuracy significantly decreases. Therefore, we further propose Spatially Localized Attention (SPA) that allows Transformer-based PDE solvers to have the capacity for resolution extrapolation, enabling them to effectively adapt and extend to high-resolution physical fields during the inference stage, even if trained only with low-resolution physical field data. Experimental results demonstrate that after applying STPE, the model not only retains the capability to uniformly model multiple physical fields but also shows improved prediction accuracy on forward tasks, outperforming PINNs, U-Net, and FNO comprehensively. Additionally, the introduction of SPA can effectively alleviate the problem of accuracy decline when the model performs resolution extrapolation.