The objective of this study is to develop a novel and efficient model for forecasting the nonlinear behavior of structures in response to time-varying random excitation. The key idea is to design a deep learning architecture to leverage the relationships, between external excitations and structure’s vibration signals, and between historical values and future values, within multiple time-series data. The proposed method consists of two main steps: the first step applies a global attention mechanism to combine multiple-measured time series and time-varying excitation into a weighted time series before feeding it to a temporal architecture; the second step utilizes a self-attention mechanism followed by a fully connected layer to predict multi-step future values. The viability of the proposed method is demonstrated via two case studies involving synthetic data from a three-dimensional (3D) reinforced concrete structure and experimental data from an 18-story steel frame. Furthermore, comparison and robustness studies are carried out, showing that the proposed method outperforms conventional methods and maintains high performance in the presence of noise with an amplitude of less than 10%.