This work presents the use of a Gabor transform to determine the instantaneous modal parameters (natural frequencies, damping ratios and mode shapes) of a structure from its dynamical responses, such as free vibration or earthquake response. The Gabor transform is applied to the measured acceleration responses of a structural system for reconstructing the time-varying time series model in time-frequency domain. The instantaneous modal parameters of a time-varying system can be directly estimated from the time-varying model coefficients. The proposed procedure is further applied to process the dynamic responses of three-storey steel frames with or without nonlinear elements from the simulated earthquake events via SAP2000 to identify the differences among the frames. The linear or nonlinear behaviors of these steel frames under earthquake input are accurately reflected in the instantaneous natural frequencies that are obtained by the proposed approach.