In thin film ferroelectric devices, switching of ferroelastic domains can significantly enhance electromechanical response. Previous studies have shown disagreement regarding the mobility or immobility of ferroelastic domain walls, indicating that switching behaviour strongly depends on specific microstructures in ferroelectric systems. Here we study the switching dynamics of individual ferroelastic domains in thin Pb(Zr0.2,Ti0.8)O3 films under electrical and mechanical excitations by using in situ transmission electron microscopy and phase-field modelling. We find that ferroelastic domains can be effectively and permanently stabilized by dislocations at the substrate interface while similar domains at free surfaces without pinning dislocations can be removed by either electric or stress fields. For both electrical and mechanical switching, ferroelastic switching is found to occur most readily at the highly active needle points in ferroelastic domains. Our results provide new insights into the understanding of polarization switching dynamics as well as the engineering of ferroelectric devices.