Traffic time prediction is a classical problem in intelligent transportation domain, which has attracted lots of attention from the research community in last three decades. The existing relevant works have been focused on how to predict the short-term traffic time for paths and roads. In fact, users may have the demand to know the future traffic time in advance as for making personal or commercial schedule. Long-term traffic time prediction is thus an emerging challenging task as there exist many complicated factors that may affect traffic situations, such as weather and congestion conditions. In this paper, we propose a novel deep learning-based framework named Deep Ensemble Stacked Long Short Term Memory (DE-SLSTM), which aims to solve the prediction bias during traffic congestion. To improve the model performance, we integrate the weather effect into the DE-SLSTM for predicting the long-term traffic time. Through a series of experiments, the proposed DE-SLSTM framework is verified to demonstrate excellent performance in terms of effectiveness. To the best of our knowledge, this is the first work on long-term traffic time prediction that considers deep learning techniques.