This paper proposes a deep learning-based method on environmental monitoring, targeting on marine contamination especially for marine oil pollution. With limited training data, it is very challenging to distinguish the oil pollution area from the ocean due to their similar colors and motions. Thus, simply optical flow feature would not be able to identify the pollution area from the ocean and its movement direction. To tackle the above challenges, image segmentation was first chosen to segment input images to different areas. Since the color and shape features of an oil pollution are not fixed, this paper takes advantages of SVM to identify oil pollution areas from the ocean so that ocean spill event can be detected based on their colors and motion energy. Furthermore, to judge this ocean spill condition, optical flow is then calculated to find the movement of polluted areas. Experimental results prove the superiority of our proposed method.