This paper proposes an optimized pedestrian and vehicle detection method based on deep learning technique. We optimize the convolutional neural network architecture by three mainly methods. The first one is the choice of the learning policy. The second one is to simplify the convolutional neural network architecture. The last one is careful choice of training samples. With limited loss of accuracy, we can greatly speed up the original deep learning method coming from CAFFE. The proposed system is developed on PCs and implemented on the platforms of both the PC and embedded systems. We can achieve around 90% accuracy when it is tested on an open-source dataset. On PCs with Intel email@example.comGHz CPU, the proposed design can reach the performance about 720×480 video at 25 frames per second. On the NVIDIA JETSON TX1 embedded system, the proposed design can reach the performance about 720×480 video at 5 frames per second.