Identifying a user precisely through mobile-device-based sensing information is a challenging and practical issue as it is usually affected by context and human-action interference. We propose a novel deep learning-based lightweight approach called DeepIdentifier. More specifically, we design a powerful and efficient block, namely funnel block, as the core components of our approach, and further adopt depthwise separable convolutions to reduce the model computational overhead. Moreover, a multi-task learning approach is utilized on DeepIdentifier, which learns to recognize the identity and reconstruct the signal of the input sensor data simultaneously during the training phase. The experimental results on two real-world datasets demonstrate that our proposed approach significantly outperforms other existing approaches in terms of efficiency and effectiveness, showing up to 17 times and 40 times improvement over state-of-the-art approaches in terms of model size reduction and computational cost respectively, while offering even higher accuracy. To the best of our knowledge, DeepIdentifier is the first lightweight deep learning approach for solving the identity recognition problem. The dataset we gathered, together with the implemented source code, is public to facilitate the research community.