In this paper, we propose a framework to infer different people's activity from the view of both the geographical habit and temporal habit of user. Such a personal activity inference framework is a crucial prerequisite for intelligent user experience, and power management of smart phones. By analyzing the real activity log data, we extract 3 kinds of features: 1)The geographical feature captures the user's activity preference of places, 2)The temporal feature records the routine habit of user's activity, 3)The semantic feature obtained from location-based social network can be used as an activity reference of public opinion for each location. Finally, we hybrid the features to build a Semantic-based Activity Inference Model (SAIM). To evaluate our proposed framework SAIM, we compared it with the state-of-art methods over a real dataset. The experimental results show that our framework could accurately inference user's activity and each feature of the three has different inferring ability for different user.