Social media has become an important part of each individual's life and an invaluable tool for companies to know their customers better in the market. The problem of popularity prediction in social media has been studied extensively over the past few years. Yet, it is still a challenging task due to various factors, including the difficulty to measure the preference of viewers towards specific post contents, the influence of user popularity, and the properties of social media itself. Accordingly, this paper focuses on image popularity prediction by analyzing early popularity patterns of posts in the same content category, fused with user information data. We conduct extensive experiments on a dataset crawled from Instagram. The experimental results show that our proposed model achieves considerable performance in predicting the number of likes of an Instagram input photo. Moreover, we also provide in-depth analysis of the importance of category-specific information in popularity prediction.