Image segmentation is a very important process in digital image/video processing and computer vision applications. It is often used to partition an image into separated parts for further processes. For some applications (i.e., concept-based image retrieval), a successful segmentation algorithm is necessary to identity the objects effectively. In addition, how to tag the objects after the segmentation associated with keywords is also a challenge for researchers. In this study, we proposed a color differentiated fuzzy c-means (CDFCM) framework for effective image segmentation to achieve segmented objects within image which is useful for further annotation. In our experiments, we compared our approach with other FCM techniques on synthetic image with excellent performance. Furthermore, CDFCM outperforms other approaches by using the Berkeley image segmentation data set with layered annotation, which can be applied for additional operations.