In this work, we present a multiple classifiers system cascades an on-line learning RGB-D appearance model framework in which detection, recognition, and tracking are highly coupled for a wheelchair robot equipped with a Kinect sensor to improve the efficiency of the care assistance and quality of accompanying service. The on-line trained classifiers use the surrounding background as negative examples in the updating which allows the algorithm to choose the most discriminative features between the target and the background, incrementally adjust to the changes in specific tracking environment. Meanwhile, a depth clustering based human detection is proposed to extract human candidates. Accordantly, an on-line learning RGB-D appearance model is cascaded to strengthen the human tracking function by dealing with color, depth and position information from the identified caregiver. Consequently, several experiments have been conducted to demonstrate the effectiveness and feasibility in real world environments.