A cost effective approach to establish benchmark for automatic workpiece classification systems

The utility of an automatic workpiece classification system depends primarily on the extent to which its classification results are consistent with users' judgments. Thus to evaluate the effectiveness of an automatic classification system it is necessary to establish classification benchmarks based on users' judgments. Such benchmarks are typically established by having subjects perform pair comparisons of all workpieces in a set of sample workpieces. The result of such comparisons is called a full-data classification. However, when the number of sample workpieces is very large, such exhaustive comparisons become impractical. This paper proposes a more efficient method, called lean-data classification, in which data on some pair comparison are used to infer the complete pair comparison results. The proposed method has been verified by using a set of 36 sample workpieces. The result revealed that the method could produce a classification that was 78% consistent with the full-data classification while using only 40% of the total data.