This paper reports the development of a vision system to automatically classify work-pieces with respect to their shape and color together with determining their location for manipulation by an in-house developed pick-and-place robot from its work-plane. The vision-based pick-and-place robot has been developed as part of a smart flexible manufacturing system for unloading work-pieces for drilling operations at a drilling workstation from an automatic guided vehicle designed to transport the work-pieces in the manufacturing work-cell. Work-pieces with three different shapes and five different colors are scattered on the work-plane of the robot and manipulated based on the shape and color specification by the user through a graphical user interface. The number of corners and the hue, saturation, and value of the colors are used for shape and color recognition respectively in this work. Due to the distinct nature of the feature vectors for the fifteen work-piece classes, all work-pieces were successfully classified using minimum distance classification during repeated experimentations with work-pieces scattered randomly on the work-plane.