Hand-Eye-Head Coordinated Learning

The objective of this project is to develop a highly adaptive method for hand-eye-head coordinated object manipulation system in an unstructured environment without requiring a complete description of the world and explicit camera calibration. Unlike conventional robot systems that depend very much on the availability of accurate global position of the manipulator and objects, the system under development learns, through interactive visual feedback, the unknown and nonlinear relationships among the sensed objects, the hand, and the visual sensors. The robot manipulator is equipped with a visual recognition system called SHOSLIF-O, which is used to recognize the objects and the robot hand from images. In the learning phase, several hierarchical networks are automatically generated to map the image-plane space to 3-D space and arm control space. In the performance phase, the network controls the manipulator to learn to perform some tasks, such as find an object to be detected, find the grasping position of the object, find the 3-D location of the object, move the arm to the object detected, pick up the object, move an object to another object, pour a cup of water into another cup, etc. Each of the above action is coded as a task command. The human user issues a single command for the machine to learn each task. At the performance phase, the human user issues a series of commands and the machine execute each task one at a time according to what it learned for each task.


Wey S. Hwang and John Weng, ``Autonomous Vision-Guided Robot Manipulation Control,'' in Proc. 3rd Asian Conference on Computer Vision, Hong Kong, Vol. II, pp. 503 - 510, January 1998. Click here to down load the paper (PostScript).
Wey S. Hwang and John Weng, ``Vision-Guided Robot Manipulator Control as Learning and Recall Using SHOSLIF,'' in Proc. 1997 International Conference on Robotics and Automation (ICRA), Albuquerque, NM, pp. 2862-2867, April 1997. Click here to down load the paper (PostScript).
W.-S. Hwang, S. J. Howden, J. Weng, ``Performing Temporal Action with a Hand-Eye System,'' in Proc. International Conference on Pattern Recognition, Vienna, Austria, vol. IV, pp. 35-39, Aug. 1996. Click here to down load the paper (PostScript).
S. Howden and J. Weng, ``Hand-eye coordinated learning using hierarchical space tessellation,'' Technical Report CPS 94-29, Department of Computer Science, MSU, April, 1994.
 Back To Weng's Home Page: http://web.cps.msu.edu/~weng/