Coordinating the Operation of a Robot’s Manipulator and Vision System

The effectiveness of autonomous service robots may be improved by coordinating the operation of the manipulators and the robot’s built-in vision system in the manipulation of objects. That allows the robot to turn on a switch, turn a lever, or hand an object to a human, for example. The tasks required to grasp and manipulate such objects include their visual detection (search, localization) and recognition, and also assessment of their position (relative to the robot’s coordinate system) and orientation. In many cases, these problems must be solved when the robot is in an indeterminate position relative to the object and/or the point (fixed or moving) to which it must be delivered. The robot’s vision system and its coordinate system are used for procedures associated with detection, recognition, and localization of the object and the intended delivery point. Coordination of the operation of the manipulators and the robot’s built-in vision system is necessary in order to plan manipulator control, taking account of the object’s position (in the robot’s coordinate system). Planning and implementation of manipulator control are real-time processes, which may pose problems if the robot’s built-in computer is insufficiently fast. In the present work, we present a method for coordinated operation of the manipulator and the robot’s built-in vision system. This method includes assessment of the object’s position relative to the robot; and an algorithm for planning manipulator control, taking account of the estimates obtained.