AIRVL     Research Projects: Real-Time Vision

[ AIRVL Home] [ People ] [Research] [ Facilities]

Flexible operation of a robotic agent in an uncalibrated environment requires the ability to recover unknown or partially known parameters of the workspace through sensing. Of the sensors available to a robotic agent, visual sensors provide information that is richer and more complete than other sensors. However, the integration of vision sensors with a robotic system raises a number of issues that have not been addressed by traditional robotics and computer vision research.

Professor Papanikolopoulos' previous research introduced a framework called controlled active vision for efficient integration of the vision sensor in the feedback loop. This framework emphasized eye-in-hand robotic systems (the vision sensor is mounted on or close to the manipulator's end-effector) and was applied to the problem of robotic visual tracking and servoing with very promising results. Full 3-D robotic visual tracking was achieved at rates of 30 Hz with targets moving at maximum speeds of 80 cm/sec. Most importantly, the tracking was successful even under the assumption of poor calibration of the eye-in-hand system. Algorithms that incorporated the use of multiple windows and numerically stable confidence measures were combined with stochastic controllers in order to provide a satisfactory solution to the tracking problem. The special relations between the characteristics of computer vision and control algorithms were highlighted through a series of experimental results.

One of the most important contributions of this work was the introduction of adaptive control techniques in order to compensate for inaccurate modeling of the environment, such as depth estimation. One example was in the case of inaccurate knowledge of the features' depth where the displacements were fed to adaptive SISO controllers that estimated the desired robot motion. The human operator was integrated with the autonomous servoing modules through a sophisticated system architecture.

We have also developed robust techniques for the derivation of depth from a large number of feature points on a target's surface and for the accurate and high-speed tracking of moving targets. These techniques are used in a system that operates with little or no a priori knowledge of the object-related parameters present in the environment. The system is designed under the Controlled Active Vision framework and robustly determines parameters such as velocity for tracking of moving objects and depth maps of objects with unknown depths and surface structure. Such determination of intrinsic environmental parameters is essential for performing higher level tasks such as inspection, exploration, tracking, grasping, and collision-free motion planning. For both applications, the Minnesota Robotic Visual Tracker (a single visual sensor mounted on the end-effector of a robotic manipulator combined with a real-time vision system) is used to automatically select feature points on surfaces, to derive an estimate of the environmental parameter in question, and to supply a control vector based upon these estimates to guide the manipulator.

Five years ago, we implemented a flexible system that performs autonomous grasping of a moving target in a partially calibrated environment. The object of interest is not required to appear in a specific location, orientation, or depth, nor is it required to remain motionless during the grasp. The proposed system was derived using the Controlled Active Vision framework and provided the flexibility to robustly interact with the environment in the presence of uncertainty. The proposed approach was experimentally verified using the MRVT system to automatically select targets of interest and to guide the manipulator in the grasping of a target. More recently, the grasping system was expanded and improved with the use of pressure snakes.

We have also worked on a model-based approach for visual tracking and servoing in robotics. Deformable active models are proposed as an effective way for tracking a rigid or semi-rigid (possibly partially occluded) object in movement within the manipulator's workspace. Deformable models imitate, in real-time, the dynamic behavior of elastic structures. These computer-generated models are designed to capture the silhouette of rigid or semi-rigid objects with well-defined boundaries, in terms of image gradient. They consist of several hundred control points that are processed in parallel. By means of an eye-in-hand robot arm configuration, the desired motion of the end-effector is computed with the objective of keeping the target's position and shape invariant with respect to the camera frame. Optimal estimation and control techniques (LQG regulator) are used in order to deal with noisy measurements provided by the vision sensor. Experimental results show that the deformable models achieve robust tracking even when the target is partially occluded. These techniques have been applied to assembly tasks that involve a combination of computer vision and force robot control.

Copyright: © 1998-2001 by the Regents of the University of Minnesota
Department of Computer Science and Engineering. All rights reserved.
The University of Minnesota is an equal opportunity educator and employer.