Coupling humanoid walking pattern generation and visual constraint feedback for pose-regulation and visual path-following

In this article, we show how visual constraints such as homographies and fundamental matrices can be integrated tightly into the locomotion controller of a humanoid robot to drive it from one configuration to another (pose-regulation), only by means of images. The visual errors generated by these constraints are stacked as terms of the objective function of a Quadratic Program so as to specify the final pose of the robot with a reference image. By using homographies or fundamental matrices instead of specific points, we avoid the features occlusion problem. This image-based strategy is also extended to solve the problem of following a visual path by a humanoid robot, which allows the robot to execute much longer paths and plans than when using just one reference image. The effectiveness of our approach is validated with a humanoid dynamic simulator. •Visual constraints embedded in a MPC-based walking pattern generation.•Comparison of different strategies to handle the orientation of the humanoid motion.•Navigation by following a visual path instead of following just one image.•Visual constraints-driven locomotion implemented in a dynamic simulator.