Pen Recognition and Control
This project aims at enabling the PincherX 100 Robot Arm to grab a pen that is held in front of it.
Video Demo
Vision pipeline
To help the PincherX 100 Robot Arm to grab a pen, we take advantage of the depth and color sensors in the Intel RealSense D435i camera. To locate a pen in the 3D space, we use the following pipeline:
- Locate pen in 2D space using RGB image stream
- Align depth map to the RGB image
- Threshold the aligned image in the HSV color space to locate the pen
- Filter the image with dilation and erosion operations to reduce noise
- Find contours of the pen
- Calculate the centroid of the pen
Since now we know the centroid of the pen in 3D space, we can move on to the robot control pipeline.
Robot control
To grab the pen given its 3D coordinate in space (from the camera), we use open loop control to move the gripper towards the pen and grab it.