Controlling an Industrial Robotic Arm using Computer Vision
by Martin S. Mason and Laurent Coudert
1. Overview
2. Using the Software
a. Preliminaries
b. Setting up the Serial Port connection to the Arm
c. Vision System Calibration
d. Running the Controller Software for the Arm
3. Using the Mouse to Direct the Robot Arm
a. Quick start guide to using the arm
b. Selecting Targets with the mouse
c. Moving the arm
d. Picking and placing objects
4. Automatic Tracking of Objects
5. Additional Functions
6. Appendices
a. Perspective Transform
b. Fiducial Tracking
c. Software Documentation
i. Python Software
ii. MelfaBasic Software
d. Troubleshooting
e. Safety
Controlling an Industrial Robotic Arm using Computer Vision
1. Overview
The following tutorial describes one way to use a vision system to identify and track objects and control a robotic arm to pickup those objects. The integration of computer vision with industrial robotic arms is one of the backbones of industrial robotics[1]. The purpose of the vision system is to identify the object and determine its position and orientation relative to workspace of the robotic arm. The position and orientation output from the vision system are transformed into the workspace of the industrial arm and then the controller of the robotic arm is used to generate a set of joint angles for the arm.
- Mitsubishi RV-2AJ Robotic Arm
- Custom fiducials which can be printed from this document.
- Python 2.6 with pyserial and wxwidgets
- Logitech Quickcam Fusion or equivalent camera
- Roborealm Vision API
- Microsoft Windows Platform (98, ME, XP or Vista)
A VGA resolution camera is required since it has fixed focus and is placed above and at an angle from the workspace as shown in figure 1 below.
Fig 1. Overview of experimental workspace including camera, arm and controller.
2. Using the software:
a. Preliminaries:
Make sure that python 2.6 is installed and the Roborealm directory is in the path. To install Python go to www.python.org and use the Windows installer, or use the Python 2.6.3 installer included on the CD. To use Roborealm, go to www.roborealm.org and download a 30 day free trial of the software, install it and then add it to the path by adding the install path (Which defaults to “C:/Program Files/Roborealm/”) to the path environment variable.
b. Setting up the serial port connection to the arm:
To run the program, double click on the Main.py module. A User Interface Window as shown in figure 2 below should appear and an image from the camera should appear in a second window.
Fig 2. The User Interface Window. This is your main window for interacting with the robot.
Once the arm control and vision windows open successfully, connect to the serial port that is currently attached to the robot. The device manager can be used to determine what port the Robot arm is currently connected to.
Use the Text box next to the “Connect” button to select the desired serial port and then click the “Connect” button to connect to the robot(See figure 4 below). The status box will show that the software has successfully connected to the robot. If the software cannot connect to the serial port, check the connections and the device manager. If that fails, it may be that another program is using the serial port, so close all programs and then re-run the Main application to connect to the serial port.
Be sure that the robot is set to use 19200 baud, 8 bits, even parity, 2 stop bit with no flow control. This is the default setting for Mitsubishi robots, but if there are continued difficulties consult the Mitsubishi manual[2] on setting the serial port properties.
c. Vision System Calibration:
Once the robot is successfully connected, the software must be calibrated by choosing the “Calibrate Button”. This opens up the “Calibration Window” (See figure 5 below). If the arm obscures the workspace, use the teach pad to move the arm out of the way so that the entire workspace is visible.
Fig 5. The calibration window allows perspective correction and image calibration.
The “Change Perspective” slider should be adjusted until the workspace is square in the camera window. See the two images in figure 6 to observe the correct final result of the perspective transform. The perspective transform is explained in detail in Appendix 1.
Fig 6. The left hand image shows the original distorted image. The right hand image shows the result after the proper perspective transform has been applied.
Once the image is square, the calibration points need to be selected. Toggle the “Select Calibration Points” button down. Four points must be placed in a Z shape pattern starting at the upper left hand corner of the workspace as indicated in figure 7. Select each of the corners of the workspace by clicking on the image. With each click, a colored cross should appear indicating the calibration point that has been selected. Click slowly and carefully for best results. Failing to do so will mean that the robot arm’s workspace will not match the camera’s workspace and result in erratic behavior. If a mistake is made in placing one of the calibration points, keep clicking until all the subsequent points have been placed and then reposition the problem point.
Fig 7. Select the calibration points in a Z shape as shown. The upper left hand corner of the block should have the white cross, the upper right hand corner the yellow cross, the lower left the green cross and the lower right the blue cross.
Once the calibration is completed, the transform and scaling factors between the visual space and the arm workspace will be displayed in the calibration window status box. Before the Calibration window is closed, switch the “Select Calibration Points” toggle to the off position. Then close the window by selecting OK.
d. Running the Controller Software for the Arm:
Now the program that runs in the robot controller can be initialized. The serial port connection to the arm should already be started(See part 2b). First choose “Stop Program”, then “Load Program” and finally “Start Program” as shown in the figure 8. The serial bus to the arm runs slowly, so watch the status window for “Write Succeeded” before executing the next command.
Once the program is successfully loaded and started on the controller, the Green Status Indicator under “Start” on the controller should light. Commands can now be sent to the robot through the interface.
Safety Note: The robotic arm is very powerful and can and probably will kill you. Be sure that the servos are turned off before putting any part of your body in the workspace. For complete safety details, please see appendix 5.
To test that the system is fully operational, click on the “Toggle Gripper” button to open and close the gripper. Leave the gripper in the open position before proceeding. If the gripper fails to function at this point, go back and check each of the steps in the section and then refer to Appendix 4 on trouble shooting.
3. Using the Mouse to Direct the Robot Arm:
a. Quick start guide to using the software.
- Connect the Serial port physically to the computer and in software using the Connect Button. (2b)
- Calibrate the Perspective Transform using the Perspective slider bar. (2c)
- Calibrate the image space to the workspace using the four point calibration. (2c)
- Stop, Load and Start the Program (2d)
- Test the Gripper.
b. Selecting Targets with the mouse.
At this point the robot arm will be directed using mouse clicks on the image. First choose the “Select Target Point Button”. The status box will report “Selecting On”. Clicking on the image will create a red crosshair and generate the appropriate X and Y coordinates for the arm coordinates for the arm controller. To start, be sure to click near the center of the workspace since clicking outside the workspace will cause the controller to reject the point and make a series of angry noises. Notice that each time there is a click, the arm message box(figure 7) will display the generated coordinates for that point.
c. Moving the Arm
Once the displayed coordinates are satisfactory, the arm must be moved to the selected point. First click the “Servos On” button. The Green Status Light for the Servo On indicator on the robot controller will light. Be sure that the area around the robot arm is completely clear of all body parts and valuables. Monitor the emergency stop button on the teach pad at all times while the servos are on. Click in the robot workspace again to generate a new set of coordinates. Failing to generate a new set of coordinates will result in the controller software crashing and require a reset of the program. (See Appendix 5 Troubleshooting) Now the arm can be moved to the indicated point by clicking the “Move Arm” button. Once the arm has moved to the indicated point, press the “Change Height” button to raise and lower the arm relative to the workspace. The heights are currently set to 64 mm and 105 mm above the workspace. Changing these and many other parameters is explained in the code documentation in Appendix 3. Warning, if the origin parameters of the robot have been modified, follow the instructions in the Mitsubishi controller manual to reset the origin to the factor specification.
d. Picking and placing objects
Practice placing objects at different points in the workspace and then completing the following tasks:
- Move the arm to the object.
- Lower the height of the arm to the object.
- Close the Gripper on the object.
- Raise the Height of the arm.
- Click “Clear Arm” to remove the arm from the workspace.
- Open the Gripper to drop the object.
At the end of operations, use the “Clear Arm” button to remove the arm from the workspace, open the gripper and turn the servo off.
4. Automatic Tracking of Objects:
An automated system should be able to determine the position and orientation of objects by locating them within the visual field. Once all of the previous sections have been completed then an initial layer of automation can be implemented. First review section 3a for an overview of how the software and robot should be configured.
To start the automatic tracking of objects, click the “Track Fiducial” button in the main window. If everything is set correctly as per 3a, then a list of coordinates will appear in the status window, or if the fiducial is not detected, then the message “No object in view” will appear. If everything is working properly, the red cross(Figure 9) will appear in the middle of your fiducial. For safety reasons, the automatic pick and place process is not entirely automated. To instruct the robot to move to the detected point, click the “Move Arm” button. Note that the orientation of the arm will change to match that of the detected object. Follow the task sequence detailed in 3d to complete the process of removing the block from the scene.
Fig 9. When the software is successfully tracking the object, a red cross appears in the center of the object.
Note: There is an automatic mode that will completely automate this process of finding objects, moving to them, picking them up and then removing them from the scene, but it is not currently activated for safety reasons.
5. Additional Functions:
The “Set Speed” button will set the speed of the arm to match the value given by the speed slider. Warning, this should be set a values of 20 or below for safety during use. The robot will run at full speed in an automated environment, but this poses severe dangers for human interaction.
b. Melfabasic software.
The Melfabasic software is the software running in the controller of the robot arm. Melfa basic uses BASIC commands and structures with additional commands designed to control an industrial robot. Simple commands are available to command the robot without having to understand the deeper process of the controller.
On initialization the software sets the speed to 20% of the full speed for safety reasons. As the robot arm can be really dangerous if it hits someone at full speed the speed is set to 20 to avoid any accident and give users the opportunity to react to dangerous situations. The Melfabasic software acts in a continuous control loop waiting for serial commands from the user. The software waits for an integer between one and six from the computer. Each integer corresponds to a function of the robot.
1 -> corresponds to move command
2 -> corresponds to close gripper
3 -> corresponds to open gripper
4 -> corresponds to turn the servos on
5 -> corresponds to switch the servos off
6 -> corresponds to set the speed
Initialization Routine
The initialization routine sets the speed, initializes the serial port and sets up the control loop so that when an integer is received the software will run the corresponding subroutine. If the integer is not between one and six, or there is other unrecognized input, the software will wait for another integer until between one and six is received.
OVRD 20′,\
*LOOP
OPEN “COM1:” AS #1
print #1, “IN”
input #1,m1
close #1
on M1 GOTO *move,*closeg,*openg,*servoon,*servooff,*speed
goto *LOOP
Move subroutine
This is the subroutine for the move command. First the controller sends P to say that it is waiting for coordinates. Next it waits until it receives the destination point from the serial port. The point sent should be of the form (x, y, z, a, b, 0)(6, 0). x, y, z correspond to the Cartesian coordinate and a and b the orientation of the gripper.
The robot moves to the point received from the serial port and when it is on position it send OP on the serial port to say that the operation has been performed. Finally the program returns to the menu sub routine.
*move
open “COM1:” as #1
print #1, “P”
input #1,p1
MOV p1
dly 0.5
print #1, “OP”
close #1
goto *LOOP
Open and Close gripper Subroutines
These subroutines to open and close the gripper. To close the gripper, the controller issues a hclose command. Sufficient delay is built in so that the command has time to execute before the next command is received. If commands are sent from the PC to rapidly, ie. before the half second delay is up, then the controller can crash and will require a reset. The controller returns a “CL” on the serial port to confirm that the operation has been done. The open gripper routine is similar except that is uses hopen to open the gripper and “GO” as acknowledgement. At the end of each routine the program returns to the main control loop.
*closeg
hclose 1
dly 0.5
open “COM1:” as #1
print #1, “CL”
close #1
goto *LOOP
*openg
hopen 1
dly 0.5
open “COM1:” as #1
print #1, “GO”
close #1
goto *LOOP
Servo ON/OFF Subroutines
This part of the software turns on and off the servos. The servos must be turned on prior to moving the arm. When the servo on subroutine is activated by sending “servo on”, the servos turn on and the controller send SO on the serial port to confirm the operation. Likewise, the servos must be turned off before a human enters the robot workspace for safety reasons. The robot can move really fast and be dangerous thus it is forbidden to approach it when the servos are on. The message SF is sent to the serial port to say that the servos are off and the controller return to the menu after that. Note, trying to move the robot arm while the servos are off will result in the controller crashing. To reset the controller, just use the stop program, load program and start program button in order. Be sure that careful attention is paid to the servo status at all times.
*servoon
servo on
dly 0.5
OPEN “COM1:” AS #1
print #1, “SO”
close #1
goto *LOOP
*servooff
servo off
dly 0.5
open “COM1:” as #1
print #1, “SF”
close #1
goto *LOOP
Arm Speed Subroutines
The last subroutine is used to set the speed of the arm. Like the arm positioning, setting the arm speed is a two part process. First the speed subroutinue is initialized on receiving a “6″ on the serial port. The subroutinue replies with “SPEED” to notify that it is waiting for the integer between 0 and 100 to set the speed. The integer is the percentage of the maximum speed of the robot. It is set at 20 by default. When an integer is received it sets the speed and sends SPEEDSET to the serial port to confirm that the speed has been set. After that the software returns to the menu.
*speed
OPEN “COM1:” AS #1
print #1, “SPEED”
input #1,m2
OVRD m2
print #1, “SPEEDSET”
close #1
goto *LOOP
end
Appendix 4 : Trouble Shooting
If the robot controller starts to make loud beeping noises, note the error number on the controller screen and immediately hit the reset button on the controller. If it persists, then turn the controller off and check what the error means in the Mitsubishi robot instruction manual “Troubleshooting” to see where the error comes from.
Common sources of controller crashes:
- Clicking the buttons on the GUI rapidly.
- Sending the arm outside of its designated range.
- Attempting to move the arm when the servos are off.
- Trying to change the height of the arm when it is in the cleared position.
Avoid each of the above behaviors. If the controller does crash, it can usually be quickly recovered by pressing the “Reset” button on the controller. When the controller is reset then the software must be reloaded into the controller using the Stop Program / Load Program / Restart Program steps detailed in 3d. The controller will restart in the menu and operations can resume from where they left off.
Appendix 5: Safety(reproduced from Mech 515 Course documents)
Robot Operator’s
Safety Rules
The following safety rules must ALWAYS be obeyed. Failure to do so may result
in exclusion from the laboratory.
1. NEVER enter the robot work space when servo power is on!
i.e., the green LED (light emitting diode) at the “SVO ON” switch is on.
2. Always make sure there is no one in the work cell before starting a program.
3. Always give an audible warning before running the program.
4 . Always keep one hand at the emergency stop and observe the robot during
its full operational cycle
5. Never leave the robot unattended when it is powered up.
Additional rules for the Mitsubishi RV-2AJ robot arm
6. Always write “OVRD 20″ as the first line in any robot program. This limits the
maximum speed
7. Never load the robot with more than 2 kg. If the arm is operated at full stretch
limit the maximum load to 0.5 kg.
[1] T. Sornakumar and C. Paramasivam,
Computer vision based positional error compensation of an industrial robot using linear regression analysis International Journal of Manufacturing Research, Vol 3, #2 / 2008
[2] Mitsubishi Industrial Robot CR1 Controller Instruction Manual “Detailed explanation of functions and operations.” Section 4-81
