Tools used
Mac 10.6 Snow Leopard with built-in Apache server
Coldfusion 9 developer edition
MXUnit 2.1.1
CFSelenium 1.7
ImageMagick 6.7.4-6
Firefox 3.6.25
Set up
So first you need access to a ColdFusion server. Then install MXUnit. As of this posting, the latest stable version is 2.1.1. I followed the docs and installed it to my webroot so I wouldn't have to configure it. Next, I added a mapping to Coldfusion Administrator named mxunit that points to the mxunit folder.
CFSelenium was just as easy to install. I downloaded the latest version which at the time of this posting is 1.7. And extracted it to my webroot in a folder named cfselenium. Then I added another mapping named cfselenium.
For ImageMagick I followed the instructions and installed it using MacPorts. It took about 10 minutes to finish installing as there were a lot of dependencies.
The Test
My sample uses a simple registration form, register.htm, with four fields for first, last, phone, and email.
My sample site has the following file structure:
register
tests
testsuite
RegisterTest.cfc
runTests.cfm
register.htm
The component I use to test this form is RegisterTest.cfc. It extends cfselenium.CFSeleniumTestCase which extends mxunit.framework.TestCase. The mappings I created earlier allow these prefixes to resolve correctly. It contains the test functions that run selenium commands and mxunit functions. You will need to replace absolutePathToWebRoot with yours.
In the formPageLooksRight function are the lines that take and compare the screenshots.
This cfexecute tag executes the ImageMagick compare command which compares baseline.png and screenshot.png and creates an image with the differences as compare.png.
<cfexecute name="/opt/local/bin/compare" timeout="60" arguments="""/absolutePathToWebRoot/register/tests/baseline.png"" ""/absolutePathToWebRoot/register/tests/screenshot.png"" ""/absolutePathToWebRoot/register/tests/compare.png""" />
The next cfexecute tag again runs the compare command, but this time it outputs the number of pixels that differ between the two images. For some reason the cfexecute tag thinks the output from this command is error output, so instead of storing it in the variable attribute it puts it in the errorvariable attribute.
<cfexecute name="/opt/local/bin/compare" timeout="60" arguments="-metric AE ""/absolutePathToWebRoot/register/tests/baseline.png"" ""/absolutePathToWebRoot/register/tests/screenshot.png"" null: 2>&1" errorvariable="diffVal" />
You can read about comparing images using ImageMagick here.
To run the tests navigate to runTests.cfm?debug=true in the browser. The debug variable allows the debug messages to show in the test output. This page runs the test functions inside any cfc in the testsuite folder that either starts or ends its name with "Test". Our example only has one, RegisterTest.cfc.
When you run the tests CFSelenium starts the selenium RC server automatically for you. Sometimes you might get a timeout error when you try to run your tests. Killing any running instances of Selenium fixed this for me. ps -A | grep "selenium". Gives the process id (pid). kill pid. To kill it.
The first time you run the tests the formPageLooksRight test fails because it is looking for a file "baseline.png" in the tests folder that does not exist. It does create screenshot.png of what the page looks like. So this will be used as the baseline so rename it to "baseline.png".
If you don't set the background color for the page the screenshot shows it as black. Specifying the background-color of the body tag fixed this.
<body style="background-color: #ffffff">
Run the tests again by refreshing the page. Now the test should pass. The links in the test output show the baseline (exp), the screenshot taken by the test (act), and any differences between the two (dif).
Now make a change to the UI in register.htm. I removed the "N" from the "First Name" label.
This time after you rerun the tests you will see that formPageLooksRight has failed with a message that shows how many pixels differ between expected and actual screenshots. Clicking on the dif link highlights the pixels that differ between the two images.
Expected:
Actual:
Difference:
