N umerous people have told me that we columnists make devel-opment look a little too easy, especially since everything we do works. Readers figure that we whip up a column in half a day and spend the rest of the month lounging on the beach having ice-cold drinks delivered right to our outstretched hands. If only it were so easy. Normally, I have to get up and schlep over to get my own drink at the poolside bar!
This month, you'll see something different in Bugslayer. A ton of work went into this column, and still it's not perfect. But don't worry. I didn't leave you with a half-implemented utility; it works as advertised. However, as you'll see, the implementation has so many tradeoffs, that it's a 99.93 percent solution that I ultimately wanted to achieve. Everyone learns from their mistakes, and I learned a lot here; hopefully you will as well. What's interesting is that some of the mistakes I made were completely self-inflicted.
As you'll see, the vaunted Tester utility is getting another upgrade. After many e-mails begging me to add mouse support to the recorder and the playback utilities in Tester, I finally did so. Now you have a complete user interface testing tool that will help you automate everything from thick client apps to Web front ends running in Microsoft® Internet Explorer to Windows® Forms-based Microsoft .NET applications. To get started, I'll begin with a little Tester primer and move into some of the usage hints. Finally, I'll discuss what went right and wrong with the implementation.
Tester Primer
The idea for Tester started in my April 1999 column in Microsoft Systems Journal when I wanted a way to test user interfaces to make unit testing easier. While I could have created a rudimentary record and playback utility like the old Windows 3.0 Recorder, that application was insufficient for real world usage. To do effective automation testing, you need to do conditional testing and branching. Instead of inventing my own programming language, I realized that the best thing to do was to write a COM object that supports various window-controlling operations so you could write the code yourself in VBScript or JavaScript. All of my code was wrapped up in TESTER.DLL and two support DLLs, TINPUTHLP.DLL and TNOTIFYHLP.DLL. The first big feature of Tester One was the fact that you could register notification handlers to be told when a specific window with a specific caption was created. The idea was to give you a means to handle ASSERT dialogs popping up without having to attempt to handle them in a normal script. The second big feature was the PlayKeys method of the TInput object, which always properly passed specific keystrokes to the window that had focus, unlike the other SendKeys methods.
After many e-mails telling me that Tester was pretty cool, but lacked one particular feature, I created Tester Two in my June 2000 column in MSDN Magazine. The feature in question was a separate program called TestRec, which recorded all your keystrokes and built a VBScript of your actions. You could immediately run the resulting script and achieve testing nirvana. Lots of people told me they were quite excited by Tester Two and were using it for their unit testing. But the one important thing that Tester and TestRec did not do was handle mouse operations. That was a conscious decision on my part; I wanted to spend extra time on it to see if I could devise a way to perform the mouse recording and playback in a screen-independent manner. That way, a teammate could record a Tester script and others on the team could use it regardless of screen setup. Before I turn to the trials and tribulations of the Tester implementation, I want to give you some tips for using Tester that will save you time and help you get the best scripts out of it.
Tester Tips
The code distribution for this month's column includes compiled versions of Tester, so if you don't have both Visual Basic 6.0 and Visual C++ 6.0 installed, you aren't stuck. Please note that the playback portions of Tester, TESTER.DLL, TINPUTHLP.DLL, and TNOTIFYHLP.DLL are in the Output directory. The recording portion, TESTREC.EXE, has two different builds, the ANSI version in the Output directory and the UNICODE version in OutputUNICODE. The reason for the two builds is that I found a bug when running on Windows XP. Using the ANSI build of TestRec on Windows XP, I was not able to save any of the generated scripts, nor could I copy out of TestRec.
TestRec is written using MFC and I tracked the problem down to the call in the CEditView to get the edit control handle by sending EM_GETHANDLE to the edit control. Even though I was using the ANSI build, the handle passed back was of the UNICODE buffer so the MFC code was always truncating the text because it was treating the buffer as ANSI. My guess is that this is a Windows XP bug because the window is created with CreateWindowExA, which the EM_GETHANDLE message says should cause the edit control to return a buffer with ANSI characters. Consequently, I simply recompiled TestRec as UNICODE and all was fine because it worked around the bug and gave me a faster application. The only difference is that now files are saved as UNICODE characters instead of ANSI, but the Windows Script Host couldn't care less.
Please keep in mind that I only tested the code on Windows 2000 SP2 and the release version of Windows XP. I may have used some flags to GetSystemMetrics that don't appear in Windows NT® 4.0, but don't think I did anything else that would prevent it from running on Windows NT 4.0, Windows 9x or Windows Me, but I make no guarantees of backward compatibility with older systems. If you want to build the code, I used the header files and libraries from the August 2001 Platform SDK so I could have the latest VK_xxx codes and other declarations. I did not attempt to compile the code with any of the default Visual C++® installations, and I doubt they would work without having the Platform SDK installed. Additionally, I defined _WIN32_WINNT and WINVER to 0x501 to ensure the latest definitions were used.
The first step you must take before running Tester is to copy the three DLLs that make up the recording portion of Tester to a directory in your path. The problem is that the supporting DLLs, TINPUTHLP.DLL and TNOTIFYHP.DLL, aren't found by the Visual Basic runtime even though they are in the same directory as TESTER.DLL. After copying the DLLs, you'll need to register TESTER.DLL with REGSVR32.EXE so the generated scripts can use the COM objects.
Before you start recording a million scripts, you'll need to do a little planning to take advantage of Tester. While Tester now handles mouse recording and playback, your scripts will still be much more robust if you can do as much work as possible with keystrokes. One nice feature is that when you're recording, Tester works hard to keep track of the window that has the focus. By default, on mouse clicks and double-clicks, Tester will generate code to set the focus to the top-level window before doing the click. Also, when recoding with keystrokes, Tester monitors Alt+Tab combinations to set the focus when you finish shifting focus.
Since mouse recording can generate a million statements in a script, there's a new option dialog in the TestRec application that is accessible from the Script menu and shown in Figure 1. The first thing you'll notice at the top is that TestRec still generates VBScript, but for those of you who love typing a semicolon at the end of each line, you can now have your scripts generated in JavaScript. When recording a script using the mouse, you'll need to determine the type of recording you need. If it's a simple script based on clicking a few buttons, the defaults are just fine (see Figure 1). For scripts in which you will be doing a lot of clicking and dragging and you want to record all the mouse movements between the mouse down and release, set the value of "Minimum pixels to drag before generating a MOVETO" to zero. If you will be recording a lot of clicks in the application without shifting focus to other applications, you'll want to uncheck "Record focus changes with mouse clicks and double clicks." That will keep TestRec from generating the code to force the focus each time the mouse goes down, and it will make your script much smaller.
.gif)
Figure 1Tester Settings
The rest of the options deserve a little mention too. The "Seconds to wait before inserting SLEEP statements" option allows you to automatically insert pauses in the script. Most of the time you'll want to let your scripts run as quickly as possible, but to help keep scripts coordinated, the extra pause time can help. The "Record all mouse movements" option does exactly what it sounds like it does. Each mouse movement is captured and output in the script to completely duplicate the actions you take. As you can imagine, this will lead to huge scripts.
The "Do child focus attempt in scripts" option will add script code to attempt to set the focus to a specific control that you click on. I left this off by default because I was already generating the statements to set the focus to the top-level window. Turning this option on will add code to attempt to set focus to the child window you are clicking on. While applications like Notepad only have a single child window, plenty of other widely used applications have deeply nested window hierarchies with many windows. Therefore, it can be quite difficult to track down child windows without all of the parents having titles and unique classes. For an example, I used Spy++ to look at the Visual C++ editor window hierarchy. What I found was that setting the top-level window focus before generating the click code was almost always perfectly fine.
The final options, "Use absolute screen coordinates" and "Record for multiple monitor playback," are tied together. Part of the bad news with Tester is that I was unable to get true virtual recording and playback to work perfectly. That's why Tester is only a 99.93 percent solution. Since I was restricted to doing absolute coordinate recording, the "Record for multiple monitor playback" option will generate code to check the complete virtual screen dimensions as well as the virtual screen origin to ensure the playback works correctly. If you uncheck "Record for multiple monitor playback," the recording will only take place on the primary monitor to make the script a little less system dependent. I still generate the code to check if the screen resolution for the primary monitor is the same as the resolution it was recorded on.
Now that you've seen the options, take a look at Figure 2 which is an example of a script recorded by Tester. Lines 1 through 5 create the two key Tester objects, TIntput and TSystem, to allow you to play input and to get system information, respectively. Lines 7 through 10 declare two TWindow variables that the rest of the generated script will access to set the various window focus. Lines 12 through 16 are generated based on the options you set in the script recording dialog.
I fixed TestRec so that when you change the options with a script open, it automatically adds lines similar to these to ensure the playback portion knows what to expect in upcoming statements. The first TInput property set on line 12 indicates that all coordinates from this point forward are absolute screen coordinates. If you set this to false, you will be warned at playback time that the virtual coordinate code does not quite work and you should not use it. The TInput.MultiMonitor property tells the playback that screen coordinates are based on the values using the virtual screen. If you set this property to false, the coordinates are based on just the primary display.
After setting the multiple monitor flag, the TSystem.CheckVirtualResolution call verifies that the screen resolution and the origin point of the multiple monitor system matches the system that recorded the script. Lines 18 through 28 are the recorded portion of the script. Lines 18 through 20 are used to find the window, bring it to the foreground, and set the size to the same as it was when recorded. The rest of the lines are the keystrokes and mouse operations directed at that window. If you are familiar with previous versions of Tester, you'll see that the old TInput.PlayKeys method has changed to TInput.PlayInput as the new name reflects what the function really does. PlayKeys is still there for backward compatibility.
The big changes to Tester were mostly in the internals, so your existing tester scripts should run just fine. The most visible changes are those that I mentioned already, which do screen resolution checking and setting the size of windows to ensure mouse playback is correct. In order to handle the new mouse operations, I added several new curly brace constructs to the PlayInput format. They are listed here with their parameters.
btn: LEFT, RIGHT, MIDDLE
x: X screen coordinate value
y: Y screen coordinate value
{MOVETO x , y}
{BTNDOWN btn , x , y}
{BTNUP btn , x , y}
{CLICK btn , x , y}
{DBLCLICK btn , x , y}
There were a few items I was not able to add to the mouse recording. The first is mouse wheel processing. I'm using a journal hook to capture keystrokes and mouse operations, and the mouse wheel message comes through. Unfortunately, a bug in the journal hook reporting does not pass the mouse wheel direction so there's no way to know if you are scrolling up or down. The second item I could not process was the new X1 and X2 buttons found on the newer Microsoft IntelliMouse®. The problem was a lack of hardware on my part. However, I'm not sure it would work as the WM_XBUTTONxxx messages pass which button was pressed in the high order word of the wParam. Since the WM_MOUSEWHEEL message passes the wheel click direction in the same way, but the journal record hook does not receive it, I doubt the X button would come through either.
Before I turn to the implementation fun, I'll want to leave you with some final script playback hints. The first is to liberally sprinkle TSystem.Sleep statements throughout your scripts. I added Sleep to handle multiple-second pauses. The TSystem.Pause statement is still supported for millisecond pauses. The reason for the Sleep statements is that you need to take into account system speed when you're waiting. I have two dual-processor machines and I found scripts that would work great on my dual 733MHz would sometimes fail on my much faster dual 1.7GHz because things such as window creation and destruction happened much faster. Putting in longer Sleep statements generally cleared up the problems. Finally, since I wrote the original version of Tester, Windows Script Host has improved dramatically. You might want to take some time to explore the ins and outs of the new scripting support to help build stronger and more robust test suites.
Tester Three Implementation Issues
When I started working to add the mouse support, I didn't think it would be difficult. When I did Tester Two, I tried to leave room in the design so that I could drop the mouse support right in. In the end, I found out that I didn't even come close to the goal. Between the mouse support and trying to find a solution to the virtual mouse issues, I spent more time on this code than almost any other column I have written. Like I said, I sure suffered for my art on this one!
When working on previous versions of Tester, I tried to always keep the future in mind and tried not to prevent myself from adding mouse support. While I did pretty well on the playback side, I found I completely missed it on the recording side. I first ran into problems when thinking about all the mouse issues I would have to handle. When doing the keystroke processing, I was extremely careful about handling the CTRL, ALT, and SHIFT keys so I could ensure that if I saw one of these keys go down, I would wait for the control key to go up before I generated the resulting script code. That way I would never get into the situation in which the script would end with one of those keys in a down state and mess up the user's system.
When thinking about mouse support, I realized that if, for example, the CTRL key was down, I could generate a script line of thousands of characters if recording all mouse movements waiting for that CTRL up! Keep in mind my PlayInput method uses the SendKeys format so a series of keys pressed with CTRL down is expressed as "^({END}{LEFT})" where the ^ character indicates the CTRL key. With the mouse support, I would need to break up statements so that I could indicate a CTRL down in one statement, any mouse movements in the following statements, and finally the CTRL up, like the following:
tInput.PlayInput ( "{CTRL DOWN}" ) ;
tInput.PlayInput ( "{BTNDOWN LEFT , 10 , 10}" ) ;
tInput.PlayInput { "{MOVETO 60 , 60}" ) ;
tInput.PlayInput ( "{BTNUP LEFT , 120 , 120}" ) ;
tInput.PlayInput ( "{CTRL UP}" ) ;
After analyzing the existing code, I realized I had to completely rewrite the whole recording engine, which is pretty embarrassing to admit. I simply didn't do a good enough job to take this situation into account. After completely rewriting the recording engine in TestRec's RECORDINGENGINE.H/.CPP, I was pleased to see how much simpler the code became. Instead of generating the special keys for CTRL, ALT, and SHIFT, I now generate special codes such as "{ALT DOWN}" and "{ALT UP}." The playback code still handles the SendKeys format for the CTRL, ATL, and SHIFT keys. One thing I would like to add is string processing because part of the recording is to analyze the string I'm about to generate and turn a string like "{SHIFT DOWN}{END}{LEFT}{SHIFT UP}" into "+({END}{LEFT})."
After taking care of the CTRL, ATL, and SHIFT processing, I next had to figure out how I was going to handle mouse clicks, drags, and double-clicks. What makes the processing fun is that the journal hook I'm using to record keys and mouse moves only reports WM_XBUTTONDOWN and WM_XBUTTONUP messages. I would have much preferred to get WM_XBUTTONDBLCLK messages to make my life easier. This was screaming to be a state machine. After whipping out Visio, I devised a complete state model for tracking everything. Figure 3 shows the mouse state machine that I implemented in RECORDINENGINE.H/.CPP. Keep in mind that I also had to do this state tracking for each button on the system. The mentions of Slot 0 and Slot 1 were to keep track of the previous event for comparison purposes.
After grinding through the code to implement the recording, everything looked good until I started hard testing. Immediately, I saw problems. Recording scripts as I was drawing items in Paint Brush worked just fine. Playing them back was a problem. For example, I'd draw a circle freehand but the playback would look like a straight line through the origin of the circle, then the rest of the circle would be drawn. I went back and thoroughly examined my recording and playback code, but found nothing wrong. As it turns out, with the script pumping a bunch of MOVETO instructions very quickly, the Windows OS input queue dumps extra input when it's getting full. What I needed to do was slow down the mouse message processing so all the mouse events would have enough time to execute. In the TINPUTHLP.DLL I use the SendInput function to do the actual playback. My first idea was to set the time on each event in the INPUT structure to allow extra time for the mouse events. That didn't work, and I found that setting the timing long enough would cause the computer to kick into power-save mode.
Looking at another approach, I thought that since my code parses the input commands into an array of INPUT structures to pass to SendInput, I might be able to spin through the array one at a time and do extra pauses on the mouse events. The question of how long to wait became an experiment. After playing around, I found it best to sleep for 25 milliseconds before and after each mouse event. In general, I think more experimentation is needed to find the perfect sleep time duration.
Finally, I was able to turn my attention to the virtual coordinate system, as I would love to have Tester-recorded scripts work at any screen resolution. My idea was to use a coordinate system of my choosing and map the absolute coordinates to it. Since the SendInput MOUSEINPUT structure already had a flag, MOUSEEVENTF_ABSOLUTE, which used a normalized coordinate system of (0,0) to (65536,65536), I thought I could use that as my system as well. That way I was always recording on a relative scale, so if you clicked 25 percent off the origin it would generally translate at other resolutions. I coded the recorder to do the translations and ran some tests.
In testing with a few programs like Notepad and Paintbrush, I got reasonable results and thought I was getting close to a solution. When recording the mouse coordinates, I was also recording the window positions in virtual coordinates. That way I was getting windows resized appropriately for any clicks. Of course my bubble burst when I started Calc to check some calculations I was making. Since Calc is dialog-based, what was going to happen if I clicked on the "1" button in a virtual system? Where I was resizing the window for the virtual resolution, I was not resizing the individual child windows. Consequently, unless I was going to record the complete window hierarchy for an application and resize each of those windows, I was never going to get virtual coordinates to work just by doing mouse recording.
I could add a Click method to my TWindow class. Then, when a child window is clicked, I could move the mouse to the middle of that child window and perform the click operation. While clicks and double-clicks would be easy, there would be no way to properly do mouse movement operations. Additionally, that would not account for controls that need spatial clicking to work properly, such as a toolbar. Despite many hours of searching for a solution, I never came up with anything that would completely work in all cases. I guess I should have expected this because none of the commercial regression testing tool recorders that I am aware of do virtual recording and playback. While it's disappointing, I have to admit failure. I left in all the code where I attempted to do the virtual coordinate system so you could see what was going on. In the TestRec recording options, I disabled the ability to record in virtual coordinates. However, you can enable it by enabling the first checkbox in the options dialog. Maybe someone else can see a better solution and get it working.
Tester Futures
Given all the fun I had with Tester Three, I don't know if there will be a Tester Four, but I'm willing to try if there is enough interest. If you want to extend Tester, here are a few things to consider:
- Add classes in TESTER.DLL for the common controls such as checkboxes and tree controls. That way you can easily see state information in your scripts. It would be challenging to add support for dynamically created windows.
- Think about updating the TestRec application to be a little friendlier and offer more help while developing scripts. For example, you could add knowledge of the supported Tester objects to help other developers write their own scripts. See the Tips section for another idea.
- Add script debugging to TestRec so you can debug your regression scripts. This could get quite interesting as you would have to work hand-in-hand with the running Tester object to coordinate window focus and suspend playback.
- Add support during recording for on-the-fly entering of validation and verification code and other necessary code on-the-fly.
Wrap-up
While it only took me three columns (the third was the toughest of all), I finally got Tester to handle all the common issues. Now armed with the ability to easily record all keyboard and mouse operations on user interface elements, you should no longer look at unit testing as a chore, but as something fun. OK, maybe you don't find it fun, but anything that makes it easier is has to be a good thing.
Tips
Tip 51 One thing that's a little weak in TestRec is that you are stuck looking at a simple edit control to do some of the on-the-fly editing of your scripts. If you're up to the challenge, Neil Hodgson has written an outstanding open source editing control called Scintilla (https://www.scintilla.org), which you could integrate into TestRec. It already supports JavaScript and VBScript syntax highlighting as well as code folding and a myriad of other excellent editing features. Also, if you're looking for a great Windows programming sample to learn from, check it out.
Tip 52 If you find yourself wanting to know how a particular DLL landed on your system (or more importantly on a customer's system), here's a link that points to a Microsoft database of every DLL they've shipped, along with the version number and link date. The final screen for each item tells what product shipped the DLL. This can be invaluable when you need to track down a "Why doesn't it work on my machine?" bug report. See the database at https://support.microsoft.com/servicedesks/fileversion/dllinfo.asp?fr=0&sd=msdn.
Send questions and comments for John to slayer@microsoft.com. |
.gif)
.gif){kind=small}