Test Run
Lightweight UI Test Automation with .NET
James McCaffrey
Code download available at:TestRun0501.exe(131 KB)
Contents
The Application Under Test
The Test Automation Script
Manipulating the Application Under Test
Checking the Application State
Discussion
Manual user interface testing is one of the most fundamental types of software testing and it's the kind of testing that most software engineers first experience. Paradoxically, automated user interface tests are probably the most technically challenging kind of test to write. The Microsoft® .NET environment provides you with many ways to write automated user interface test automation. One common and useful approach is recording keystrokes, mouse movements, and clicks and then playing them back to the application to ensure that it performs as expected. (For more information on this approach, see John Robbins's Bugslayer column in the March 2002 issue of MSDN®Magazine. Paul DiLascia's column in this issue of MSDN Magazine also demonstrates how to use .NET to send this sort of input to another application.) In this month's column, I'll explore another approach to writing lightweight UI test automation for .NET applications.
The best place to start the discussion is with a screen shot. Figure 1 shows that I have a dummy application to test. It is a color mixer app which allows a user to type a color into a textbox control, then type or select a color in a combobox, click the button, and have the listbox display a message that represents the result of "mixing" the two colors. In Figure 1, red and blue produces purple according to the application. The UI test automation is a console application that launches the form under test, simulates a user moving the application form, sizes and resizes the application form, sets values for the textbox and combobox controls, and clicks the button control. The test automation checks the resulting state of the application under test, verifies that the listbox control contains the correct message, and logs a "pass" result. I captured the screen shot in Figure 1 just before the test automation simulates a user clicking on File | Exit, which closes the application under test.
Figure 1** Form UI Test Automation **
In the sections that follow I will briefly describe the dummy application I'm testing, explain how to launch the application's form from the test automation program using reflection and the System.Windows.Forms.Application class, show you how to simulate user actions and check application state using methods in the System.Reflection namespace, and describe how you can extend and modify the test system to meet your own needs. I think you'll find the ability to write lightweight UI test automation quickly to be a useful addition to your skill set no matter what role you play in the software production environment. Additionally, these same techniques can be incorporated into your own unit testing harness and are relevant even if you're using an existing framework like NUnit.
The Application Under Test
Let's look at the application under test so you'll understand the goal of the test automation. The color mixer application under test is a simple Windows® form. I used C# to code the application, but the UI automation techniques I'll show you apply to applications written using any .NET-targeted language. I accepted the Visual Studio® .NET default control names of Form1, textBox1, comboBox1, button1, and listBox1. Of course, in a real application you'd change the names of the controls to reflect their functionality. I added three dummy menu items: File, Edit, and View. The heart of the app under test is the code listed in Figure 2.
Figure 2 Color Mixer Application Code
private void button1_Click(object sender, System.EventArgs e) { string tb = textBox1.Text; string cb = comboBox1.SelectedItem.ToString(); if (tb == cb ) listBox1.Items.Add("Result is " + textBox1.Text); else if (tb == "red" && cb == "blue" || tb == "blue" && cb == "red") listBox1.Items.Add("Result is " + "purple"); else listBox1.Items.Add("Result is " + "black"); }
When a user clicks on the button1 control, the application grabs the values in the textBox1 and comboBox1 controls. If the two color strings match, a message with that color is displayed. If the textbox and combobox controls contain "red" and "blue", a result message with "purple" is displayed. If any other color combinations are in the textbox and combobox controls, a result message with "black" is displayed. Because this is just a dummy app for demonstration purposes and I wanted to keep the code as short as possible, I didn't check my input arguments as I would have for a realistic app. Although this app is extremely simplistic, it has most of the fundamental characteristics of Windows-based apps needed to demonstrate automated UI testing.
Manually testing even this minimal application through its user interface would be tedious, error prone, time consuming, and inefficient. You'd have to enter some inputs, click the button control, visually verify the result message, and record the result into an Excel spreadsheet or other data store. Because the app accepts free-form user input into the textbox control, you'd essentially have an infinite volume of possible test input, so you'd have to test hundreds or even thousands of inputs to be satisfied that you understood the behavior of the application. On top of all this, every time there was a change in the application code, you'd have to perform the same manual tests all over again. Writing unit tests is a much better approach as these tests allow you to simulate a user exercising the app and then determine if the app has responded correctly.
The Test Automation Script
The overall structure of the test automation harness is shown in Figure 3 and an outline of the code is shown in Figure 4. I used C# here, but you could easily modify the code to be any .NET-based language.
Figure 4 UI Test Automation Harness Code
using System; using System.Windows.Forms; using System.Reflection; using System.Threading; namespace RunScenario { class Program { static Form testForm = null; static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static int delay = 1500; static void Main(string[] args) { try { Console.WriteLine("\nStarting UI test scenario"); // launch the application // move and resize the form // set textBox1 and comboBox1 // click button1 // check message in listBox1, display pass or fail result // click File->Exit } catch(Exception ex) { Console.WriteLine("Error: " + ex.Message); } } static void SetControlPropertyValue(string controlName, string propertyName, object newValue) { // code here to set a control property } static void SetFormPropertyValue(string propertyName, object newValue) { // code here to set a form property } static object GetControlPropertyValue(string controlName, string propertyName) { // code here to access a control property } static void InvokeMethod(string methodName, params object[] parms) { // code here to click a button } static Form LaunchApp(string exePath, string formName) { // code here to launch the application } } }
Figure 3** UI Test Automation Structure **
I begin by adding and declaring references to the System.Windows.Forms, System.Reflection, and System.Threading namespaces. Because the System.Windows.Forms.dll is not referenced in a console application by default, to use the classes in this namespace you need to add a project reference to the System.Windows.Forms.dll file. The System.Windows.Forms namespace contains the Forms class as well as the Application class, both of which I use in this solution. I use classes in System.Reflection to get and set the values of the form's controls and to invoke methods associated with the form. I use methods in System.Threading to launch the form from the console application test harness.
I declare three class-scope objects because they're used by several methods in the test harness:
static Form testForm = null; static BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance; static int delay = 1500;
Because the color mixer application is just one Windows Form, I declare a Form object to represent it. The BindingFlags object is used by many methods in the System.Reflection namespace as a filter. I set an int delay variable to 1500 (milliseconds) to feed to the Thread.Sleep method so I can pause 1.5 seconds at various points in the test automation. To launch the app under test I use this code:
Console.WriteLine("\nLaunching WinApp under test"); string exePath = "C:\\FormUIAutomation\\WinApp\\bin\\Debug\\WinApp.exe"; testForm = LaunchApp(exePath, "WinApp.Form1");
I've defined a LaunchApp method and its helper method RunApp in Figure 5. There aren't very many lines of code but they are very powerful. Notice that for simplicity I hardcoded the path to the executable of the application under test (in your own tests, you'd want to parameterize this information to make the test automation more flexible). The LaunchApp method accepts the path to the application executable and the name of the application form, and returns an object representing the form. LaunchApp creates an instance of an Assembly object using the Assembly.LoadFrom static method rather than by using an explicit constructor call.
Figure 5 Launching the App
static Form LaunchApp(string exePath, string formName) { Thread.Sleep(delay); Assembly a = Assembly.LoadFrom(exePath); Type formType = a.GetType(formName); Form resultForm = (Form)a.CreateInstance(formType.FullName); Thread t = new Thread(new ThreadStart(new AppState(resultForm).RunApp)); t.ApartmentState = ApartmentState.STA; t.IsBackground = true; t.Start(); return resultForm; } private class AppState { public AppState(Form f) { FormToRun = f; } public readonly Form FormToRun; public void RunApp() { Application.Run(FormToRun); } }
Next, the Assembly.GetType method returns a type that represents the application form, and then I use the Assembly.CreateInstance method to create a reference to the form under test. Then I spin up a new Thread to actually launch the application form. The Application.Run method starts a message loop on the current thread; since I want to be able to perform work while the Form is visible, I need to run Application.Run on its own thread so that its loop doesn't block my progress. By using this technique, the test automation console application harness and the form are running on different threads but in the same process. This way they can communicate with each other—in other words, the test harness can send instructions to the Windows Form.
Manipulating the Application Under Test
After I've launched the application under test, I simulate a user manipulating the application form. The example test scenario begins by moving and resizing the form like so:
Console.WriteLine("Moving form"); SetFormPropertyValue("Location", new System.Drawing.Point(200,200)); SetFormPropertyValue("Location", new System.Drawing.Point(500,500)); Console.WriteLine("Resizing form"); SetFormPropertyValue("Size", new System.Drawing.Size(600,600)); SetFormPropertyValue("Size", new System.Drawing.Size(300,320));
All the work is done by method SetFormPropertyValue (see Figure 6). I use the Object.GetType method to create a Type object that represents the application form, and then use that object to get a PropertyInfo object that references a property on the form such as the Location property or the Size property. Once I have the property information object, I can manipulate it using the PropertyInfo.SetProperty method. SetProperty accepts three parameters. The first two are what you might expect—a reference to the object containing the property that will be changed and a reference to the new value of the property.
Figure 6 SetFormPropertyValue
static void SetFormPropertyValue(string propertyName, object newValue) { if (testForm.InvokeRequired) { Thread.Sleep(delay); testForm.Invoke( new SetFormPropertyValueHandler(SetFormPropertyValue), new object[]{propertyName, newValue}); return; } Type t = testForm.GetType(); PropertyInfo pi = t.GetProperty(propertyName); pi.SetValue(testForm, newValue, new object[0]); } delegate void SetFormPropertyValueHandler( string propertyName, object newValue);
The third parameter is necessary because some properties, such as the Items property of the listbox control, are indexed. Moving and resizing the form as I've done here isn't really relevant for testing the application's functionality but I do it to show you how it's done in case your test scenario requires it. Note, too, that I'm taking advantage of the ISynchronizeInvoke interface exposed by the Form class (actually, by its base Control class). You should only access properties of a control (that includes a Form) from the thread that owns the control's underlying window handle. For the form under test, that thread is the one I spun up on which to run Application.Run. Since my test harness is running on a separate thread, I need to marshal my access to properties and methods on the controls to that thread, and the control's Invoke method and InvokeRequired property make that a piece of cake (both this method and property are part of the ISynchronizeInvoke interface). For more information on ISynchronizeInvoke, see Ian Griffith's article in the February 2003 issue of MSDN Magazine, at Give Your .NET-based Application a Fast and Responsive UI with Multiple Threads.
Now I'm ready to simulate a user typing a color into the textbox control and selecting a color from the combobox control:
Console.WriteLine("\nSetting textBox1 to 'yellow'"); SetControlPropertyValue("textBox1", "Text", "yellow"); Console.WriteLine("Setting textBox1 to 'red'"); SetControlPropertyValue("textBox1", "Text", "red"); Console.WriteLine("Selecting comboBox1 to 'green'"); SetControlPropertyValue("comboBox1", "SelectedItem", "green"); Console.WriteLine("Selecting comboBox1 to 'blue'"); SetControlPropertyValue("comboBox1", "SelectedItem", "blue");
I set textBox1 to "yellow" and then to "red", and then set comboBox1 to "green" and then "blue". All the real work is done by a SetControlPropertyValue method shown in Figure 7.
Figure 7 SetControlPropertyValue
static void SetControlPropertyValue( string controlName, string propertyName, object newValue) { if (testForm.InvokeRequired) { Thread.Sleep(delay); testForm.Invoke( new SetControlPropertyValueHandler(SetControlPropertyValue), new object[]{controlName, propertyName, newValue}); return; } Type t1 = testForm.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(testForm); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName, flags); pi.SetValue(ctrl, newValue, new object[0]); } delegate void SetControlPropertyValueHandler( string controlName, string propertyName, object newValue);
I use the Thread.Sleep method to pause the test automation to make sure that the application under test is up and running. After creating a Type object that represents the application form's type, I retrieve information about a specified field (a control) on the Form object by using the Type.GetField method. Next I call the FieldInfo.GetType method to get a Type object that represents the control I want to manipulate. Once I have the control object I can manipulate it exactly like I manipulated the Form object by getting the control's PropertyInfo and then calling the SetValue method. As with SetFormPropertyValue, I need to ensure that all of the property changes I'm making are taking place on the correct thread.
Notice that the test automation does not directly simulate user actions at a very low level. For example, instead of simulating individual keystrokes into the textBox1 control the automation sets the Text property directly. And instead of simulating clicks on the comboBox1 control, the automation sets the SelectedItem property. This is a design limitation of my test automation system. To test in that way, you can follow the advice of John Robbins in the article mentioned earlier.
After simulating the actions of a user entering colors for the textbox and combobox controls, the automation simulates clicking of the button control:
Console.WriteLine("Clicking button1"); InvokeMethod("button1_Click", new object[]{null, EventArgs.Empty} );
I've defined the InvokeMethod method as shown in Figure 8.
Figure 8 Invoke Method
static void InvokeMethod(string methodName, params object[] parms) { if (testForm.InvokeRequired) { Thread.Sleep(delay); testForm.Invoke(new InvokeMethodHandler(InvokeMethod), new object[]{methodName, parms}); return; } Type t = testForm.GetType(); MethodInfo mi = t.GetMethod(methodName, flags); mi.Invoke(testForm, parms); } delegate void InvokeMethodHandler( string methodName, params object [] parms);
InvokeMethod gets a Type object representing the application form under test by calling the Object.GetType method. Then I get information about a specified method using Type.GetMethod and execute the specified method by calling MethodBase.Invoke . Invoke accepts two arguments. The first is the instance of the form on which to invoke the method, and the second is an array of parameters to the method. Now, in the case of a button control click method, the signature looks like this:
private void button1_Click(object sender, System.EventArgs e)
To satisfy the parameter requirements of the button1_Click method I need to pass an object that represents the sender and an EventArgs object that represents optional event data. For a button click I ignore the value of the first parameter, although for a real test system I should pass the control as the sender that was the cause of this method being invoked (the implementation of the method might depend on accessing that control, and this information is especially useful if this event handler method is used as the handler for multiple controls). For the second argument, I pass an empty EventArgs object.
Notice that the test automation simulates a button click by directly invoking the button control's associated method rather than by firing an event. When a real user clicks on a button it generates a Windows message which is processed by the control and turned into a managed event. This event causes a particular method (or set of methods) to be invoked. So the UI test automation will not catch the logic error if the application has the wrong method wired to a button click event (although every test would fail and you'd find the problem quickly). This can be rectified by getting the underlying multicast delegate for the event using reflection, and then using that delegate's GetInvocationList method to get a list of all of the delegates that will be invoked when the event is raised. Each delegate could then be invoked separately. Alternatively, you could use the event's EventInfo and its GetRaiseMethod method to get the MethodInfo for the method that raises the event, but that only returns a custom raise method, and the only Microsoft languages that support custom raise methods are C++ and Microsoft intermediate language (MSIL). Again, all of this can be avoided by using the send keys method discussed earlier.
Checking the Application State
After the automation sets the state of the application form by simulating user typing and clicking, it's time to check the system state to see if the application has responded correctly (see Figure 9).
Figure 9 Checking the State of the App Under Test
Console.WriteLine( "\nChecking for state:\n 'red', 'blue', 'Result is purple'"); bool pass = true; if ((string)GetControlPropertyValue("textBox1", "Text") != "red") pass = false; if ((string)GetControlPropertyValue("comboBox1", "SelectedItem") != "blue") pass = false; if ( !((ListBox.ObjectCollection)GetControlPropertyValue("listBox1", "Items")).Contains("Result is purple") ) pass = false; if (pass) Console.WriteLine("\nUI test scenario result: PASS"); else Console.WriteLine("\nUI test scenario result: *FAIL*");
I set a Boolean variable, called "pass", to true—I will assume the application state is correct and examine the state, setting pass to false if anything is wrong. I check to make sure that the textBox1 control's Text property is correctly set to "red". Then I check to make sure comboBox1 has "blue" and listBox1 has the correct message, "Result is purple". If everything checks out, I print a pass message; otherwise I print a fail message.
The key to checking the state of the application system is a GetControlPropertyValue method I've coded as shown in Figure 10. I start by using Object.GetType to create a Type object that represents the application form. Then I use Type.GetField to grab information about a specified control. Next I use GetType again to get a Type object representing the control. Finally, I use GetProperty to get information about a specified property of the control and then get the value of the control property using the GetValue method. GetValue requires an indexed object argument because properties can be indexed (for example if I was trying to get an Items property of a listbox control).
Figure 10 GetControlPropertyValue
static object GetControlPropertyValue( string controlName, string propertyName) { if (testForm.InvokeRequired) { Thread.Sleep(delay); return testForm.Invoke(new GetControlPropertyValueHandler( GetControlPropertyValue),new object[]{controlName, propertyName}); } Type t1 = testForm.GetType(); FieldInfo fi = t1.GetField(controlName, flags); object ctrl = fi.GetValue(testForm); Type t2 = ctrl.GetType(); PropertyInfo pi = t2.GetProperty(propertyName, flags); return pi.GetValue(ctrl, new object[0]); } delegate object GetControlPropertyValueHandler( string controlName, string propertyName);
Notice that checking the text in the listBox1 control is a little trickier than checking the text in the textBox1 control. I use my GetControlPropertyValue method to access the Items property and then check using the Contains method.
After I've examined the application state and logged a pass or fail result, I can easily exit the application under test:
Console.WriteLine("\nClicking menu File->Exit in 5 seconds . . . "); Thread.Sleep(3500); InvokeMethod("menuItem4_Click", new object[] {null, new EventArgs()} );
Even though the app under test will terminate when the test harness terminates because they're both running in the same process and because the application under test is running on a background thread, it's much better to explicitly exit the app from the test harness in order to explicitly clean up any system resources allocated.
Discussion
If you wanted UI test automation in the days before .NET, you really had only two choices. First, you could purchase a commercial UI automation tool. Second, you could create your own UI automation tools using Microsoft Active Accessibility (MSAA) APIs. The system I've presented complements these two other strategies quite nicely. There are several excellent commercial UI automation tools available to you. The advantage of these tools is their comprehensive list of features. The disadvantages are that you have to pay for them, they have a steep learning curve, and they don't allow you access to their source code in case you need to modify a feature. Using MSAA gives you full control over your automation tools but it can take a long time to learn. In fact, on several projects I worked on, the MSAA-based UI test automation was easily as complex as the application under test!
The automated UI test approach I've presented here has been used successfully on several medium- to large-scale products. It is very quick and easy to implement so it can be used early in the product cycle when the system under test is highly unstable. However, because this UI test system is relatively lightweight, it can't handle all possible UI testing situations. You can modify and extend this design in many ways. Because this presentation was intended to be primarily instructional, I've removed most error checking and I've hardcoded most information for clarity and simplicity. It's always important to add lots of error-checking code to test automation—after all, you're expecting to find errors.
Depending on your production environment, you may want to parameterize some parts of the test system. In testing terminology, the system I've presented is called a test scenario—a sequence of actions that manipulate the state of the application under test, (as opposed to a test case, which typically refers to a much smaller action like feeding an argument to a method and checking the return value.) To parameterize the scenario you could, for example, create an input file like this:
[set state] textBox1:Text:yellow textBox1:Text:red comboBox1:SelectedItem:green comboBox1:SelectedItem:blue button1:button1_Click [check state] textBox1:Text:red comboBox1:SelectedItem:blue listBox1:Items:Result is purple
Then you could have your test automation read, parse, and use the data in this file. You could use XML or SQL for test scenario input data, too. The test system I've presented logs its results to a command shell. You can easily redirect results to a text file from a command line or recast the automation to log results directly.
The next generation of Windows, code-named "Longhorn," will have a new presentation subsystem code-named "Avalon." Avalon promises to take the UI test automation concepts I've presented to the next level. It is slated to provide platform-level support for automation of all UI elements and expose a consistent object model for all user controls. This will allow developers and testers to create extraordinarily powerful UI test automation quickly and easily. The techniques in this column give you a hint of how revolutionary Avalon will be.
Before .NET, writing automation was often a very resource-intensive task and test automation, especially UI test automation, was often relegated to the bottom of the product task priority list. But with .NET, it's now possible to write very powerful test automation in a fraction of the time it used to take.
Send your questions and comments for James to testrun@microsoft.com.
James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.