Test Run
Lightweight UI Test Automation for ASP.NET Web Apps
James McCaffrey
Code download available at:TestRun0504.exe(116 KB)
Contents
The Web App Under Test
The Test Automation
Extending and Adapting the Automation
Wrap-Up
The release of ASP.NET revolutionized Web development and made it easy to create full-featured Web applications. Visual Studio® 2005 and ASP.NET 2.0 will let you add even more functionality to your applications, but the more features a Web application has, the more important testing becomes.
In this column I'll show you how to create lightweight test automation that can verify the functionality of your ASP.NET Web application through its user interface. To get a better idea of where I'm headed, take a look at the screen shots in Figure 1 and Figure 2. Figure 1 shows a simple but representative ASP.NET Web application. To use it, the user enters a color in a textbox control and clicks the Submit button. Information is sent to the Web server, some processing is done, and the Web application's state is updated.
Figure 1** The Web Application to Test **
Of course, your real Web application will be much more complex, but the techniques I'll show you can be employed to test complex applications as well. Having to test even this simple Web app manually through its user interface would be tedious and inefficient. And in a development environment you'd have to repeat the testing every time there was a change in your application's code base. A much better approach is to write test automation that simulates a user's actions. Figure 2 shows a screenshot of a test harness based in JScript® that does just that.
Figure 2** Sample Test Run **
If you examine Figure 2 you'll see that the automation tests the Web application through its user interface. The harness loads the application under test into an HTML frame (on the right) and uses a second frame (on the left) to hold a JScript test script that manipulates the app, determines if the app's functionality is correct, and logs a pass or fail result.
As you'll see, the test automation is easy to create, is applicable to a wide range of Web applications, and works with both traditional ASP and ASP.NET Web applications. In the sections that follow I will briefly examine the Web application under test so you'll know exactly what is being tested and how, and I'll explain in detail the test harness that generated the image in Figure 2. Then I'll discuss how you can extend the techniques presented here to meet your own needs.
The Web App Under Test
Let's take a look at the Web application under test so you can understand the goal of the test automation. Figure 1 shows the initial state of the Web app. After the user types in a color like "red" into the textbox control and clicks the Submit button, an HTTP request is sent to the Web server. The server processes the request and generates an HTTP response that contains a comment of some sort and an additional textbox control for the user to enter a second color. After another round-trip to the Web server, a second comment is displayed and an "All done!" message is placed in the document body. After a user types "red," submits, types "blue," and then submits again, the Web application will look like the right-hand frame in Figure 2.
If you were to run this sample, you'd notice that the test harness simulates typing "red," and then typing "blue" in the newly visible textbox control. Then the harness checks if the comment reads, "The sea is blue" and if "All done!" is displayed in the page body to determine if the test scenario has passed or failed. It then logs the result to the results.txt text file.
In order to create my test automation I need to examine the Web app's code. The entire code is listed in Figure 3. I used C# for the Web application logic code but the technique discussed in this column does not depend on your language. To keep the code as short as possible for the column, I created the ASP.NET Web application using an ordinary text editor rather than using Visual Studio .NET as I normally would.
Figure 3 Web Application Under Test
<script language="c#" runat="server"> private void Button1_Click(object sender, System.EventArgs e) { if (TextBox1.Text == "red" && TextBox2.Visible == false) { TextBox3.Text = "Roses are red"; Label2.Visible = true; TextBox2.Visible = true; } else if (TextBox1.Text == "blue" && TextBox2.Visible == false) { TextBox3.Text = "The sky is blue"; Label2.Visible = true; TextBox2.Visible = true; } else if (TextBox2.Visible == false) { TextBox3.Text = "Unknown color"; Label2.Visible = true; TextBox2.Visible = true; } else if (TextBox2.Text == "red") { TextBox3.Text = "Some apples are red"; Label3.Visible = true; } else if (TextBox2.Text == "blue") { TextBox3.Text = "The sea is blue"; Label3.Visible = true; } else TextBox3.Text = "Try again"; } // Button1_Click() </script> <html> <head><title>color.aspx</title></head> <body bgColor="#ccffcc"> <h3>ASP.NET Web Application under Test</h3> <form method="post" name="theForm" id="theForm" runat="server"> <p><asp:Label id="Label1" runat="server">Enter a first color:  </asp:Label> <asp:TextBox id="TextBox1" runat="server" /> <p><asp:Label id="Label2" visible="false" runat="server">Enter a second color:  </asp:Label> <asp:TextBox id="TextBox2" runat="server" visible="false"/> <p><asp:Button id="Button1" runat="server" text="Send" onclick="Button1_Click" /> <hr> <p>My comment:  <asp:TextBox id="TextBox3" runat="server" /> </form> <p><asp:Label id="Label3" runat="server" visible="false">All done!</asp:Label> </body> </html>
Let me emphasize that I am purposely ignoring good coding style here because I want to keep the application example short. Plus, it simulates the raw nature of a typical prerelease application. The Web application's initial state has a prompt for the user to enter a color and a textbox control to accept the input. After the user submits the HTTP request, the application logic displays a comment and dynamically generates a second prompt and textbox control by modifying the textbox's Visible property. This is obviously rather artificial but makes explicit the idea that your ASP.NET Web application's state changes with each HTTP request-response pair. In other words, even if the Web application you want to test accesses a SQL Server™ database or does very complex processing, it still just changes state and will be reflected in the resulting HTTP response and in the user interface.
The Test Automation
The test harness system consists of three files. Figure 4 shows a block diagram of the overall structure. The Web application under test (color.aspx) is loaded into an HTML frame, and an HTML page (left.html) that contains the test automation JScript code is loaded into another HTML frame. The third Web page (main.html) is a container for the two frames and holds a global variable called timesAppLoaded which tracks how many times the Web application under test has been loaded into its test frame. The test scenario script uses the timesAppLoaded value to determine exactly what actions to take on the Web app and then reloads the Web application. The cycle continues until the system reaches a final state that you specify in the test script, and a pass or fail result is determined.
Figure 4** Test Harness System **
You saw the Web application's code in the previous section, but to test it using the technique described in this column you do not have to modify or instrument the application code—a nice feature of this technique. The code for main.html is extremely short, as you can see in Figure 5.
Figure 5 Main.html
<html> <head> <script language="JScript"> var timesAppLoaded = 0; var scenarioID = "001"; </script> </head> <frameset cols="40%,*"> <frame src="left.html" name="left"> <frame src="color.aspx" name="right" onload="left.updateState();"> </frameset> </html>
The key to my test automation technique is keeping track of the Web application's state so I can synchronize the test harness with application events. I also declare and assign a variable named scenarioID. In a production environment you can add other test harness metadata into this section of code.
In main.html I named the two frames "left" and "right" to reflect their position relative to each other in the test harness from a tester's point of view. Notice that every time the color.aspx Web application is loaded or reloaded into the test harness right frame, a function called updateState in the left frame is called. As you'll see later, updateState initiates the simulated user's actions on the Web application that is under test.
The heart of the test harness is the left.html file (see Figure 6). I will walk you through it in detail so that you'll be able to modify it to meet your own needs. Although my JScript code is organized into function blocks, it is relatively tightly coupled, which is usually not quite as good as a loosely coupled design. This design tradeoff keeps my code shorter, however. You may want to parameterize the code to make it a little more flexible.
Figure 6 Left.html
<html> <head> <script language="JScript"> function updateState() { parent.timesAppLoaded++; if (parent.timesAppLoaded > 1) runScenario(); } // updateState() function runScenario() { try { if (parent.timesAppLoaded == 1) { addComment("Setting TextBox1 to 'red'"); addComment("Clicking submit button"); parent.right.document.theForm.TextBox1.value = "red"; parent.right.document.theForm.Button1.click(); } else if (parent.timesAppLoaded == 2) { addComment("Setting TextBox2 to 'blue'"); addComment("Clicking submit button"); parent.right.document.theForm.TextBox2.value = "blue"; parent.right.document.theForm.Button1.click(); } else if (parent.timesAppLoaded == 3) { addComment("Checking app state"); checkFinalState(); } } catch(e) { addComment("Unexpected fatal error: " + e); } } // runScenario() function checkFinalState() { var pass = true; if (parent.right.document.theForm.TextBox1.value != "red") pass = false; if (parent.right.document.theForm.TextBox2.value != "blue") pass = false; if (parent.right.document.theForm.TextBox3.value != "The sea is blue") pass = false; var trange = parent.right.document.body.createTextRange(); if (trange.findText("All done!") == false) pass = false; addComment("Determining pass / fail"); if (pass == true) result.value = " Pass "; else result.value = " *FAIL* "; addComment("Saving to 'results.txt'"); saveResults(); addComment("End test scenario"); } // checkFinalState() function addComment(comment) { var currComment = document.all["comments"].value; var newComment = currComment + "\n" + comment; document.all["comments"].value = newComment; } // addComment() function saveResults() { var fso = new ActiveXObject("Scripting.FileSystemObject"); var f = fso.CreateTextFile("C:\\Results\\results.txt", 2, true); f.WriteLine("Scenario = " + parent.scenarioID); if (result.value == " Pass ") f.WriteLine("Result = Pass"); else f.WriteLine("Result = FAIL"); f.Close(); } // saveResult() </script> </head> <body bgColor="#ee7755"> <h3>Test Scenario Script</h3> <p><input type="button" value="Run Scenario" onclick="runScenario();"> </p> <p><textarea id="comments" rows="10" cols="27">Actions:</textarea></p> <p>Scenario Result = <input type="text" name="result"></p> </body> </html>
Let's suppose that you launch Microsoft® Internet Explorer and request main.html. Main.html will load and in turn load its two frames containing the test script in file left.html and the colors.aspx Web application. Figure 7 shows what the test system will look like at this point.
Figure 7** Test Harness Initial State **
The variable timesAppLoaded is initialized to 0 and control is immediately transferred to the updateState function in left.html:
function updateState() { parent.timesAppLoaded++; if (parent.timesAppLoaded > 1) runScenario(); } // updateState()
The variable timesAppLoaded is next incremented by 1. Notice that because timesAppLoaded is declared in main.html I have to use the JScript keyword "parent" to access it. Next, the function updateState checks to see if the value of timesAppLoaded is greater than 1. Because this is the initial load, timesAppLoaded is not greater than 1 so the thread of execution ends.
The body portion of the test harness just consists of a button to start the automation, an HTML <textarea> to display comments describing the progress of the automation, and an HTML text field to display the final test scenario result:
<body bgColor="#ee7755"> <h3>Test Scenario Script</h3> <p><input type="button" value="Run Scenario" onclick="runScenario();"> </p> <p><textarea id="comments" rows="10" cols="27">Actions:</textarea></p> <p>Scenario Result = <input type="text" name="result"></p> </body>
Now suppose you click on the Run Scenario button to initiate the test automation. Control is transferred to the runScenario function, which immediately checks the value of variable timesAppLoaded and branches if that value is 1:
if (parent.timesAppLoaded == 1) { addComment("Setting TextBox1 to 'red'"); addComment("Clicking submit button"); parent.right.document.theForm.TextBox1.value = "red"; parent.right.document.theForm.Button1.click(); }
The harness code calls a helper function, addComment, that displays some comments into the harness <textarea> comments field. Then the harness sets the value of the Web app's TextBox1 control to "red" and invokes the Web app's Submit button. This will trigger an HTTP request to color.aspx where the comment "Roses are red" will be generated and a second prompt and textbox will become visible to the test harness. The Web app will accept the resulting HTTP response, reload into its frame, which in turn will fire an onload event, and call the updateState function again.
Function updateState increments the global variable timesAppLoaded to 2 and in turn calls the runScenario function again. This time through, the runScenario function will branch when timesAppLoaded equals 2:
else if (parent.timesAppLoaded == 2) { addComment("Setting TextBox2 to 'blue'"); addComment("Clicking submit button"); parent.right.document.theForm.TextBox2.value = "blue"; parent.right.document.theForm.Button1.click(); }
Some new comments are added to the test harness comment area, the now-visible TextBox2 control is given a value of "blue," and the Submit button click event is simulated. This sends an HTTP request to the Web server, a new comment "The sea is blue" is generated, and an "All done!" message is written into the HTTP response page body. The HTTP response reloads into its frame in the test harness, which once again fires an onload event and calls the updateState function.
The updateState function increments timesAppLoaded to 3 and calls the runScenario function, which branches for the last time:
else if (parent.timesAppLoaded == 3) { addComment("Checking app state"); checkFinalState(); }
After adding a comment, the checkFinalState function is called. This function examines the Web application to see if the correct final state exists. I declare a variable named pass and initialize it to true, the idea being that I will assume the Web application's state is correct and scan to see if anything is wrong. First I check the three TextBox controls:
var pass = true; if (parent.right.document.theForm.TextBox1.value != "red") pass = false; if (parent.right.document.theForm.TextBox2.value != "blue") pass = false; if (parent.right.document.theForm.TextBox3.value != "The sea is blue") pass = false;
Notice that because the harness code is in a different frame from the code for the app under test, I must use the keyword "parent" and the frame name "left" to access the Web app's data by element ID. I have to use a slightly tricky technique to examine the text in the body of Web application:
var trange = parent.right.document.body.createTextRange(); if (trange.findText("All done!") == false) pass = false;
The TextRange object is a very useful part of the Internet Explorer Document Object Model (DOM) and allows you to examine and manipulate those parts of a Web page outside of an HTML element. Now I can determine whether the test scenario passed or failed and save my results to an external file using a saveResults helper function, as shown here:
addComment("Determining pass / fail"); if (pass == true) result.value = " Pass "; else result.value = " *FAIL* "; addComment("Saving to 'results.txt'"); saveResults(); addComment("End test scenario");
The helper functions addComment and saveResults do not use any special tricks. The saveResults function uses ActiveX® technology to write results directly to an external file. The advantage of this technique is that it is very simple. A disadvantage is that by default Web pages cannot write to files for security reasons. So, I had to add the test harness page to the list of Internet Explorer Trusted Sites (if you examine Figure 2 you'll see the Trusted Sites icon in the Internet Explorer status bar) and I had to enable ActiveX objects not marked as safe to avoid getting a warning message when saveResults is called. I'll discuss two alternatives to this in the following section of this column.
Extending and Adapting the Automation
You can extend and adapt the lightweight ASP.NET Web application UI test automation technique I've presented here in several ways. Notice that the test harness is not truly automated because you have to manually click on the Run Scenario button to launch the automation. If you want to fully automate the harness, all you have to do is change the code in updateState function from
if (parent.timesAppLoaded > 1) runScenario();
to
if (parent.timesAppLoaded > 0) runScenario();
and create a batch file with the single statement:
iexplore.exe https://localhost/WebUIAutomation/main.html
Recall that when main.html loads it will set the value of the global variable timesAppLoaded to 0 and then load the application under test (in this column colors.aspx) which will cause updateState to be invoked, incrementing timesAppLoaded to 1. If you specify timesAppLoaded > 0, this will be immediately true and cause runScenario to be called, initiating the test scenario. By doing this, you can schedule the BAT file and hence the test harness to run automatically (for example, using the Windows Task Scheduler or from an automated build process) and automatically send test-run result summaries via e-mail.
The code I've presented manipulates the Web application under test through several states and then checks the final state to determine if the scenario passes or fails. An alternative is to check the state of the app after each state change. I've used both techniques in the past and have found that the best method depends entirely on the testing situation you're facing.
Testing ASP.NET app functionality through the UI is just one type of testing you'll need to perform. Performance testing, stress testing, security testing, and testing at the HTTP request/response level (see Test Run: Test Automation for ASP.NET Web Apps with SSL), are also critical. You can modify this technique to handle many of these testing situations. Load-testing tools usually accept a list of Web pages to navigate to, so by adding a test scenario page that simulates a user navigating to multiple page states, you'll get a very realistic profile.
The technique presented here calls a saveResults function that uses methods in Scripting.FileSystemObject to write results to an external file. But this means you have to modify the Trusted Sites list in Internet Explorer and the ActiveX control scripting execution properties. If you do this, you'll invalidate many kinds of security testing scenarios. There are two alternatives. First, you can add an HTML form to the test-automation script, then use the form to send the test scenario results to an ASP.NET page that reads the form pass/fail data, and writes it to external storage (text file, XML file, SQL table, and so on). Second, you can write results to a cookie on the test client machine and then create a small JScript helper function that reads and parses the cookie and saves the test scenario result to external storage.
Wrap-Up
Because this column is primarily instructional, for clarity I used only minimal error checking. In a production system you'll want to properly implement error and exception handling in your test harness to make sure your automation does not stop in the middle of the run. I also hardcoded most of the information in the test harness. You may want to parameterize some of the data, such as the simulated user inputs, to make your test system more flexible.
With Web application systems growing in complexity and with security becoming increasingly important, testing your software is more vital than ever before. Lightweight ASP.NET Web application UI testing as I've described here is an important part of your product's testing effort and the tools provided by Microsoft make this kind of testing simple to implement.
Send your questions and comments for James to testrun@microsoft.com.
James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.