Introducing Microsoft Visual Studio 2005 Team System Web Testing

 

Mark Michaelis

September 2005

Applies to:
   Microsoft Visual Studio 2005 Team System
   Web development

Summary: This article introduces the Web Testing functionality in Microsoft Visual Studio 2005 Team System, and demonstrates how to create and customize a Web test case. (22 printed pages)

Contents

Introduction
Creating a Personal Website
Creating a Test Project
Running a Web Test Case
Request Rules
Binding Test Data to a Data Source
Generating Web Test Code
Extending VSTS Web Testing
Browser User Interface Testing
Test-Driven Development
Conclusions

Introduction

During a recent meeting, I was confronted by a development team that insisted Web testing was impossible, and anyone who figured out how to do it would make a lot of money. Well, Microsoft has already figured out the money part, and in their upcoming Microsoft Visual Studio Team System release, they also provide a tool for the Web testing portion. This article provides an overview of this functionality. It begins by providing a step-by-step approach on how to set up a Web test case and customize it without writing any code. This demonstrates the approachability of VSTS Web Testing by all those participating in the development process, including non-developer types. Web test cases can easily be coded as well, and we will describe how to use coded Web tests or extend the built-in Web testing support.

Before we begin, readers should be aware that VSTS functionality is not targeted at user interface testing. It doesn't run a Web page's JavaScript or validate the appearance of a page within multiple browsers. Rather, the testing approach is to examine HTTP data flowing over the wire and provide various rules for validating this data.

To begin, we need to set up both the website and the test project itself, because VSTS Web Testing supports activity recording, as long as there exists a website to record against. After we create the website, we will add automated tests against it (not exactly test-driven development, but the recording functionality is too convenient to ignore.)

Creating a Personal Website

The site we are going to test in this article is a personal website. This is one of the wizard-created sites that come with Microsoft Visual Studio 2005. Below are the steps for creating the site:

1. Click the File menu, and then click New Web Site. Note that in Visual Studio 2005, the new website projects are in a separate dialog box from the other new project options.
2. Select Personal Web Site Starter Kit and select a programming language. In this article, we will use Microsoft C#.

Figure 1. Creating the new website (Click the image for a larger picture)

Once the site has been created, we create a new account. We will do this manually, just to verify that everything is working correctly. Later, we will provide an automated test that logs in to the site using the new account.

1. Click Website and then ASP.NET Configuration to open up the website administration tool.
2. Click the Security link, followed by the Create User link.
3. Enter a test user's information, selecting Administrators for the role, before clicking the Create User button. Note that the e-mail address must have a valid format and that the password must be seven characters, including at least one that is non-alphanumeric.
4. After creating the account, shut down the ASP.NET Configuration website.

From within Visual Studio, we can launch the website and login using the new account, to verify that everything works correctly.

By default, Microsoft Internet Information Server (IIS) is not used to host the site. Instead, Visual Studio 2005 starts a Microsoft ASP.NET Development Server, as indicated by the system tray icon in Figure 2.

Figure 2. ASP.NET Development Server system tray icon

The ASP.NET Development Server randomly chooses an open port and begins hosting the site on that port, using a virtual path corresponding to name of the Web project. However, the port could change between launches of Visual Studio, so any tests recorded against the original port would no longer function. To avoid this, we can freeze the port to a particular value:

1. In Visual Studio, go to the Properties window (by pressing F4) of the Web project to get the port number. Note that this is not the project's Property Pages dialog box, but rather a docked window within Visual Studio 2005. The Use Dynamic Ports value defaults to True.
2. To prevent Visual Studio from selecting a different port number, change the Use Dynamic Ports option to False.

Another way to launch the website is by running WebDev.WebServer.exe directly. This also provides a means to specify the port number on which to host the site. The command line is as follows.

WebDeb.WebServer /port:<port number> /path:<physical path> 
  [/vpath:<virtual path>]

For example, we could launch the personal website using the following.

  start /B webdev.webserver.exe /port:4955 
  /path:"c:\documents and settings\MMichael\Local 
  Settings\Temp\HelloWorldWebSite"  /vpath:/HelloWorldWebSite

Start makes sure the command returns, rather than waiting for it to exit.

If no port is specified it defaults to 80, but if IIS is running and using the same port, this will fail. In addition to allowing for a specific port, WebDev.WebServer.exe is important because it can be invoked as part of a build/test script without launching Visual Studio 2005.

Upon verifying that the site is functioning, record the URL (including the port and virtual directory) of the site's default page and close the browser window. We will use the URL shortly when we record a Web test case. However, we need a test project before we can run a Web test.

Creating a Test Project

We create the test project just as we would any VSTS unit testing project. The steps below outline the process. In order to successfully record a test, the website must be running.

1. Right-click the solution and click Add Project.
2. Choose the programming language and select the Test node.
3. Name the project HelloWorldWebSite.Test.

Figure 3. Creating a test project (Click on the image for a larger picture)

Since we have already created an initial website, we can record the activity against the site.

There is nothing specific about creating a project that defines it as a Web test project. Instead, the test files added to the project are specifically Web test files rather than another test file type (unit test, load test, manual test, generic test, and so on). The next step, therefore, is to add a Web test case. Since we are not going to use any of the other types of test files added to a test project, these can be deleted.

1. Right-click the project, click Add, and then click Web Test. This opens up a browser with a special Web Test Recorder explorer bar.

   

   Figure 4. Recording a Web test case

2. On the address bar, enter the URL of the personal website, including the port selected by the ASP.NET Development Server. Browsing to this location will be recorded in the Web Test Recorder explorer bar as would any other URLs that are entered.

3. Enter the user name and password that were added earlier. Upon clicking the Login button, another entry will be recorded, along with the form post parameters. That way, when the test is run, the same data will automatically be sent. Even the X and Y coordinates of where the button was clicked are saved as part of the test, since these are also submitted as part of the request.

4. Add additional steps to the test by logging out of the site and then re-attempting the login with invalid credentials.

5. Once the desired tests have been recorded, close the browser windows and save the test.

Automatically, the project will now include the Web test case file along with each of the recorded requests.

Figure 5. Viewing a recorded Web test case

Selecting any of the nodes within the WebTest tree will allow you to modify the data inside the Properties window.

We can also group requests together using a transaction. Be careful not to confuse the term "transaction" with a programming concept in which state is committed or not committed as a unit. Within Web test cases, transactions only encapsulate actions into a group that can later be enumerated using code. Also, transactions are used when reporting on load—how many transactions per second, for example.

Another tweak that can be added to each request and the entire Web test case itself is a comment. This involves right-clicking a request and clicking Insert Comment. Comments appear between requests and provide a means of documenting data submitted, as well as any response validation that may be added.

Running a Web Test Case

After recording a test we are ready to begin executing it. To execute all the tests within a project, simply run the project. This will open up the Test Results windows and mark each test as pending while it is in progress, and Passed/Failed once execution completes. Test selection and execution is also available from the Test Manager and Test View windows.

Figure 6. Selecting and executing tests

Individual Web test files (test cases or test fixtures) can also be run by opening them up and clicking the Run button.

Figure 7. Run Settings dialog box

Requests can also provide credentials for logging on to the targeted site using standard authentication methods. The dialogs for credentials allow for loading the login data from a data source.

Each result from a request is saved, and selecting each request allows you to navigate its detail, viewing the resulting page's HTML or raw request/response text.

Figure 8. Test results viewer

As part of a test's execution, the Web test engine checks all URLs on the page to verify that they are valid links. These links appear as child nodes below the request, and each shows the HTTP status returned by a request to that URL.

Request Rules

Although checking for valid hyperlinks on a response page is a useful feature, it is not sufficient in validating that the page is functioning correctly. For example, upon entering valid credentials, the Web test needs to verify that the login was successful. Similarly, when the credentials are invalid, we need to check that an appropriate error message is displayed on the page. To support this, each Web request can include extraction rules and validation rules.

Extraction Rules

Extraction rules capture a response value so that at a later time the value can be used within a request. Rather than parsing the entire HTTP response manually, the extraction rules provide a means of focusing in on a particular data item within the response. The extracted data item can then be validated or used again in a subsequent post back. Our recorded example automatically added an extraction rule when we logged on.

Figure 9. Extraction rules can be linked between requests

The rule is an ExtractHiddenFields rule whose data is posted back in the second request of the Web test case. During the subsequent request, this data is submitted back in a hidden field on the page. Other extraction rule options are ExtractAttributeValue, ExtractHttpHeader, ExtractRegularExpression, and ExtractText. Rather than relying on automatically recorded hidden field data, extraction rules can be manually added and customized as needed.

Validation Rules

Validation rules allow the test writer to examine the entire HTTP response and verify that it is correct. For example, valid credentials should cause the LOGOUT hyperlink and "Welcome <username>" to appear in the HTML response. A validation rule that checks for these items in the response should be added. The validation rules verify that this text appears somewhere within the response body.

Figure 10. A validation rule that checks for LOGOUT

If a particular rule fails, the request will be marked as failed, and the Details tab will provide an explanation for what failed.

Figure 11. Displaying the validation and extraction rule results

This type of validation rule is a ValidationRuleFindText. It searches the entire response body, looking for the specified text. In order to handle complex searching, regular expressions are supported as part of ValidationRuleFindText.

In addition to ValidationRuleFindText, built-in validation rules include ValidationRuleRequestTime, ValidationRuleRequiredAttributeValue, and ValidationRuleRequiredTag.

Binding Test Data to a Data Source

Both validation and extraction rules provide for text entry within the Properties window of virtually every node. However, what makes Visual Studio Web Test powerful is the fact that the text can be pulled from a database. In our logon example, this means we could define a new user and verify that a collection of weak passwords that don't conform to the specified strong password requirements are not allowed.

To test using a data source, click the Add Data Source button on the toolbar of the Web test. In the ensuing dialog box, specify an OLE DB Provider, perhaps using an *.mdf file that can also be added to the test project. After opening the database in the server explorer, define a table that will contain the necessary test data. In Figure 12, we use the TestUser table, which has columns for the primary key, user name, and password.

Figure 12. Assigning a parameter to a data source

Once a test has been configured with a data source, it is necessary to return to the Edit Run Settings dialog box (Figure 7) and change the run count to one run per data source row. In this way, the test will repeat for each row in the newly configured data source, and during each run, the parameters associated with the data source will be assigned the value in the column for the particular row.

Generating Web Test Code

Taking advantage of all the functionality we have considered so far has not required any code to be written. However, additional Web test customization is available using code. This is necessary to handle constructs like looping and branching within a test, or to call out to another Web test. VSTS Web Testing provides the facility of generating the code for a particular test case. Included on the toolbar for a web test case is a Generate Code button. Clicking this button prompts for a test name, and then generates a CS/VB file corresponding to Web case. The generated code includes each validation and extraction rule that may have been added. In addition, data such as the view state is set and passed as part of the Web request. The generated code from the test case described earlier appears below.

namespace HelloWorldWebSite.Test
{
  using System;
  using System.Collections.Generic;
  using Microsoft.VisualStudio.QualityTools.WebTestFramework;
  using Microsoft.VisualStudio.QualityTools.WebTestFramework.Rules;


  public class LogonCoded : WebTest
  {

      public LogonCoded()
      {
      }

      public override IEnumerator<WebTestRequest> GetRequestEnumerator()
      {
          WebTestRequest request1 = 
               new WebTestRequest("https://localhost:4955/HelloWorldWebSite");
          request1.ThinkTime = 17;
          ExtractHiddenFields rule1 = new ExtractHiddenFields();
          rule1.ContextParameterName = "1";
          request1.ExtractValues += 
               new EventHandler<ExtractionEventArgs>(rule1.Extract);
          yield return request1;

          // Redirect to the default page.

          WebTestRequest request2 = 
               new WebTestRequest(
               "https://localhost:4955/HelloWorldWebSite/default.aspx");
          request2.ThinkTime = 7;
          request2.Method = "POST";
          BindHiddenFields request2BindHiddenFields = 
               new.BindHiddenFields();
          request2BindHiddenFields.HiddenFieldGroup = "1";
          request2.PreRequest += 
               new EventHandler<PreRequestEventArgs>(
               request2BindHiddenFields.PreRequest);
          FormPostHttpBody request2Body = new FormPostHttpBody();
          request2Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$UserName", "Administrator");
          request2Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$Password", "G00d-P@ssw0rd");
          request2Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$LoginButton.x", "45");
          request2Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$LoginButton.y", "5");
          request2.Body = request2Body;
          ValidationRuleFindText rule2 = new ValidationRuleFindText();
          rule2.FindText = "Administrator";
          rule2.PassIfTextFound = true;
          request2.ValidateResponse +=
               new EventHandler<ValidationEventArgs>(rule2.Validate);
          ValidationRuleFindText rule3 = new ValidationRuleFindText();
          rule3.FindText = "LOGOUT";
          rule3.PassIfTextFound = true;
          rule3.IgnoreCase = true;
          request2.ValidateResponse += 
               new EventHandler<ValidationEventArgs>(rule3.Validate);
          ValidationRuleRequestTime rule4 = 
               new ValidationRuleRequestTime();
          rule4.MaxRequestTime = 500;
          request2.ValidateResponse += 
               new EventHandler<ValidationEventArgs>(rule4.Validate);
          yield return request2;

          // Logon using valid credentials and verify successful logon.

          WebTestRequest request3 = 
               new WebTestRequest(
               "https://localhost:4955/HelloWorldWebSite/default.aspx");
          request3.ThinkTime = 17;
          request3.Method = "POST";
          FormPostHttpBody request3Body = new FormPostHttpBody();
          request3Body.FormPostParameters.Add(
               "__EVENTTARGET", "ctl00$LoginStatus1$ctl00");
          request3Body.FormPostParameters.Add(
               "__EVENTARGUMENT", "");
          request3Body.FormPostParameters.Add(
               "__VIEWSTATE", ...);
          request3.Body = request3Body;
          ExtractHiddenFields rule5 = new ExtractHiddenFields();
          rule5.ContextParameterName = "1";
          request3.ExtractValues +=
               new EventHandler<ExtractionEventArgs>(rule5.Extract);
          yield return request3;

          // Logout

          WebTestRequest request4 = 
               new WebTestRequest(
               "https://localhost:4955/HelloWorldWebSite/default.aspx");
          request4.Method = "POST";
          BindHiddenFields request4BindHiddenFields = 
               new BindHiddenFields();
          request4BindHiddenFields.HiddenFieldGroup = "1";
          request4.PreRequest += 
               new EventHandler<PreRequestEventArgs>(
               request4BindHiddenFields.PreRequest);
          FormPostHttpBody request4Body = new FormPostHttpBody();
          request4Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$UserName", "IMontoya");
          request4Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$Password", "bad-password");
          request4Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$LoginButton.x", "64");
          request4Body.FormPostParameters.Add(
               "ctl00$Main$LoginArea$Login1$LoginButton.y", "10");
          request4.Body = request4Body;
          ValidationRuleFindText rule6 = new ValidationRuleFindText();
          rule6.FindText = "LOGIN";
          rule6.IgnoreCase = true;
          rule6.PassIfTextFound = true;
          request4.ValidateResponse += 
               new EventHandler<ValidationEventArgs>(rule6.Validate);
          ValidationRuleFindText rule7 = new ValidationRuleFindText();
          rule7.FindText = "LOGOUT";
          rule7.IgnoreCase = true;
          rule7.PassIfTextFound = false;
          request4.ValidateResponse += 
               new EventHandler<ValidationEventArgs>(rule7.Validate);
          yield return request4;

          // Logon using invalid credentials and verify 
          // unsuccessful logon.
      }
  }
}

For a C# project, the new C# 2.0 iterator is used: after each request, the code returns the next Web request through a yield return statement that separates out one request from the next.

Although there is obviously no reason to generate the code if it isn't going to be customized, doing so provides a great starting point for customization of a particular Web test case, or even multiple cases, with a little refactoring.

Extending VSTS Web Testing

The available request rules and the ability to write custom code from generated Web tests covers the most common Web testing scenarios. However, it sometimes makes more sense to extend VSTS Web Testing by creating custom validation and extraction rules, or by coding custom Web test plug-ins. Such extensions must be defined in a separate assembly, and can be used across multiple Web testing projects. Once defined, the new Web test extension assemblies may be referenced by the Web test's project so that it appears in a requests selection of Add dialog boxes.

The available dialog boxes are not only for validation and extraction rules, but for Request and Test plug-ins as well. Request callbacks can be added separately to each request, performing pre-interception and post-interception on the request. Test callbacks run pre-interception and post-interception code on entire set of requests. They are initially called at the beginning of the test case, and then again at the end of the same test case. For example, consider defining a callback that checks whether all Web pages in the test conform to XHTML, or one that sets up a cookie for use within each Web request. If the plug-in that performed this validation was a request callback, then it could be added individually to each request within the test. Alternatively, the callback could be a test callback that hooked up validation for all requests during the pre-test execution stage. The following steps demonstrate creating a Web test callback:

1. Create a new Class Library project called WebTestFramework.
2. Add references to Microsoft.VisualStudio.QualityTools.UnitTestFramework and Microsoft.VisualStudio.QualityTools.WebTestFramework.
3. Define a new class derived from Microsoft.VisualStudio.QualityTools.WebTestFramework.WebTestPlugin, called ValidationRuleXHTML.
4. Implement the abstract methods PreWebTest() and PostWebTest().

These last two methods conform to standard event signatures, with sender as the first parameter and PreWebTestEventArgs/PostWebTestEventArgs as the second argument. Below is a sample Web test plug-in class, demonstrating how to create an XHTML validator.

namespace WebTestFramework
{
    public class ValidationRuleXHTML :
        Microsoft.VisualStudio.QualityTools.WebTestFramework.WebTestPlugin
    {
        public override void PostWebTest(
            object sender, PostWebTestEventArgs args)
        {
            // No PostWebTest processing required.  Implemented
            // because it is an abstract method in base class.
        }

        void WebTestPostRequest(
            object sender, PostRequestEventArgs args)
        {
            // 1. Retrieve the ResponseBody from 
            //    args.Response.ResponseBody.
            // 2. Validate the ResponseBody.
            // 3 .Throw an exception if the HTML is not valid XHTML.  
            //    Be sure to set the message of the exception to be as
            //    descriptive as possible.
        }

        public override void PreWebTest(
            object sender, PreWebTestEventArgs args)
        {
            // Register an event callback for each PostRequest
            // within the test.
            args.WebTest.PostRequest += WebTestPostRequest;
        }
    }
}

In this example, all the PreWebTest() does is register a delegate (WebTestPostRequest()) for the PostRequest event. This way, whenever a request is processed by the Web testing engine, it will call into registered delegate listener. The delegate's PostRequestEventArgs parameter includes a WebTest property that provides a string ResponseBody property. This last property provides needed access to the response body when we check for XHTML. One caution about things like XML validators is that when running a load test, they use up a significant chunk of the load generation capabilities. Validation tests that require significant resources when used in mass should remain within unit tests that are not used within load testing scenarios.

The object model for building a web test plug-in is shown in Figure 13.

Figure 13. The Web test plug-in object model (Click on the image for a larger picture)

This model provides an overview of what functionality can be accessed within the Web test plug-in, and therefore, what validation can be provided in a Web test plug-in. In the pre/post Web test methods, we can access information about the Web test case as a whole—for example, data sources associated with the test, along with access to the credential information. Perhaps most importantly, these methods provide access to the pre/post request events that enabled us to hook up an XHTML validator callback on each request.

Both the pre-request and post-request event signatures provide access (through their respective EventArgs class) to a WebTest object. They also provide access to a WebTestRequest object for accessing details about the request itself. The PostRequestEventArgs property includes access to a WebTestResponse object, enabling examination of the data returned from the request.

Earlier, we mentioned that custom validation and extraction rules are also possible. To create such rules, we derive from Microsoft.VisualStudio.QualityTools.WebTestFramework.Validation and Microsoft.VisualStudio.QualityTools.WebTestFramework.ExtractionRule instead of Microsoft.VisualStudio.QualityTools.WebTestFramework.WebTestPlugin or Microsoft.VisualStudio.QualityTools.WebTestFramework.WebTestRequestPlugin. An overview of the rule base classes is shown in Figure 14.

Figure 14. Extending with custom extraction and validation rules

The ValidationEventArgs and ExtractionEventArgs classes are the type parameters on the respective Validate() and Extract() methods of the rules. When working with individual requests, these event arguments provide direct access to more request-related data than their plug-in counterparts. Furthermore, properties such as IsValid provide a better means of reporting an error than the throwing exception mechanism of a Web test plug-in described earlier. However, as indicated earlier, rules must be inserted on an individual request-basis, rather than across an entire suite of tests. Another advantage of the plug-in approach is that it can intercept the request itself—it is not only called after the request has been made and the response returned. In summary, use validation and extraction rules unless providing validation across all requests or needing to intercept the request before it is submitted to the server.

Browser User Interface Testing

Web requests through JavaScript, ActiveX controls, and applets are not supported within the VSTS Web Testing functionality. Similarly, VSTS Web Testing is not designed to be a user interface (UI) testing tool. It will not execute client-side JavaScript and verify the results. Even a simple menu click-and-expand type action cannot be simulated by the tool. Even though it simulates particular browser clients to the server, it does nothing in the way of verifying that the response back renders correctly within that client browser, even when it is Microsoft Internet Explorer. VSTS Web Testing is a wire-based testing protocol. It verifies what is sent and received across the wire, and provides no built-in capability for testing how the data is rendered by the browser.

Providing this type of testing is difficult and cumbersome. However, there are some methods to consider for certain situations. It is reasonable to assume that if the same response occurs multiple times, it will render and function within the browser in the same manner. Therefore, if you manually verify that a particular response is correct—for example, by checking that the JavaScript is behaving appropriately and that the page renders correctly—you can expect the same response will behave correctly the next time. Using this principal, you can visually verify the response, manually checking that a script behaves appropriately. Now a validation test can be created that checks for a similar response, using wildcards to handle minor data changes such as variances in data time, user name, advertising, and the like. If, in future runs of the test, the page changes, then the response should be re-verified and the test updated accordingly, thereby providing a level of change control for the page. This is not something a team is likely to deploy in mass, but it does provide a good baseline testing mechanism and forces controlled variation.

Test-Driven Development

Throughout this article we have tested an existing website, rather than taking a test-driven development (TDD) approach in which we write the test first and then the pages to satisfy those tests. The driving force behind not using a test-driven methodology is the recording functionality of VSTS Web Testing. Rather than manually entering the URL for each request, we browse around the website and have the Web Recorder save each request into the test automatically.

The recording feature is a significant enough advantage that it outweighs manually entering each request. However, it is still possible to take advantage of a test-driven approach. First, design the page, including any links corresponding to proceeding requests and any buttons that submit requests. Other than this, coding should be minimal. Next, record the navigation around the site, adding the appropriate requests into the test case structure. Last, open the Web test case and modify it to include validation and extraction rules. At this point, running the Web test case should produce a failed result, because code has not been added that fulfills the rules. Therefore, we are ready to proceed from writing tests to writing code as the TDD approach dictates.

For example, consider the login scenario using TDD. Initially, a developer creates a login page and adds a Login button that submits the credentials. Next, we open up the Web recorder and record the submit request. Before writing the implementation, we add validation to the stubs, to verify that the user ID appears in the response and that there is a logout link. Running the tests at this point reveals that we have a failing test and that we need to write the implementation of what we tested.

Conclusions

In this article, we have walked through the VSTS Web Testing functionality. We demonstrated how to create a test project that includes a Web test case that was generated by recording the activity against the target website. In so doing, we saw how VSTS Web Testing is very easy to set up, and how a significant percentage of testing is supported without ever having to write any code. This is a significant feature that should compel teams to begin testing early and often within the development cycle, not just waiting until QA engineers obtain access to the product.

VSTS Web Testing doesn't stop with recording. There are many possibilities for extending the recorded tests. The ability to generate test code makes it easy to move to coded tests when special customization is required. The code is simple enough that many developers may choose to rely on code rather than a mouse-oriented UI for creating Web test cases. Regardless, extending VSTS Web Testing is simple, providing an excellent platform for additional functionality to be added.

© Microsoft Corporation. All rights reserved.