Microsoft Visual Studio Team Edition for Software Testers

The many test types and how to manage the many tests you generate

Microsoft Corporation

October 2005

Applies to:
   Microsoft Visual Studio .NET
   Microsoft Visual Studio Team Edition

Summary: This article looks at Web and Load Testing with Microsoft Visual Studio Team Edition for Software Testers. (8 printed pages)

Note   This content originally appeared in the January 2005 edition of .NET Developer's Journal. It is reproduced here with permission of the publisher.

Contents

Introduction
An Example
Visual Studio Team System
In This Article
Conclusion

Introduction

A common problem faced by companies looking to undertake software development is determining which tools to incorporate into their process. On the surface it sounds like an easy, perhaps even fun, task: to go shopping for cool new software that is going to make the project that much more productive.

The challenge of selecting that tool, however, goes a little deeper than a quick visit to your favorite software Web site. It quickly becomes a question of how well those tools will integrate with other tools used on a project.

An Example

Take, for example, the case of a source code control system (what's also known these days as a configuration management system). How well will the tool you select tie-in with your testing tools, requirements management system, issue and defect tracking database, work item tracking engine, and build process? Does it also integrate with a reporting system that gives details about test run results, code churn, and defect trends? What about combined reporting, such as a "Test Effectiveness" report (source code exercised as a result of tests executed), "Defects per Lines of Code," and "Requirements Validated by Tests?" And what about defects that are shown as resolved when the test that verifies that defect is, in fact, failing? These are all valuable reports, and the integrated ones aren't available unless your tools play together nicely.

What many companies end up doing when faced with this dilemma is to write layers in between, to force the necessary integration. The problem is that this becomes an entirely new development project as developers write code to create the tools they will then use on their main development project.

Visual Studio Team System

Along comes Microsoft Visual Studio Team System (VSTS), which answers the question, "Which tools should we use?" It also helps solve the challenge of tool integration. VSTS does this by providing source code control, work item tracking, testing tools, developer tools, and architecting tools. They are all designed to work together in a team setting. In addition, Team System opens its architecture to third-party tool developers, allowing them easy integration into an existing framework. The net result is a group of tools that work together seamlessly, with reports showing cross-tool details, such as the ones mentioned earlier.

In This Article

Although we'd love to go into great detail about how wonderful VSTS is, we have a limited space to tell you about a portion of the product. We have chosen to talk about the various test types and how you can manage the large number of tests you will soon generate as you put these tests to use.

It should be noted that although we will look specifically at the Web and Load testing types, there are many other types that come with VSTS (and even more created by third-party testing tool vendors).

The Need for Web and Load Testing

Here's the scenario: You have created a new Web site that's truly a work of art. The source code is some of the best you have written, the HTML is top notch, the artwork on the site is polished and hip, and your writer's content hits your target audience right between the eyes. Your site goes live and it's a big hit, just as you knew it would be. But wait. There are problems when your site starts getting heavy traffic. Worse yet, the problems occur before you have reached the predicted peak hours. Good grief! It's a disaster as you scramble to track down the problem and redeploy your Web site before too many people visit and decide never to return because the experience was too painful.

This problem can be avoided by taking the time to create some tests to exercise your site and simulate user load. The best part is that creating tests doesn't take a huge amount of effort.

Web Test Type

The Web test type that comes with VSTS allows you to record interactions with your Web site. This is done by first creating a new project type in Visual Studio called a Test Project and then adding a Web test. The moment the Web test is added to your project, an instance of Internet Explorer is launched. This allows you to start recording your actions.

In Internet Explorer, a pane appears on the left side of the window. This pane tracks your progress as you surf your Web site, from tracking simple links to recording the posting of complex forms, as shown in Figure 1. Once you have completed your interaction and touched the areas of your site that need to be tested, saving your results takes you back into Visual Studio, where you can see the results of your interaction.

Figure 1. Web Test Pane

After saving your results into Visual Studio, you are then able to step through your recorded Web test, set check points to verify that your site is behaving as expected, and tie your test to a database so that entries on a Web form can be parameterized and allow a data-driven approach to testing. One of the best parts is that you can also render your recorded Web test into source code: Visual Basic, C++, or C#.

The Web test is included with your other test types and can be part of any test run you create, mixing it in with the other test types provided.

Load Test Type

The Load test type is a special kind of test in that it does not have any source code associated with it. It's basically a container of other tests. So, for example, if you want to stress an application for which you have written a number of unit tests, you can create a load test and select any of your coded test types, including Web tests.

Creating a Load test is the same as creating any new test in Visual Studio Team System. You may go to the new top-level Test menu and select Create New Test, or you can right-click on a Test Project and select Add New Test. After selecting the Load test from your varied list of test types, the Load Test Wizard walks you through the configuration of your test, as shown in Figure 2. It is in this wizard that one or more Web tests are selected and the distribution of the execution of those tests is set. In addition, this wizard allows the test author to select which browsers and bandwidths to simulate during the test run. This is also where the user specifies which machines to watch during the execution of the test (for example, Database server, Web server, and any other machine for which you care to gather statistics). Finally, this wizard provides a way to control how to step up the test (have 1,000 simulated users for the entire test or step up the number of users over a given period of time, for example), as well as the total duration of the test.

Figure 2. Load Test Wizard

When the test is executed, it appears in the Test Results window and shows the user that the test is in progress. At this point, the test can run to completion without any user interaction. What is more fun, however, is displaying the test details so that the progress of the load test can be observed, as shown in Figure 3. It is in this view that the machines specified in the wizard can be observed for any anomalies. Although the common counters are shown to the user by default, one can also select other measured parameters and place them on the real-time chart.

Figure 3. Load Test Details

If a problem is identified, the person running the tests can create a bug and assign it to a developer with the results automatically attached to the problem report. When the developer reviews the bug and opens the attached test results, he or she can see exactly what the tester saw when the test was originally executed.

Test Management

Over the course of the life cycle of most projects, test teams typically end up with a large number of test cases. On some teams, a variety of test case types are shared across a diverse set of team members with different roles and different geographical locations. Whether the pain comes from a large number of test cases or a diverse set of team members (or both), managing test cases can become both complex and time consuming.

The Test Manager Window was specifically introduced to help facilitate the organization and use of a large number of test cases. The Test Manager displays all of the test cases loaded in Microsoft Visual Studio Team Edition for Software Testers. As shown in Figure 4, using Test Manager with Test Lists lets you select a set of test cases just by selecting the Test List.

Figure 4. Selecting the tests to run

Other elements of the Test Manager window include the following:

• Tasks: These shortcuts let you complete common tasks with a single click.
• Group By: This feature lets you group the test cases listed in the left-hand side of the window into any type of group, based on the properties of the test case (in Figure 4, the tests are grouped by class name).
• Information header: Displays the total number of test cases currently selected to execute.

Test Lists

Test Lists are exactly what the name implies: a list of test cases grouped together that can be easily managed as a single artifact. In previous drops, Test Lists were called Categories. Test Lists have the following characteristics:

• Tests can be in more than one List.
• Order in a List is not implied.
• Test Lists can contain other Test Lists to represent a hierarchal structure.

Defining a Test List

To define a new Test List, right-click on the location in the Test Manager tree where you want the new list to be created. For example, to make a root Test List, right-click on the List of Tests node and select Create Test List. The result is the dialog box shown in Figure 5.

Figure 5. Creating a New Test List

Now just fill out the Name and Description fields, click OK, and you have a new Test List. To add tests to the List, just select the test under an existing Test List or other node in the Test Manager Tree and drag, copy, or cut the test case into the new Test List.

Conclusion

Web and Load testing give you powerful tools to test any Web site without writing a single line of code. Load Testing can additionally be used with any type of test to create any stress test scenario you need to verify your product.

In addition to helping you write various test types, Microsoft Visual Studio Team Edition for Software Testers can help you manage your test cases. Test Manager lets you control and organize a large number of tests cases in a quick and easy manner using Test Lists.

Additional features included in Microsoft Visual Studio Team Edition for Software Testers that we did not cover include Remote(Distributed) Execution, Test Results Management, and additional test types including Manual Tests and Ordered Tests.

Microsoft Visual Studio Team Edition for Software Testers is a first-class testing tool integrated with the whole Test System product family. By leveraging Microsoft Visual Studio Team Edition for Software Testers, your test team is able to help provide an essential part of the overall software health picture of your product development project.

About the authors

Jason Anderson is a program manager on Microsoft's Enterprise Development and Test tools team. Before rejoining Microsoft in 2003, Jason was a senior software design engineer in IBM's Applied Web Services Design Center, focusing on distributed technologies and the next generation of IBM products. While at IBM, Jason co-authored WebSphere and .NET Coexistence, a book detailing the integration details of building software on these two platforms. Prior to working at IBM, Jason was the CTO of Agital Software Corporation. Before Agital, he worked in a variety of product teams at Microsoft, including Internet Explorer, MS's Java Virtual Machine, the Common Language Runtime, and as an architect for Microsoft's XML Services group.
Tom Arnold is a program manager in the Visual Studio Team System group working on the new Visual Studio Team System testing tools which will be shipping in 2005. Tom has worked with and designed software testing tools for the past 14 years in the software industry, has written two books on test automation, and regularly speaks on the topic at software testing and development conferences.

© Microsoft Corporation. All rights reserved.