Unit Testing and Generating Source Code for Unit Test Frameworks Using Visual Studio 2005 Team System

Scott Dockendorf
Telligent Systems, Inc.

September 2005

Applies to:
   Microsoft Visual Studio 2005 Team System Beta 2
      Team Architect & Team Test Editions
   Microsoft Visual C# 2005

Summary: Scott takes a detailed look at the foundation of automated unit testing, and the code-generation engine included in the Unit Testing Framework provided by Microsoft Visual Studio 2005 Team System. (20 printed pages)

Contents

Introduction
Rethinking Unit Testing
Enter Automated Unit Testing
Why Generate Code?
Let's Generate Some Code!
What Was Generated?
After The Generation: What Do I Do Now?
Regenerating Unit Test Code
Automated Unit Testing Recommendations
Conclusion

Introduction

As businesses evolve through innovation and growth, so do the systems that run them. These systems will continue to increase in complexity as businesses evolve, innovate, and integrate with partners, customers, and vendors.

This complexity is forcing IT leaders to ensure quality during the development process—that is, before implementation. One approach is to challenge your development staff to reduce the amount of defects entered into the QA cycle, by rigorously executing automated unit tests against your custom code. Enforcing automated unit tests in the development cycle provides team members with easy-to-use and self-documenting examples of how to consume your custom code.

One of the challenges to committing to the use of structured, automated unit tests is the sheer amount of code needed to accomplish these tasks. ("That's a lot of code needed to test your code!") The concept of code generation, simply defined as "software that creates software," is finding its way into corporate IT development shops more and more with each fiscal quarter. Some people believe that code generation helps reduce your "time to market" strategy, helps enforce internal standards/conventions, and evolves your development process.

Microsoft recognized this need, and included a feature-rich code generation engine with its next-generation development studio, Visual Studio 2005 Team System (VSTS). This article provides a step-by-step guide to unit test code generation and an insight into how this can be used on a routine basis.

Rethinking Unit Testing

Consider this situation: You're assigned to build the next-generation system for your company, and you are a member of a large development team. You are the UI developer, assigned to crank out as much Microsoft ASP.NET/Microsoft WinForms as you possibly can. You rely on the "middle-tier" team to complete their middle-tier components, designed to perform database CRUD (Create-Retrieve-Update-Delete) as well as the business rules associated to each entity in the system.

After weeks of UI development, your forms are complete and you've received word that the middle-tier developers are ready to deliver their class libraries to you. Table 1 provides a sample dialog that many of us have faced at one point in our careers.

Table 1. Sample Dialog Between UI Developer and Middle-Tier Developers

After wondering how you landed on such a fun project, you mumble a few choice words and decide to check out Middle-Tier's unit test suite. Upon digging into the code, you notice that the form has two unlabeled text boxes and three buttons labeled button1, button2, and button3 (if you're lucky, they are aligned on the form). Next, after interrogating on the events associated to these buttons, you realize that none of the code is commented, and data variables are named x, y, and z. Luckily, you noticed that button1 and button2 perform the object's Save() method, and button3 performs the Delete() method. Upon execution, you receive a slew of System.Exception errors, because you were missing various configuration settings.

This is definitely an extreme case, and I hope there aren't many shops out there that go through this, but let's take a look at the problems encountered with the "unit tests" in this scenario:

• This form of unit test code is not structured: the code is thrown into button click events, and is hard to decipher.
• This form of unit test code is not well documented.
• This form of unit testing is not based on data "known" to be good or bad—it's all up to what was entered into those unlabeled text boxes.
  • Unit Test code is not automatically repeatable, based on code entered.
  • Unit test code coverage is unknown—data dictates how much of your code was actually tested.
• Implementation details are not easily communicated between team members.

Enter Automated Unit Testing

The xUnit framework was introduced as a core concept of eXtreme Programming in 1998. It introduced an efficient mechanism to help developers add structured, efficient, automated unit testing into their normal development activities. Since then, this framework evolved into the de facto standard for automated unit testing frameworks.

Making the Case for Automated Unit Testing

Simply put, automated unit tests are:

• Structured.
• Self-documenting.
• Automatic and repeatable.
• Based on known data.
• Designed to test positive and negative actions.
• Ideal for testing implementation across different machines.
• Examples of configuration, implementation, and execution.

xUnit Framework Elements

Table 2 breaks down the basic concepts of the xUnit framework, and their Visual Studio 2005 Team System Unit Testing Framework equivalents.

Table 2. Corresponding xUnit Framework and VSTS Unit Testing Framework Concepts

Test Fixture Example

Consider the following class diagram for the BankAccount class, and a sample test fixture (BankAccountTests.cs).

Figure 1. BankAccount class

Sample Test Fixture: BankAccountTests.cs

using BankAccountDemo.Business;
using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;

namespace BankAccountDemo.Business.Tests
{
    [TestClass()]
    public class BankAccountTest
    {

        [TestInitialize()]
        public void Initialize()    {
        }

        [TestCleanup()]
        public void Cleanup()   {
        }

        [TestMethod()]
        public void ConstructorTest()   {
            float currentBalance = 500; 
            BankAccount target = new BankAccount(currentBalance);
            Assert.AreEqual(currentBalance, target.CurrentBalance, 
                "Balances are not equal upon creation");
        }

        [TestMethod()]
        public void DepositMoneyTest()  {
            float currentBalance = 500; 
            BankAccount target = new BankAccount(currentBalance);
            float depositAmount = 10; 
            target.DepositMoney(depositAmount);
            Assert.IsTrue( (currentBalance + depositAmount) >  
                            target.CurrentBalance, 
               "Deposit not applied correctly");
        }

        [TestMethod()]
        public void MakePaymentTest()   {
            float currentBalance = 500; 
            BankAccount target = new BankAccount(currentBalance);
            float paymentAmount = 250; 
            target.MakePayment(paymentAmount);
            Assert.IsTrue(currentBalance - paymentAmount == 
              target.CurrentBalance, 
                "Payment not applied correctly");
        }
    }
}

Primary Unit Test Concept == Assertions

The primary concept used for this form of unit testing is that automated unit tests are based on "assertions," which can be defined as "the truth, or what you believe to be the truth." From a logic standpoint, consider this statement, "when I do {x}, I expect {y} as a result."

This translates to code very easily, using any of the three "assertion" classes available in the Microsoft.VisualStudio.QualityTools.UnitTesting.Framework namespace: Assert, StringAssert, and CollectionAssert. The primary class, Assert, provides assertions used to test fundamental conditional statements. The StringAssert class has customized assertions helpful when working with string variables. Likewise, the CollectionAssert class includes assertion methods helpful when working with collections of objects.

Table 3 shows the assertions available for the current release of the Unit Testing Framework.

Table 3. VSTS Unit Testing Framework Assertions

What Do You Test?

These automated unit tests are very low-level in nature. They are intended to test your objects down to the constructor, method call, and even down to a property on your object.

This subject of "public vs. private" offers much debate in the unit testing circles. Many believe that unit tests should only test the public interface of an object. Others feel that every call should be tested—including internal, private methods. VSTS supports both levels of unit testing. VSTS supports private testing through the use of private accessors, or wrapper classes that provide the ability to generate unit tests based on "private" methods and properties.

What Runs These Automated Unit Tests?

As mentioned earlier, the xUnit framework defines concept of "test runner" as the application responsible for (a) executing unit tests, and (b) reporting on the test results. For the purpose of this article, the Unit Testing engine included with Visual Studio 2005 Team System (VSTS) serves as our "test runner". Figure 2 represents the execution results for the BankAccountTests.cs class.

Figure 2. Test Results pane: Unit test execution results

Microsoft Visual Studio 2005 dynamically populates this view using the code model of the source project. It dynamically discovers information about the test suite, based on custom attributes in the source code. Table 4 presents the most common unit test attributes (and the order of execution).

Table 4. Common Unit Test Attributes

What Types of Tests Do I Write?

Rarely will you have a one-to-one relationship between a method and its associated tests. Writing automated unit testing requires developers to "think outside the box" and know everything about your object—how it will be consumed, used, disposed of, and how it reacts positively, negatively, and inconclusively under any circumstance.

For example, consider a typical object meant to perform CRUD (create, retrieve, update, delete) functionality for Customer entries in a database. For the Load() method of this object, you will want to write tests for the following scenarios:

• Constructor Test—To make sure your object loads properly, with the correct information.
• PositiveLoadScalarTest—To test the successful Load of a Customer that exists in the database.
• NegativeLoadScalarTest—To test the unsuccessful Load of a Customer—that is, one that does not exist in the database.
• PositiveLoadTest—To test the successful load of Customers, based on known data.
• NegativeLoadTest—To test the unsuccessful load of Customers that do not exist in the database.
• NegativeValidationTest—To make sure your validation logic is working correctly.

These are just some of the many uses for an automated unit testing suite. I once heard about a shop using unit tests to check for known security attacks on their components. In a larger-picture perspective, unit tests should definitively certify your components for normal consumption. Having a well-rounded set of test gives your team confidence that you've completed exactly what you set out to do: write efficient software. Whatever it takes to provide this confidence—those are the tests you need to write.

Why Generate Code?

After reading the list above, you're probably remembering monolithic objects from previous projects and thinking, "If I were to do that with THOSE objects, that would be A LOT of code to write!" Consider the fact that developers still write "unit testing" code—it just takes on different forms (such as the WinForms example mentioned earlier). In addition, the benefit of having a self-documented, reusable example of implementation outweighs the fear of generating more code. Lastly, spending more cycles designing thorough unit tests has been shown to reduce defects during the quality assurance cycle.

As mentioned earlier, code generation is the process of "software creating software." It is ideal for creating code based on repeatable processes. For example, some of the good cases to use code generation include: scripting data, creating objects that represent entities and their persistence in a repository (database CRUD), or creating UI controls geared towards data maintenance. Some benefits of using code generation include:

• Saving Time—Why spend hours/days/weeks creating something that you can create within seconds/minutes?
• Enforcing Standards/Conventions—Nothing is better for enforcing your standards and naming conventions than to remove the human element of development and rely on a repeatable process based on "your" rules.
• Ability to Test Private Methods—As mentioned earlier in the article, the Unit Testing Framework provides the ability to test private methods using the "private accessor" classes. The code generation engine creates all the "plumbing code" related to these accessor classes.
• Gaining Knowledge of Existing Components—Researching another developer's components? Generating code based on these components might offer a quick example of implementation and the object's interfaces. Also, developers who design and "stub out" their object's public interfaces (by using the VS 2005 Class Designer, for example) before coding will greatly benefit from the code generation engine.

As you can expect, automated unit tests fall into the category of "good code generation candidates." Wouldn't it be nice to point something at already existing components and generate the initial code for these automated unit tests? And not just the unit test framework stubs, but generating an example of implementation, built around an object's public interface? Future owners of Visual Studio 2005 Team System will have this and more!

Let's Generate Some Code!

In this example, we're going to generate code for the BankAccount class presented earlier in the article. This portion of the article is meant to step you through the code generation process, as well as highlight the features provided and the benefits of using the Unit Testing engine from VSTS.

Step #1: Create Your Implementation Code

First, let's create a class library project that will serve as the business layer for your application.

To create this library in VS 2005:

1. Launch Visual Studio 2005 Beta 2.
2. Click the File menu, New, and then Project.
3. Select your language of choice, Windows, and select the Class Library project template
4. Set the Name and Solution Name to BankAccountDemo.Business, select a location, and click OK to create the class library.

Once VS 2005 creates the class, our next task is to create the BankAccount class designed for your project. To do so:

1. In the Solution Explorer, right-click, and click Delete to remove the file from the project and delete it off the disk.
2. Right-click the BankAccountDemo.Business project, and click Add, and then Class.
3. Choose the filename BankAccount.cs, and click Add to create the class file.
4. Make the following code changes to the BankAccount.cs file.

using System;
using System.Collections.Generic;
using System.Text;

namespace BankAccountDemo.Business
{
    public class BankAccount
    {
        // Properties
        private float _currentBalance;

        public float CurrentBalance
        {
            get { return _currentBalance; }
        }

        // Constructors
        public BankAccount(float initialBalance)
        {
            this._currentBalance = initialBalance;
        }

        // Methods
        public void DepositMoney(float depositAmount)
        {
            this._currentBalance += depositAmount;
        }

        public void MakePayment(float paymentAmount)
        {
            this._currentBalance -= paymentAmount;
        }
    }

}

Step #2: Generate Your Initial Unit Test Code

With the Unit Testing engine built into Visual Studio 2005 Team System, generating code is easier than ever. In addition to generating the unit test structure, it will generate instance-specific information such as object creation, typed parameters, and method execution.

VS 2005 provides the ability to generate unit test code at any class structural level—namespace, class, method, property, constructor, and so on. You can do so by right-clicking these code elements and clicking Generate test(s) (Figure 3).

Figure 3. Generate test(s) method

Thus, to begin the code generation process, perform the following step:

• Right-click the class name, BankAccount, and click Create Tests.

Now, you should be presented with the Generate Unit Tests dialog box (shown in Figure 4). This dialog box and its components provide the ability to customize the code generated during this process. Let's examine each of these elements.

Figure 4. Generate Unit Tests dialog box (Click on the image for a larger picture)

The Current selection: tree view allows you to navigate through your custom class and its elements. VS 2005 uses reflection to populate this tree view, and will automatically select the components, based on where you right-clicked and clicked Create Tests. In Figure 3, since I did so at the Class-level, the dialog box automatically selected all of the class elements for code generation. If you select generation at the individual levels (that is, constructor, property, or method), only those elements would have been selected.

The Filter option, located in the upper right-hand corner, provides the ability to modify the results shown in the tree view (Figure 5), including displaying non-public items, base types, and "My code only." This is beneficial if you working with a large solution, or if you feel that displaying your private, internal structures clutters up the selection window.

Figure 5. Filtering selection results

Next is the Output project list box, located under the Current selection: tree view. This list box allows you to select the destination project for your generated text fixtures (Figure 6). If your solution contains a previously created test project, the test project will be included for selection. Since this is our first visit to the dialog box, the Create a new … Test Project options are available for selection.

Figure 6. Output project selection (Click on the image for a larger picture)

To continue with our process:

• Select Create a new {0} Test Project, based on your language of choice.

Lastly, the dialog box provides the ability to customize the code generation process through the Settings button, located on the bottom left-hand corner. Clicking this button loads the Test Generation Settings dialog box, as shown in Figure 7.

Figure 7. Test Generation Settings dialog box

This dialog allows you to make the following changes:

• Change naming conventions used to generate names for the file, class (test fixture), and method (test).

• Turn on/off the ability to mark all test results Inconclusive by default. Selecting this option will include the following placeholder statement in each generated Test() method.

  Assert.Inconclusive("TODO: Implement code to verify target");

• Turn on/off the ability to enable generation warnings—that is, reporting if any warnings occur during the code generation process.

• Globally qualify all types. This setting tells the code generation engine to add a global qualifier (global:: in Microsoft Visual C# 2005) to variable declaration. Use this when you have like-named objects that reside in multiple namespaces. If you do not, the code generation engine will create logic to create the object, but the compiler will not be able to determine which class to create, and will error.

• Enable/disable the ability to generate a test for items that already have tests. We will discuss the topic of subsequent code generation attempts below.

• Enable/disable documentation comments. This allows you to disable the creation of XML documentation above each Test() method

To complete our configuration and generate our unit test code (and more):

1. Click the OK button to begin the code generation process.
2. Enter the name BankAccountDemo.Business.Test as the name for the new project, and click the Create button to complete the process.

VS 2005 will display a progress bar, providing status during the code generation process. Within seconds, the process will complete and you should be looking at a class called BankAccountTest.cs.

What Was Generated?

Before we review the test fixture specifically, let's take a look at what was created during the code generation process.

First, it created the Test Class Library project BankAccountDemo.Business.Test. Notice how the project contains references your implementation class, BankAccountDemo.Business (where you generated your code from), and the Microsoft.VisualStudio.QualityTools.UnitTestFramework class library. While inspecting the contents of this class, you will notice the following files:

• AuthoringTests.txt—This is some informational content, defining how to work with unit tests (Opening, Viewing, Running, Viewing results, Changing how tests are run), and definitions of the different test types included with VSTS.
• ManualTest1.mht—This is the manual testing harness used in VSTS to execute your tests and report on the results. Manual tests are an additional type of tests supported by VSTS. For more information, checkout the MSDN library for the "Manual Tests" topic.
• UnitTest1.cs—This is a reference class that simply provides a base unit test, including definitions for the TestClass, TestInitialize, TestCleanup and TestMethod.
• BankAccountTest.cs—This is the generated unit test code specific to your assembly. Let's take a closer look at this code, the most important part of the code generation process.

The class generated by the Unit Testing engine includes the following components:

• Using/imports statements for the referenced assemblies.
• TestClass() definition for the class that contains the test (BankAccountTestFixture).
• A private accessor and public property for TestContext. This is used by the unit test runner (that is, VSTS Unit Test Framework) to provide information about, and functionality for, the current test run.
• TestInitialize() and TestCleanup() methods. These are commonly used to acquire and release any objects needed for your tests.
• TestMethod() for each method selected.

Let's take a closer look at DepositMoneyTest(), responsible for ensuring that the current balance reflects the original plus the deposited amount.

/// <summary>
///A test case for DepositMoney (float)
///</summary>
[TestMethod()]
public void DepositMoneyTest()
{
float initialBalance = 0; // TODO: Initialize to an appropriate value
BankAccount target = new BankAccount(initialBalance);
float depositAmt = 0; // TODO: Initialize to an appropriate value
target.DepositMoney(depositAmt);
Assert.Inconclusive("A method that does not return a value" +
   "cannot be verified.");
}

Notice how the generation engine did much more than just creating a stub TestMethod() object. It created sample unit tests tailored to your interface, including:

• Assignment and construction of the BankAccount object (the object subject of the test).
• Creation and default assignment of local variables that represent parameters needed for the method/constructor as the subject of this test.
  • TODO comments, reminding the developer to assign the parameter variables appropriately.
• If the test is based on a source object method call, the generated code will contain the call to that method, with local variables for the parameters.
• Initial Assert() method call, based on the return values of the method.
• Assert.Inconclusive() method call, as a reminder to complete the code for the test. Inconclusive tests will show as failures in the Test Results dialog box.

After The Generation: What Do I Do Now?

The benefits of code generation are often realized by considering what you did not have to do to accomplish the same thing. In our example, we didn't have to:

• Create the unit test project.
• Set the project references.
• Add the appropriate test class(es).
• Build out the skeleton Unit Test Framework classes and attributes.
• Create individual test methods.
• Create interface-specific logic.

Because the code generation process created sample unit tests customized to our object's interface, we should be close to completion of our initial test. Most likely, all that is needed is to "fill in the blanks" and complete the assertion(s), by assigning "known data values" to your property variables and create appropriate Assert() methods. Obviously, this will not be the case for all tests, especially complex ones with multiple assertions.

Within just a few seconds (and with relatively few keystrokes), you should be able to convert the generated unit test code into these actual tests.

For example, consider that we started with the following.

[TestMethod()]
public void DepositMoneyTest()
{
float initialBalance = 0; // TODO: Initialize to an appropriate value
BankAccount target = new BankAccount(initialBalance);
float depositAmt = 0; // TODO: Initialize to an appropriate value
target.DepositMoney(depositAmt);
Assert.Inconclusive("A method that does not return a value " + 
  "cannot be verified.");
}

We're able to complete the test with relative ease and limited keystrokes (changes bolded).

[TestMethod()]
public void DepositMoneyTest()  {
float currentBalance = 500; 
BankAccount target = new BankAccount(currentBalance);
float depositAmt = 10; 
target.DepositMoney(depositAmt);
Assert.AreEqual(currentBalance + depositAmt, target.CurrentBalance, 
                  "Deposit Test: Deposit not applied correctly");
}

Regenerating Unit Test Code

The good news is that the code generation process will not overwrite your previously generated (and modified) unit tests. With the Beta 2 release of Visual Studio 2005 Team System, the code generation options provide a checkbox to enable/disable the creation of already existing tests. If this is selected, and if the process finds an already existing test method with the same name, the process will leave that test method alone, and create subsequent tests, appending a number to the end of the method name. This commonly happens when using overloaded methods or constructors in your object, or when you click the Generate button without deselecting your existing tests.

Automated Unit Testing Recommendations

While this section could be an article in itself, here are some essential recommendations to follow when creating your unit tests.

• Design your unit tests independent of each other, where they can run on their own (since they can be selected or deselected by the testing UI at will).
• Don't just test the positive. Make sure your code responds to any and all scenarios, including when the unexpected happens (resources are not available, database is read only, and so on).
• Put on your QA hat, and think like a tester, not just a developer. The time you take to design your unit tests will help reduce the time spent resolving defects later. Focus on the minute details of your objects: How is data transferred between them? Who consumes them? How easy can you break the object? What happens if I "do this"?
• Think outside "your own" box. Carve out time to brainstorm as many tests as you can. When you're complete, take a step back and check for anything that you might have missed. Request feedback from team members—for example, what other types of tests do they create? Others might provide a perspective that is hard for developers intimate with their own code.
• Code Coverage. Use the VSTS Code Coverage instrumentation to provide information about how much of your code actually executed during each test run (number of lines of code, percentage of all code). If coding is complete and all your tests pass, but code coverage shows a small amount of the logic was executed, were your tests really successful? High code coverage does not necessarily mean that you have a complete set of "tests," but code that is not covered is generally a good candidate for a new test case.
• When building your unit tests, to help educate others about your code:

  • Use a project structure that mirrors the structure of the assembly being tested.

    Each assembly has a related test assembly.

    Each class has a related test class.

    Include each method name in the respective test methods (that is Load() will have test methods of PositiveLoadTest(), NegativeLoadTest(), PositiveScalarLoadTest(), and so on).

  • Use consistent naming conventions, including the names of the object's properties and methods.

• Also, when all else fails, debug. Automated unit testing should help reduce the amount of time you spend in the debugger. However, if testing results and code coverage do not help in providing the reason why your test is failing, don't be afraid to debug your unit tests. Beginning with the Beta 2 release of Visual Studio 2005 Team System, developers can debug their unit testing assemblies using the Debug checked tests option in Test Manager.

Conclusion

Automated unit testing provides a structured, self-documenting, highly portable, and repeatable process to your development cycle. If you find yourself researching existing assemblies, or if your development environment requires your complete design before starting development, consider utilizing the code generation engine built into Microsoft Visual Studio 2005 Team System. The unit test code generation feature of Visual Studio 2005 Team System is sure to save you valuable time, and help enforce your team's development standards and conventions. By generating the foundation for your automated unit testing, including generating test methods with object creation, parameter variables, and base assertion classes, you should be well on your way to adopting automated unit testing in your development methodology.

About the author

As Director of Professional Services for telligent systems, Scott Dockendorf specializes in delivering highly performing and scalable applications in .NET. Scott is passionate about solution architecture, secure development, and helping companies adopt recommended practices through standards and proven methodologies. Scott is an active member of the .NET community, volunteering his time as the Program Director of the North Dallas .NET User Group. He also delivers sessions to local .NET User Group, and is an active volunteer for INETA's International Academic Committee for Texas. You can contact Scott by e-mail at scottd@telligent.com, or through his weblog at https://weblogs.asp.net/scottdockendorf.

© Microsoft Corporation. All rights reserved.