How to: Configure and Run Scheduled Tests After Building Your Application
You can run tests after a build has completed to evaluate the quality of your build. These tests are often called build verification tests (BVTs) or smoke tests. These tests typically consist of a broad suite of tests that are used to verify key areas of an application in a particular build. A build is considered a success if all the tests in the BVT have passed.
You can use one or more types of automated tests as part of your build verification test. You can run the following types of tests:
Unit tests
Coded UI tests
Web performance tests
Load tests
Generic tests
Creating BVTs uses Visual Studio, Team Foundation version control, and Team Foundation Build. First, you will check in the automated tests that you want to use for your BVT to source control by using Team Foundation version control. You then add tests to your build definition. Finally, you use Team Foundation Build to queue a build which will then run your tests if the build is successful.
Note
The client computer must have Team Explorer installed, and your Visual Studio user session must be connected to a Team Foundation Server computer. For information about how to connect to Team Foundation Server, see How to: Connect to a Team Project in Team Foundation Server.
This topic describes all the procedures that are required to create and run build verification tests:
Check In Your Build Verification Tests
Create a Build Definition
Add the computers or user accounts for build and test agents to the TeamTestControllerUsers group
Set Up Your Agent to Run Coded UI Tests
Run the BVT Build Definition
Check in Your Build Verification Tests
In this procedure, you check in all the files of your solution.
To check in your build verification tests to source control
Connect to a Team Foundation Server computer. For more information, see How to: Connect to a Team Project in Team Foundation Server.
If your solution is not already in source control, add it to source control. For more information, see Add Files to Version Control.
Choose Team Explorer and then choose Pending Checkins. The Pending Checkin window is displayed.
Check in all the files of your solution. For more information, see Check In Pending Changes.
Note
You might have a specific team process that governs how BVTs are created and managed. For example, the process might require that you verify your build locally before you check in that code and the tests that run on it.
After the check-in operation is finished, a padlock icon is displayed next to each file in Solution Explorer to indicate its Checked In status. For more information, see Identify Version Control Item Status in Solution Explorer.
Your checked-in tests are available to use in a build. You can now create a build definition that contains the tests you want to run in your BVT.
Create a Build Definition
To create the BVT build definition
In Team Explorer, choose your team project.
On the home page of Team Explorer, choose Builds.
The Builds page is displayed.
Choose New Build Definition.
The New Build Definition tab is displayed.
Enter the information for your new build definition**.** For more information, see Create a Build Definition.
Specify the name to associate with the build definition in the Build definition name text box.
(Optional) Add an appropriate description in Description.
Choose Workspace.
The Working folders table includes the source control folder for the team project for which you are creating the new build definitions and a local folder on the build agent. The local folder on the build agent is listed in the Local Folder column. All the workspace paths on the build agent are mapped relative to the default root directory shown.
To copy an existing workspace to the list of working folders, choose Copy Existing Workspace to open the Select a Workspace dialog box.
The workspace you select is normalized to the common root directory on the build agent, $(SourceDir). SourceDir is an environment variable that expands to $(BuildDir)\Sources.
You can also choose an empty table cell in the Source Control Folder, and then choose the ellipses (…) to browse to a source control folder to add as a working folder. The source control folder you select is also normalized to the common root directory on the build agent.
Choose Build Defaults.
On the Build Defaults pane, you can select build controller, if one exists, from the Build controller drop-down list.
In the Copy build output to the following drop folder type the UNC path, such as (\\server\share) location. The built binaries and log files will be located in this folder as soon as the build finishes. For more information about how to set up a drop folder, see Set Up Drop Folders.
If you plan to run coded UI tests, Web performance tests, or load tests as part of your build definition, you must use the output from the build that is in this location to launch or install your application. To automatically install your application after the build completes and before the tests are run, you can either use the lab default template that can deploy your application to an SCVMM environment or you can modify this lab default template to deploy your application to a standard environment.
Note
Before you complete this step, you must have created a public folder where the TFSService account has full rights. For more information about Team Foundation service accounts, see View Team Foundation Server Services.
Choose Process. In the Process pane, choose Show Details.
In the Required section, choose the ellipsis in the second column next to Items to Build.
The Items to Build dialog box is displayed.
Choose Add. Then locate the solution or project you want to build in the version control tree, and then choose OK.
On the Items to Build dialog box, choose OK.
To specify the search pattern for locating test assemblies, choose Test Assembly Filespec. Type your search string. For example, **\*test*.dll if your DLLs all contain the word "test" in their names.
Note
This search string will search recursively through directories looking for any DLLs that match *test*.dll in the Binaries directory. For more information, see Define Your Build Process.
(Optional) To select a test settings file to use when you run the tests, open Automated Tests, open Test Assembly and choose TestSettings File and then choose the ellipses (…).
The Browse dialog box is displayed. Locate the .test settings file that contains the test settings that you want to use, and then choose OK.
If your test setting file uses a test controller and test agents, see the following procedure: Add user accounts or computers for build and test agents to the TeamTestControllerUsers group.
If you are running coded UI tests, see the following procedure to set up your agents based on your test settings file: Set Up Your Agents to Run Coded UI Tests.
Note
You can use the Agent Settings in the Advanced section to select a specific agent to use. For example, if you are running coded UI tests and must select an agent that is running as an interactive process, you can select it here.
For more information about test settings files, see Specifying Test Settings for Visual Studio Tests.
(Optional) To run load tests as part of the build process, you must set up a load test results repository and configure your test controller specified in your test settings to use that repository. For more information, see How to: Select a Load Test Results Repository.
To save your new build definition, choose Save.
Your new build definition appears in the Team Explorer window under the Builds folder.
Add the user accounts or computers for build and test agents to the TeamTestControllerUsers group
If your test settings file that you added to your build definition uses a test controller and test agents, you must add the computers for any build or test agents used to the TeamTestControllerUsers security group on the test controller computer. For example, if you want to run coded UI tests as part of your build process, you must add these computers to this security group.
Warning
If your build agents or test agents are set up to use a different user and not the Network Service account, you must make sure that this domain user account is added to the TeamTestControllerUsers group instead.
To add the users or computers for build and test agents to the TeamTestControllerUsers group
From the test controller computer, click Start, choose Control Panel, choose Administrative Tools, and then choose Computer Management.
The Computer Management dialog box is displayed.
Open Local Users and Groups and then choose Groups.
The groups are displayed.
To add the users or computers, right-click TeamTestControllerUsers and point to Add to Group.
The TeamTestControllerUsers Properties dialog box is displayed.
Look at the Members list to see whether the domain user account or computers are already added. If they are not, choose Add.
The Select Users, Computers, or Groups dialog box is displayed.
By default, only the users or groups are shown. To add computers, choose Object Types, select Computers and then choose OK.
To add a computer, type the name of the computer in Enter the object names to select, and then choose OK.
Warning
You must add both the computer where the build agent is running and the computer for your test agent.
To add a domain user account, verify the location is correct, type the name of the user account in Enter the object names to select, and then choose OK.
Important
Repeat this step to add all the user accounts that you require.
To apply your changes, choose OK.
Set up your agents to run coded UI tests
If you want to run coded UI tests as part of your scheduled tests after building your application, you must do one of the following:
Use a test settings file that specifies a test controller and the roles for your application to run your tests. Create these test settings using Visual Studio. For any one of the test agents assigned to the role that runs your tests in the test settings, you must follow the steps in this procedure to set up the test agent to run as a process instead of a service. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data and Installing and Configuring Test Agents and Test Controllers.
Use a test settings file that does not specify a test controller. If you do this, then you must set up your build agent service to be able to interact with the desktop. Select the property for the service to Allow the service to interact with the desktop. This enables the build agent to run the coded UI test.
Warning
If you are running a coded UI test that starts a browser, the service account for the build service is used to start that browser. This service account must be the same as the user account that is the active user on this computer. If it is not the same user account, the browser will not start.
To set up your test agents to run coded UI tests
- To set up your test agents to run coded UI tests, follow the steps in this topic: How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.
To run the BVT using Team Build
To run the BVT using Team Build
In Team Explorer, choose your team project.
Right-click Builds and then choose Queue New Build.
(Optional) Change the build location and directory.
Choose Queue.
The build starts and the Build Explorer dialog box is displayed.
When the build has completed, choose Completed to see the details.
To view the details of the build, double-click the build in the list.
A new tab is displayed with the build information. You can view the status of the test run.
To view the test result details, choose the arrow to open the test and then choose View Test Results.
Select a folder to store your test results locally.
The test results are now displayed in the Test Results window.
For more information, see Building the Application.
See Also
Concepts
Running Automated Tests in Microsoft Test Manager
Testing Performance and Stress Using Visual Studio Web Performance and Load Tests