Authors:

Jezz Santos (Microsoft)
Edward Bakker (LogicaCMG)
Rene Schrieken (LogicaCMG)

Contributors:

Mark Nichols (Microsoft)

Reviewers:

David Scruggs (Microsoft)
Attila Hajdrik (Microsoft)

October 2007

Applies To:

.NET Framework 2.0
Visual Studio 2005
Visual Studio SDK 4.0
Guidance Automation Extensions/Toolkit 1.2
Visual Studio 2005 Team Foundation Server 1.0

Summary: Welcome to part 2 of the short series on packaging and building software factories with team foundation build.

Software Factory solutions vary in structure today. Typically, they are composed of multiple project types like the ones identified in this series (e.g. VSX, GAT and DSL packages). In team build scenarios, these project types present unique challenges when automated build processes are applied to them.

Get the reference implementation.

Download the Reference Implementation for this article. You can use this as an example in your own organization. Please read the EULA before using this reference.

Contents

Introduction
Team Build with Software Factories
The Build Process
Automating the Team Build
Automating GAT Package Registration
Automating DSL Package Compilation
Automating Model Validation
Summary
About the Authors

Introduction

Welcome to part 2 of the short series on packaging and building software factories with team foundation build.

In the first part of the series ‘Packaging with Visual Studio 2005’ we saw what project types are commonly used in software factory solutions today (VSX, GAT and DSL), and how to create a combined installation package for deploying such a software factory using either standard VS setup projects and WIX setup projects. In this second part of the series we are going to look at how to automate the development and build of such software factory solutions, and their installers, in a team foundation build environment using Visual Studio Team Foundation Server (TFS).

In normal solution development where source code files represent the primary design-time artifacts, there exist source language compilers to verify whether source code is valid, and these tools compile that source into assemblies for packaging with installers. This compilation is invoked both on a developer’s machine and easily automated on a build server, in team build environments.

However, project types such as: GAT and DSL (common to software factory solutions), utilize ‘intermediate design artifacts’ which require explicit validation and explicit pre-compilation or transformation before other source artifacts and can be generated or compiled and then packaged.  In the case of GAT package projects, the intermediate design artifacts are the package XML manifest file, the recipes and the templates. The verification (and pre-compilation to some extent) is performed by package registration process (recipe). In the case of DSL package projects, the intermediate design artifacts are the DSL definition file (*.dsl) and the text templates (*.tt) files. The verification step is performed by the model validation engine, and the pre-compilation or transformation is performed by any text template transformations.

Invalid intermediate design-time artifacts (e.g. invalid domain models or recipes and templates) in GAT and DSL projects, should have the same meaning as invalid source code (whether its hand crafted or generated). If the intention of a build process is to ensure all checked-in artifacts are valid for building, then it follows that invalid intermediate artifacts in GAT and DSL package projects should carry the same consequence as invalid source code - i.e. it breaks the build.

In very small development environments, verifying, transforming and compiling a software factory solution, containing these types of projects, is typically individually micro-managed by those who develop the solution. They themselves control the development of the projects and enact the various individual micro-validation and micro-transformation processes to ensure that: (a) the design artifacts are valid, (b) all generated source artifacts are synchronized with their design artifacts, and (c) the packages compile before checking in code or compiling installer packages.

However, in larger development teams, with multiple developers, potentially all working on same design artifacts, these manual ‘micro-processes’ are difficult to synchronize reliably across multiple team members regularly and reliably. Therefore, centrally managed build processes are designed to ensure completeness and validity of all artifacts before creating and packaging regular releases.

Clearly then, to ensure all design-time artifacts are valid, the verification, transformation and pre-compilation steps required by GAT and DSL package project design artifacts should be automated by the team build environment. In this way, the build processes can quickly identify which design artifacts are invalid, ensure generated artifacts are is sync, and synchronize those artifacts (design and generated) with all members of the team.

Team Foundation Server (TFS) provides the ideal platform and build environment for defining, hosting, enforcing development policies and executing automated team builds for these types of solutions.

Team Build with Software Factories

In order to automate the build of a software factory solution that contains VSX, GAT and DSL package projects on a TFS build server today, several unexpected issues will be confronted. These are mostly due to differences between the build server environment and the developer environments for these package technologies at compile and debugging time. But also, there are issues of how to manage the pre-compilation and validation steps of intermediate design artifacts of specific to these types of solutions.

Most of the build environment issues are related to tools, build dependencies, hard coded paths to various SDK’s and toolkit resources, and configuration settings that are valid on a developer’s machine but not on a build server (with no pre-knowledge of these package project types). This is particularly prevalent in centrally hosted or strictly managed IT environments.

Most issues encountered are related to DSL development and extensibility components developed against the Visual Studio SDK.

In this article we focus only on team build issues with VSX, GAT and DSL packages specifically and exclusively. This article is not intended as a comprehensive guide to describing the general setup of TFS Team Builds.

Furthermore, this article does not address all issues that could be potentially encountered in setting up a build process for your particular software factory. However, it does aim to identify some of the key common issues and provide information and resolutions that will further help to resolve other classes of issues that may be encountered specific to your software factory solutions.

Team Build Environments

Ideally, as build managers, we would like to minimize any special compensations and configurations required on the build server just to build these types of projects For example, in the case of software factories, installing and configuring SDK’s and toolkits such as: the Visual Studio SDK, the GAX runtime and the DSL Tools. If this was an assumed practice, then all build servers would need to be installed with all SDK’s and toolkits (and all potential versions of them) expected for all current and future software development projects. This is simply not realistic in most software development organizations with managed shared development and build infrastructures. A number of these restrictions are enforced and carefully managed by enterprise infrastructure administrators and are uncompromising. They must be worked around for many development teams.

Therefore, it makes more sense to take the reliance off the particular build infrastructure, and create a specific build environment per solution. In this way, the specific solutions can target specific toolkit versions independently, provide all dependencies and resources, and have little to no impact on an existing shared development and build infrastructures.

The basic approach we take in this article is one that avoids the installation and configuration of any non-standard (out-off-the-box) components (e.g. additional SDK’s, toolkits, etc) specific to the project types contained within any given the solution. The rationale behind this is simply that we don’t assume that in all cases, in all projects, and all organizations that the development team will have full control over the build infrastructure, and would be able to tailor the build infrastructure to suit its own exclusive needs.

It is possible in certain projects or organizations that some of these restrictions may not be strictly enforced and the issues arising from them therefore not encountered. For example, in dedicated development projects where the development team has full control over the build environment, and evolve and maintain it as the development progresses.

Regardless, versioning is still an issue. Some development projects may still need to target multiple versions of their dependent SDK’s and toolkits, and it may not be possible or difficult to install these dependencies side-by-side in the build environment.

The content provided in this article will serve to gain an understanding of the various issues encountered for team build scenarios, their causes and resolutions, and provide information and practical guidance to understand the dependencies that new software factory technologies like the DSL Tools and GAT have for automating the build of software factory solutions today.

Sample Code

This article comes with a Reference Implementation (RI) sample demonstrating how to implement the required changes to an example software factory solution. In addition, an open source community project (www.codeplex.com/sfteambuild) has been created to provide additional resources to supplement the guidance given in this document.

More Information

The remaining sections of this article discuss how to create and customize an automated team build process that is capable of verifying and building software factory solutions (containing VSX, GAT and DSL project types), and their solution installers.

Information about creating and customizing automated builds using TFS can be found on MSDN:

·         Customizing Team Foundation Build

·         MSBUILD Overview

Best practices for using and customizing TFS can also be found in the patterns & practices Team Development with TFS Guide.

The Build Process

In team development scenarios, normally a centralized build process (i.e. ‘Daily Build’, Continuous Integration (CI) etc.) is designed and managed to ensure that regular development changes to solutions from multiple developers in the team are verified across the whole source code base, and then synchronized for next development iteration. For software factory solutions containing VSX, GAT and DSL project types this represents unique challenges not usually encountered in ordinary language developments such as with ordinary project types. This is primarily because VSX, GAT and DSL project types, as Visual Studio extensibility technologies, utilize generation, compilation, registration and design-time tools. Furthermore, GAT and DSL project types introduce intermediate development artifacts that require special ‘pre-processing’ before compiling their source code, and being packaging.

In a normal centralized regular build process, the following conceptual steps occur for each development iteration (omitting steps for: source control labeling, testing, distribution, notification etc):

On Development Machine:

1.     Developer(s) checks-out source artifacts from source repository

2.     Developer(s) adds/modifies source artifacts

3.     Developer(s) compiles source artifacts, and runs tests locally

4.     Developer(s) checks-in source artifacts

On Build Server:

5.     Build Process retrieves latest source artifacts from source repository

6.     Build Process compiles source artifacts

7.     Build Process packages solutions (installers)

8.     Build Process delivers solutions for installation and testing

The difference that compiling software factory package project types (containing GAT and dsl project types) presents in this process are primarily to do with the environment within which the compilation is executed, and the verification/transformation steps of those project types.

To the above process list the additional steps need to be added to support the project types commonly found in software factory solutions:

On the Development Machine:

2a. Developer(s) validates intermediate artifacts (i.e. model definitions, recipes etc)

2b. Developer(s) transforms intermediate artifacts to source code artifacts (i.e. using text templates)

And then:

3a. Developer(s) registers packages with runtime environment.

On the Build Server:

5a. Build Process validates intermediate artifacts (i.e. model definitions, recipes etc)

5b. Build Process transforms intermediate artifacts to source code artifacts (i.e. using text templates)

In the following sections we will explore the differences between the development machines and build server environments that are involved in implementing these additional steps, and introduce techniques to address the various issues faced. Then we will delve into options on how to implement the missing steps (5a-5b) for the various package project types.

Automating the Team Build

The following sections contain information about general infrastructure required to implement team builds on a build server, and the technical details on how to address these issues for software factory solution development.

Solution Structure

The compile and automated build process (for VSX, GAT and DSL project types) requires a number of support files and dependencies (i.e. tools, assemblies, configuration, include files etc) which need to be made available to both the build process on the build server, and at compile and debugging time on the developer’s machines.

The basic approach taken in this article is to provide all the files required to build the solution within the solution itself. These files would include various command line tools for performing operations on the projects in the solution with their supporting dependencies, configuration and include files.

These tools and supporting files would normally be installed and configured by specific versions of required SDK’s and toolkits for the solution (e.g. Visual Studio SDK, GAX/GAT). However, because we are avoiding installation of these toolkits to the build server, special steps have to be taken to provide these files and their configuration to the solution itself.

Since these SDK’s and toolkits are normally installed to a developer’s machine, we have access to all resources to put into our solution.

In order to create the necessary solution structure detailed in the sections below, the following process is enacted.

1.     Install the required SDK’s and toolkits to a clean development machine.

2.     Copy the required resources from the development machine into the solution structure

3.     Modify the resources to reference and operate from the solution structure, rather than from resources installed to the local machine.

The last point is critical here, since these resources need to operate on both the developers’ machine and on the build server. In some cases this is challenging and special workarounds are involved. The rest of the paper is dedicated to itemizing these workarounds.

Supporting Build Resources

For team build projects in TFS, a root project folder is created (‘MyTFSProject’) containing the solution (‘MySoftwareFactory’) and TFS team builds (‘TeamBuildTypes’).

The ‘TeamBuildTypes’ folder contains a sub folder called ‘MyFactoryTeamBuild’ containing the TFS project file (‘TFSBuild.proj’) which is created by the TFS ‘Project Creation Wizard’ (PCW).

The software factory solution folder (‘MySoftwareFactory’) contains the actual Visual Studio solution files and all the projects. In addition, this folder contains an additional folder (‘BuildExtension’) which will contain specific versions of the support files required for this solution.

For example:

The following tables identify the required files to populate the folders defined for this TFS project.

These supporting files are all obtained from installing the Visual Studio SDK (v.4.0) to a developer’s machine.

*Where %VSSDK% is by default: ‘%ProgramFiles%\Visual Studio 2005 SDK\2007.02’

*Where %GAC% is by default: ‘%WINDIR%\assembly’

Solution Dependencies

The various SDK’s and toolkits involved in authoring software factory solutions (e.g. DSL Toolkit, GAX/GAT etc.) have various additional assembly dependencies that are required at development and build time in order to build the solution. These dependencies would also include any 3^rd party components, application blocks, class libraries, or frameworks etc. used by the solution. Common libraries for software factories may include: Enterprise Library (EntLib) or the GAX Extensions Library  (GEL) or Designer Integration Service (DIS) etc.

Typically all dependencies for these SDK’s and toolkits are installed to developer’s machines by the relevant SDK, toolkit installers.

To build in both the developer’s machine and on in the team build these dependencies are copied from a developer’s machine to the ‘ExternalDependencies’ folder in the software factory solution.

The following table lists the specific dependencies of GAX and the DSL Tools.

*Where %VSSDK% is by default: ‘%ProgramFiles%\Visual Studio 2005 SDK\2007.02’

This approach makes the assumption that theses assemblies have few or no references to other configuration (registry settings, configuration files, includes etc) installed by the toolkit on the build server (other than those in the ‘BuildExtension’ folder. Unfortunately, this is not entirely the case in practice. Later sections in this article detail those individual cases and provide the workarounds that enable this approach.

For the solution to use these dependencies, now in the solution, the following changes need to be made to the project files to reference these dependencies in the solution structure rather than on the local machine:

1.     Modify all references (HintPath) in all project files (*.csproj). For example:

Ensure that the path is relative to the location of the project in the solution structure.

    <Reference Include="Microsoft.VisualStudio.TextTemplating,
Version=8.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a,
processorArchitecture=MSIL">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\..\MySoftwareFactory\ExternalAssemblies\Microsoft.VisualStudio.TextTemplating.dll</HintPath>
    </Reference>

2.     Add the following element to the ‘tfsbuild.proj’ file, to ensure the team build knows where to find our external assemblies.

<AdditionalReferencePath Include="$(SolutionRoot)\MySoftwareFactory\ExternalAssemblies" />

3.     Add the following <PropertyGroup> element containing the <SolutionDir> property to the ‘tfsbuild.proj’ file, to allow quick reference in other MSBUILD files to the software factory solution directory.

<PropertyGroup>
  <SolutionDir>$(SolutionRoot)\MySoftwareFactory\</SolutionDir>
</PropertyGroup> 

Developer or Team Build?

As we will see throughout the following sections in this document, some of the build steps that will need to execute will be either optional or different for a team build on the build server, than an individual build on a developer machine. To differentiate the build environments we have to implement a mechanism that detects in which environment the build is being executed. In other words, we need to know if we running a local build that is executed by the developer or a team build running on the build server.

In fact, there are 3 different build environments we need to consider:

·         Visual Studio Build – a build executed by a developer, on their own development machine inside the Visual Studio IDE

·         Team Build – a build executed by TFS (manually or scheduled), on the build.

·         Desktop Build – a build explicitly executed manually, on the development workstation using the command 'msbuild.exe tfsbuild.proj'.

A ‘DesktopBuild’ and a ‘TeamBuild’ are very similar in nature except that ‘DesktopBuild’ does not perform a ‘GetLatest’ function from source repository, will not ‘Label’ the source tree and will not determine the changeset.

When using MSBUILD tasks (as we will use primarily in following sections), one common way to achieve this is to use the ‘IsDesktopBuild’ and ‘BuildingSolutionFile’ properties as conditions to test in the tasks.

The ‘IsDesktopBuild’ property is declared in the ‘Microsoft.TeamFoundationBuild.targets’. The ‘BuildingSolutionFile’ property is declared and assigned automatically by MSBUILD.

The following table lists the values of each of these properties in each of the build environments.

One caveat with using the ‘IsDesktopBuild’ property is that it is not defined in many target files by default. This property will have an ‘empty’ value in a Visual Studio build, so we initialize it to a value of ‘true’ as the default value. Therefore we need to be explicitly define it in all MSBUILD target files where it will be tested.

We simply add the following <IsDesktopBuild> element to all target files where we need to differentiate between a build on the development machine and a build on the build server (within the first <PropertyGroup> section).

<IsDesktopBuild Condition="'$(IsDesktopBuild)' == ''">true</IsDesktopBuild>

MSBUILD Tasks

MSBUILD tasks provide a managed, maintainable and reusable way to implement many of the tasks we wish to perform on both the development and build server machines during the build process. The MS Build engine uses a ‘target’ to group several ordered build tasks together. A target often represents a logical unit of work like: deleting files from a directory or compiling source code files, etc.

Normally, targets are declared in the project files that use them, but for reuse and maintainability purposes these targets are often defined in shared files that are imported into the project files. For example:

<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />

For a list of MSBUILD properties that can be used as paths for theses includes see ‘MSBuild Reserved Properties’

In case of software factory solutions (that include VSX and DSL’s packages) there are commonly three target files (part of VS SDK installation) that we have to deal with:

·         Microsoft.DslTools.targets – contains tasks specific to DSL solutions (%ProgramFiles%\MSBuild\Microsoft\VisualStudio\DSLTools\v2.0)

·         Microsoft.VsSDK.targets – contains tasks for normal VSX packages (%ProgramFiles%\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Tools\Build)

·         Wix.targets – contains tasks for WIX setup projects (%ProgramFiles%\MSBuild\Microsoft\VisualStudio\DSLTools\v2.0)

Currently, GAT project types do not make use of additional MSBUILD tasks, and therefore there are no specific target files for these project types.

These targets contain a number of tasks that have multiple dependencies on files, tools and registry settings normally installed by the various SDK’s and toolkits on developer machines.

To use these tasks in a team build scenario (where we want to minimize the impact of installing toolkits on the build server), we need to modify not only the project files that reference the targets, but also the targets files themselves (provided by the various toolkits) and enhance them. We need to do this because the target files that are installed by default won’t be using the ‘IsDesktopBuild’ condition – as noted before. We need to provide our own versions of these target files in our source tree, and we need to redirect or provide the location of dependencies of the targets to dependencies and support files in our source tree. We don’t want to have to update the target files on every developer’s machine, so instead use a single modified target file from within the source code structure to be accessible to the solution no matter where it is built.

Modifying Target Files

For each of the targets files referenced above, copy the original version from a development machine and add to the ‘BuildExtension’ folder of the solution.

For more details on creating and customizing build tasks see: Walkthrough: Customizing Team Foundation Build with a Custom Task on MSDN.

Referencing Targets Files

In all project files that use any of the above targets, we need to make sure we import the modified target files that are available on the build server.

In all project files of the software factory solution (*.csproj or *.vbproj), replace all occurrences of <Import> statements, such as:

<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\DSLTools\v2.0\Microsoft.DSLTools.targets" />

With the appropriate <Import> statements:

<Import Project="$(SolutionDir)MySoftwareFactory\BuildExtension\Microsoft.DSLTools.targets" />

Notice that throughout this article, the examples paths given may be different depending on the naming used and where the solution is located in respect to the current project in the source tree.

VSX Project Specific Issues

The following issues are specific only to any VSX package projects (includes DSL package projects) contained within the software factory solution.

VSSDK Tools Dependencies

In the Microsoft.VsSDK.targets file, the ‘FindSDKInstallation’ target has a dependency on the ‘FindVsSDKInstallation’ task.

<Target Name="FindSDKInstallation"
          Condition="'$(VsSDKInstall)'==''" >
    <FindVsSDKInstallation SDKVersion="$(VsSDKVersion)">
      <Output TaskParameter="InstallationPath" PropertyName="VsSDKInstall"/>
      <Output TaskParameter="IncludesPath" PropertyName="VsSDKIncludes"/>
      <Output TaskParameter="ToolsPath" PropertyName="VsSDKToolsPath"/>
    </FindVsSDKInstallation>
</Target> 

The ‘FindVsSDKInstallation’ task uses the ‘Microsoft.VsSDK.Build.Tasks.dll’ assembly (from the VS SDK) to locate some values from the registry and populate some output parameters (i.e. VsSDKInstall, VsSDKIncludes and VsSDKToolsPath).

<UsingTask TaskName="FindVsSDKInstallation" AssemblyFile="Microsoft.VsSDK.Build.Tasks.dll" />

On the team build server, the registry settings required here will not present since we are not installing the VSSDK on the build server  to remove the requirement to pre-install additional SDK’s and toolkits, and minimize the impact of actually installing them.

Instead, we can provide the values of these properties by defining them in the target file to avoid the registry lookup, by adding the following properties.

<PropertyGroup>
    <VsSDKInstall>$(SolutionDir)MySoftwareFactory\BuildExtension\</VsSDKInstall>
    <VsSDKIncludes>$(SolutionDir)MySoftwareFactory\BuildExtension\inc\</VsSDKIncludes>
    <VsSDKToolsPath>$(SolutionDir)MySoftwareFactory\BuildExtension\</VsSDKToolsPath>
</PropertyGroup>

Where the paths above are relative to the solution files in the source tree structure.

Registering VSX Packages

For debugging purposes, VSX packages are automatically registered with Visual Studio when rebuilt by executing the following command line:

devenv.exe /setup 

This re-registers the VSX package with new instances of Visual Studio by refreshing a number of its caches. This is implemented with a number of targets in the Microsoft.VsSDK.targets file, such as: ‘SettingUpDevenv’, ‘GenerateParserCodeFromGrammar’ and ‘GenerateCodeFromLex’.

However, this registration is not required as part of an automated build process, since the build process does not debug the compiled packages. These targets therefore should not be executed on the build server, and requires that the ‘IsDesktopBuild’ condition restrict their execution to development machines only.

  <Target Name="SettingUpDevenv"
          Inputs="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')" 
          Outputs="$(DevenvSetupStateFile)" 
          DependsOnTargets="$(SettingUpDevenvDependsOn)"
 Condition="'$(IsDesktopBuild)'=='true' >
    <DevEnvSetup CTOFiles="@(CtcFile->'$(IntermediateOutputPath)%(FileName).cto')"
                 ProductVersion="$(ProductVersion)"
                 RegistryRoot="$(TargetRegistryRoot)"
                 StateFile="$(DevenvSetupStateFile)" />
  </Target>

CTC Task Dependencies

CTC command structure is used by the DSL Tools for versions contained in VSSDK 4.0 and below. For higher versions of the DSL Tools, a new type of command structure is used.

The ‘CTCCompile’ task in the Microsoft.VsSDK.targets file depends on a registry key that holds the location of the VS SDK resources. In previous sections we have avoided the registry lookup by providing hardcoded property values. However in the case of the ‘CTCCompile’task this workaround is not effective due to architecture of the ‘CTCCompile’ task.

In this case we can’t prevent a registry key lookup on the build server. Instead, we employ a strategy to set the registry key using a new target called ‘SetVsSDKInstallDir’. In this way provide the required value, and again avoid the installation of the VSSDK.

By adding the ‘SetVsSDKInstallDir’ target we introduced a dependency on the ‘Microsoft.Sdc.Common.tasks.dll’ assembly from http://www.codeplex.com/SdcTasks. This Microsoft provided library contains a number of reusable MSBUILD targets for creating customized build processes.