Understanding Custom Build Rules in Visual C++ 2005: Building Flex and Bison Files from the IDE

Marian Luparu
Microsoft Corporation

May 2006

Applies to:
   Microsoft Visual C++ 8

Summary: Custom build rules support is an important addition to the Project System in Visual C++ 2005. It enhances the way developers integrate external tools in the build process. The article describes how to create and use custom build rules inside Visual C++ 8, and it also provides some tips on how to handle various rule configuration options. (14 printed pages)

Contents

Introduction
What Are Custom Build Rules?
Fast Track: Using the "FlexBison.rules" Custom Build Rule
In-Depth Track: Creating a New Custom Build Rule
Tips for Configuring Custom Build Rules Effectively
Conclusion

Introduction

The article contains three main sections, and, depending on your knowledge of the topic, you can either walk through all of them or just jump to the one of interest to you.

If you are reading this article because you want to use flex and bison inside your project, you should get the FlexBison.rules file attached to this article and focus on the first main section, "Fast Track," because it describes the steps you need to take in order to use an existing custom build rule in your project.

On the other hand, if you are reading the article because you are planning on writing your own custom build rules file for another command line utility that you need to use or develop, go through the "In-Depth Track" and "Tips" sections.

What Are Custom Build Rules?

In older versions of Visual C++, you can specify, for a specific file, a custom command that gets executed before the C++ compilation step. This functionality, called Custom Build Steps, has some limitations when working with large projects and complex external tools. You cannot set up a custom build command to be run on all files of a particular type. Also, just for using an external application as a custom build step, you need to have extended knowledge of the configuration flags available on the command line.

This functionality is also available in Visual C++ 2005, but it is complemented by a new feature called Custom Build Rules. The drawbacks to custom build steps mentioned before are addressed by custom build rules.

Custom build rules, just like custom build steps, are commonly used for invoking external tools that translate files from one format to the other, often resulting in C++ sources that can be compiled at the C++ compilation step.

Custom build rules are associated with a specific extension, so after enabling the rule in a project, all files in the project that have that particular extension will be custom-built. The information about the rule resides in a XML file with the .rules extension, and you can share this file with your team by adding it to source control; you can even deploy it together with your command-line utility.

The process of defining a custom build rule is split into two steps:

1. Create the .rules file, in which you define an association between the flags of the command-line utility and the user properties that you create.

   This step is detailed in the "In-Depth Track" section.

2. Associate a rule to a project and, optionally, tailor the properties defined in the .rules file for the files.

   This step is detailed in the "Fast Track" section.

When configuring the rule in a particular project, you don't necessarily need to have knowledge of the flags that are used by the command-ine utility. A well written rule file would allow complete configuration using the properties defined. Tips on how to increase the usability of your custom build rules are presented at the end of this article in the "Tips" section.

Fast Track: Using the FlexBison.rules Custom Build Rule

Scenario: You and your team are required to start working in one of your projects with Flex and Bison tools, because you must write a fairly simple language interpreter. In search of the easiest way of automating these external tools, one of your colleagues built a file, FlexBison.rules, and then sent it to you so that you could check it out. You need to configure the TinyMath sample project (attached to this paper) in order to work with this file for building flex and bison files.

Before configuring Visual C++ to work with FlexBison.rules, you must install the Flex and Bison command-line utilities on your hard drive. You can download Flex and Bison tools for Windows from one the following links:

• Flex and Bison
• Flex only
• Bison only

The custom build rule file does not contain absolute paths for flex.exe and bison.exe. You should add the installation folder to the path environment variable, so that when the utilities are invoked during the build process, the IDE will be able to find them.

Importing a .rules File in Visual C++

Importing a .rules file in Visual C++ is fairly simple. After you have opened the TinyMath project, right-click the project in the Solution Explorer, and then click Custom Build Rules.

The Custom Build Rule Files window is displayed. Click Find Existing, and select the location where you saved FlexBison.rules. The rule will be added to the list of available rules for the selected project. When prompted to add the location of the file to the rule file search path, click Yes if you are planning to use this rule for other projects too. If you add the file location to the rule file search path, FlexBison.rules will be in the list of available rules for every project.

Enabling a Rule in Visual C++

After adding the file to the list of custom build rules, by default, the rules defined in the file are not enabled. To enable them, select the rule in the Available Rule Files list in the Custom Build Rule Files window (see figure 1).

Figure 1. Available Rule Files: Flex and Bison Tools rule enabled

Interfacing with the Generated Code

After the Flex and Bison Tools rule is enabled, when you build the project, all files in the project having the extension .l or .y will be custom-built by using the default command line for flex.exe and bison.exe as they are defined in FlexBison.rules.

To make a call to the parser implemented in gmath.y, add the following code to declare the main parse function.

extern "C" {
   extern FILE*      gmathin, *gmathout;
   extern int         gmathparse();
}

In the function where you want to make the call, first set up the input and output of the parsing. When you do not open a file for input (for example, in gmathin), the default stdin will be used. When you do not open a file for output (for example, in gmathout), the default stdout will be used.

int main(int argc, char* argv[]) {
   if( argc < 2 ) return;
   FILE*      fin = fopen( argv[ 1 ], "rt" );
   gmathin = fin;
   if( gmathin == NULL ) {
      printf( "Input Error: File [%s] was not found\n", argv[ 1 ] );
      return 1;
   }

   int      ret = gmathparse();
   fclose( fin );
   return ret;
}

In the preceding example, the input is read from the file received as parameter by the program, and the output is displayed on the standard console output (stdout).

Notice the prefix gmath on all declarations. Bison.exe implicitly prefixes the symbols from a .y file with yy (for example, yyparse(), yyin, and yyout). The same is true for Flex.exe (for example, yylex()). The problem materializes when you add more than one .l or .y file to the project, and their symbol names collide. This is the case for the sample project that contains two .l files (gmath.l and lc.l). The default configuration in FlexBison.rules ensures that the generated files do not have any filename or symbol name collision. See Table 1 for default naming in the sample project.

Table 1. Output files and defined symbols in the TinyMath project

In order to interface with the lexical analyzer defined in lc.l, you first declare the symbols that you need to refer to in your code.

extern "C" 
{
   extern FILE*      lcin, *lcout;
   extern int         lclex();
}

Then, just as the bison example, you set up the input and the output files for the analyzer. The same convention applies: if you do not explicitly specify a file for input or output, the default stdin and stdout (respectively) will be used.

int main(int argc, char* argv[])
{
   if( argc < 3 ) return 0;
   lcin = fopen( argv[ 1 ], "rt" );
   lcout = fopen( argv[ 2 ], "wt" );

   lclex();
   fclose( lcin );
   fclose( lcout );

   return 0;
}

Starting from these default property values, you can further configure the way the .l or .y files are processed. For a list of properties available for flex files, see Table 2. For a list of properties available for bison files, see Table 3. You can change these properties project-wide in the property pages for the project (right-click the project, and then click Properties), in which case the values will be inherited by all files involved (that is, flex files or bison files); or, you can change the properties on a per-file basis (right-click the file, and then click Properties), thus overriding the setting inherited from the project.

The values that you can specify in the property window, just like the default values, can contain macros that get expanded during the build process. For example a property value set to $(InputName) will get expanded to gmath for the gmath.y file. At build time, the command line is constructed by concatening the switches for all the properties, and replacing [value] strings with the actual property values.

Table 2. Properties available for flex files

Table 3. Properties available for bison files

The properties are wrappers for the command-line switches. For more details on the meaning of the properties, check their description in the property pages (see Figure 2), or read the documentation that comes with Flex and Bison.

Figure 2. Description for the Don't generate #line directives property

In-Depth Track: Creating a New Custom Build Rule

Scenario: You and your team are required to start working with Flex and Bison tools in one of your projects, because you must write a fairly simple language interpreter. In search of the easiest way to automate these external tools, you come across the Custom Build Rules feature in Visual C++ 2005. You decide to take advantage of it and write a .rules file.

Before creating a new rule file, we'll define the terms, and the relationships between some of the entities we'll use during this chapter—rule file, rule, and user property (see Figure 3).

Figure 3. Entity relationship (Click on the image for a larger picture)

• Rule File—An XML file having the extension .rules that contains the definition of one or more custom build rules.
• Rule—Contains the list of settings and properties that define how a command-line utility will be invoked during the build process.
  • Rule Setting—A required or optional characteristic of the rule (for example, Name, Display name, Additional dependencies, or Outputs).
• User Property—The definition of a switch exposed by the command-line utility.
  • User Property Setting—A required or optional characteristic of the user property (for example, Name, Description, Switch, or Default value).

Creating a New Rule File

To create a new rule file, from the Solution Explorer, right-click the project, and then click Custom Build Rules. In the Custom Build Rule Files window, click New Rule File.

In the New Rule File window that appears, specify the display name of the rule (this name will appear in the Custom Build Rule Files window), the filename, and the location on disk where the file will be stored (see Figure 4). The file will be created as soon as you click OK.

Figure 4. New Rule File window

A rule file can contain more than one rule (as you can see from Figure 3). You can start adding rules to the rule file by clicking Add Build Rule.

In the Add Custom Build Rule window, you can specify the settings for the rule. See Table 4 for some of the most important rule settings.

Table 4. Some rule settings described

flex.exe -olex.$(InputName).c

flex.exe [OutputFile]

In the bottom area of the Add Custom Build Rules window, you can also define properties that will be attached to the rule. User properties can be seen as user-friendlier representations of the switches exposed by the command-line utility.

See Table 5 for the settings for the user properties. Note that support for some settings depends on the property type. The Custom Build Rules editor supports four types of user properties:

• Boolean—The value of the property is either On or Off.

  The switch associated to the property is rendered in the command line only when the Boolean user property value is set to On.

• Enum—The value is one of the elements of a defined list.

  The value of the user property is chosen from a predefined set of constant values. Each value in the set has an associated switch that is rendered into the command line, depending on the user choice. One of these values in the set is considered to be the default value. This type of property is useful when configuring switches that are excluding one another on the command line.

• Integer—The value is a number.

• String—The value is a string.

  Both integer and string user properties have a specific way of rendering the switch in the command line. The switch can contain the keyword [value] in its definition. When the user property has a value specified by the user (either integer or string), the switch is rendered by replacing the keyword [value] with the actual value. For example, an integer property called Warning Level can have the switch set to /W[value] and, depending on whether the user sets the property value to 0, 1, or 2, the rendered switch will be /W0, /W1, or /W2.

Table 5. User property settings described

Listing 1 and Listing 2 provide examples of user property usages. They are presented in the XML format, because they are stored in the .rules file.

Listing 1. Sample string user property, using the [value] keyword and Default Value

<StringProperty
      Name="OutputFile"
      DisplayName="Output File"
      Description="The output file containing the implementation of the analyser"
      Switch="-o[value]"
      DefaultValue="lex.$(InputName).c"
/>

Listing 2. Sample enum user property

<EnumProperty
   Name="TableCompression"
   DisplayName="Table Compression"
   PropertyPageName="Performance"
   Description="Controls the degree of table compression and, more generally, trade-offs between small scanners and fast scanners.">
      <Values>
         <EnumValue
            Value="0" Switch="-Cem"
            DisplayName="Equivalence classes and meta-equivalence classes (slowest &amp; smallest)"/>
         <EnumValue
            Value="1" Switch="-Cm"
            DisplayName="Meta-equivalence classes"/>
            <!-- Other values not included... -->
      </Values>
</EnumProperty>

You can also use the FlexBison.rules file attached to this article for some additional examples of user property types.

Tips: Configuring Custom Build Rules Effectively

1. Use Macros When Possible

Do not hard-code paths, filenames, or other switch values that may result in conflicting or duplicate information, overriding of output files, or impossibility of reuse from one project to the next. When possible, make use of the list of macros available when setting the values of the properties—for example:

• Suppose that you are planning to use a switch that instructs outputting a report file for each file in your project. In this case, it would be a good idea to specify the name of the report to be dependent on the input filename, by using the $(InputFileName) or $(InputName) macro.
• Suppose that you refer an external tool that is installed by the .NET Framework—let's say, regasm.exe. The thing to consider is that the location differs from one version of .NET to the other. In this case, you might consider referring the tool by $(FrameworkDir)$(FrameworkVersion)\regasm.exe.

2. Make the Most of File Batching

When building your custom build rule, one property that you can set for the rule is Supports File Batching. By default the value is False.

If you switch the flag to True, when building, the external tool will get invoked only once, and the [inputs] tag in the command line will get expanded to the full list of files matching the extension. You can also alter the separator between the filenames—by default, this is a space character. To change this, use the Batching Separator property.

You can take this approach when the external tools that you use have a large execution cost in the overall build process, and invoking it for each file is not practical. You should also make sure that the tool supports receiving a list of input files as arguments.

3. Group Properties by Topic

When building rules, you may group related properties in categories and property pages. To do that, you can specify the Category and Property Page Name fields for a specific property (see Figure 5).

Figure 5. Properties grouping in: (1) Property Pages, (2) Categories

4. Create a User Property for a Repeating Flag

Some command-line utilities permit repetition of the same flag in order to allow the accumulation of values—for example, when specifying a collection of search paths.

   > utility.exe -p C:\Demo\ -p C:\Work\

To achieve this behavior, you must create a string user property and specify its Delimited setting to True, and its Delimiters setting to ,. Set the Switch setting to -p [value].

The value will be specified as a comma-separated list of items (for example, C:\Demo\, C:\Work\). When the command line is rendered, the switch -p will be applied to each item in the list.

5. Document the Properties

When configuring the rule, do not forget the Description field when setting up the properties. It can contain a short sentence, or even a more detailed sample usage. It can be of great help for the user of the rule, given that this description is always displayed in the bottom area of the property pages.

6. Provide Defaults for Properties

For each user property, you can specify a default value. It is a great opportunity to conceive a default usage scenario, so that the user does not have to completely specify the value of all user properties before starting to use the rule, but can just fine-tune as he or she goes along.

Default values support macros, just like the values.

7. Order Custom Build Rules from the Tool Build Order Window

In large projects with more than one build rule, depending on the file dependencies the project has, the order in which the build rules are executed may become relevant. You can alter this order from the Tool Build Order window that is accessible by right-clicking the project in the Solution Explorer, and then clicking Tool Build Order. In this window, you can enable or disable a build step, or change the order in which the steps are invoked, by using the up and down arrow buttons.

You should use this functionality with caution, though. Change the order of Visual C++ internal build steps only when you have a full understanding of the effect that the change has. If you experience unexpected behavior due to changes to this list, you can always restore the initial order by clicking Restore in the Tool Build Order window.

Conclusion

Custom build rules are a completely new functionality in Visual C++ 2005, and a valuable addition to custom build steps in leveraging the command-line tools inside the IDE. We are encouraging you, as a build tool developer, to write your own custom build rule files and distribute them along with your products. This will enable your customers to have a similar user experience in configuring your product as the one they are accustomed to in Visual C++ property pages.

We are also convinced that the Custom Build Rule functionality can be further improved in future versions. As we struggle to enable more usage scenarios and remove any existing pain points, we welcome your comments and suggestions on this matter.

© Microsoft Corporation. All rights reserved.