Foundations

Tracking Services in Windows Workflow Foundation

Matt Milner

Code download available at:  Foundations 2007_03.exe(178 KB)

Contents

Tracking Architecture
Tracking Profiles
SQL Tracking Service
Querying Tracking Data
Custom Tracking Services

One of the most powerful features in Windows Workflow Foundation is tracking. It allows you to monitor events, activity properties, and custom data in your workflows. In this column, I will examine the tracking infrastructure, showing you how to use the built-in SQL Server™-based tracking service and how to create custom tracking services for a variety of purposes. Along the way, I'll demonstrate how to consume information that is being tracked and how to solve some common requirements of applications through the use of tracking.

Many applications require insight into the execution of program logic and processing steps. This could be for regulatory compliance, managing task lists based on execution milestones, or a variety of other reasons. Windows® Workflow Foundation provides a flexible infrastructure where you can overlay your custom implementations, rather than creating a different tracking system for each application. This simplifies the development model, allowing you to focus on the business requirements for tracking.

Tracking Architecture

When hosting a workflow in Windows Workflow Foundation, you host both a workflow runtime and a set of runtime services. Many runtime services, such as the persistence service, are required to have at most one instance registered in the runtime. Tracking services are the exception, allowing for multiple tracking services to be added to the runtime in order to support different business requirements. This allows you to apply several different tracking services to one application, with each tracking service responsible for a specific implementation.

Tracking in Windows Workflow Foundation is based on a central interceptor, which monitors activity and workflow events and sends the appropriate data to each of the configured tracking services. (This architecture is illustrated in Figure 1.) The tracking interceptor receives notification about all events, but in most cases only a subset of these events will be useful. Moreover, if you are using multiple tracking services, it is likely that each service will be interested in a unique subset of information. Thus, the tracking interceptor queries each configured tracking service for a tracking profile for each workflow type.

Figure 1 Tracking Architecture

Figure 1** Tracking Architecture **(Click the image for a larger view)

Tracking profiles provide information to the interceptor about the types of events the tracking service wants to receive. These events can fall into three categories: workflow, activity, and user.

Workflow events allow you to track the various states that a workflow can be in and the time at which they occurred. For events that include custom event arguments, such as with WorkflowTerminated or WorkflowCompleted, the event arguments themselves are tracked and can later be queried.

Activity events allow you to track when an activity entered a particular state and the specific values of properties on the activity when it entered that state. For example, if you have an activity that sends e-mail, you can track the To address property when the activity enters the Closed state.

Finally, user events let you track data manually in the code of your workflow or activities.

Tracking Profiles

Each tracking service must provide tracking profiles to the tracking interceptor. A service can provide the same profile for all workflows or different profiles for different workflow types. It is up to the service author to determine how to manage and produce profiles.

Tracking profiles can be created with an object model found in the System.Workflow.Runtime.Tracking namespace or directly in XML. They are usually created using the object model, but this is not the only way to represent the profile, nor is it always the best way. For this reason, the tracking profile can be serialized to and deserialized from XML. You can use the TrackingProfileSerializer class to convert a TrackingProfile object into XML text and vice versa. A useful example of this, and a good tool for basic tracking profile editing, is the Tracking Profile Designer sample in the Windows SDK. This app allows you to visually define the tracking profile using the workflow design surface. While the tool has some limitations, such as lack of support for user tracking points, it does let you view the serialized form of the tracking profile. It can also speed the development and deployment of tracking profiles in many scenarios.

A tracking profile is essentially a collection of track points. Each track point correlates to the types of events that can be tracked, providing information to the runtime about when to extract data and what data to extract. To indicate that workflow events should be tracked, for example, a WorkflowTrackPoint must to be added to the tracking profile. A track point contains one or more tracking locations, which actually define where and when to track the data. For instance, on a WorkflowTrackPoint, the MatchingLocation property identifies the various workflow events to track. The code in Figure 2 shows a tracking profile being created. It has a single WorkflowTrackPoint, which indicates that the service wants to track failure conditions, such as exceptions, terminations, and abortions of workflows.

Figure 2 Workflow TrackPoint

//create the profile object and set the version
TrackingProfile profile = new TrackingProfile();
profile.Version = new Version(1, 0, 0);

//define a tracking point
WorkflowTrackPoint wfTrack = new WorkflowTrackPoint();

//create a list of events we are interested in
List<TrackingWorkflowEvent> events = new List<TrackingWorkflowEvent>();
events.Add(TrackingWorkflowEvent.Aborted);
events.Add(TrackingWorkflowEvent.Terminated);
events.Add(TrackingWorkflowEvent.Exception);

//create a tracking location to indicate which events should be tracked
WorkflowTrackingLocation wfLocation = 
    new WorkflowTrackingLocation(events);

//set the matching location on the track point
wfTrack.MatchingLocation = wfLocation;

//add the track point to the profile
profile.WorkflowTrackPoints.Add(wfTrack);

ActivityTrackPoint supports much richer tracking. This is logical since activities make up the heart of workflows. The ActivityTrackPoint contains both MatchingLocations and ExcludedLocations collections, allowing you to closely control when data is collected. For both collections, you define an ActivityTrackingLocation that represents the set of conditions you want to match.

The ActivityTrackingLocation class allows you to define the activities you want to track based on their type. You can also indicate whether you want to match derived activities. For example, if you want to match all activities, you would use the type System.ComponentModel.Activity and set MatchDerivedTypes to True. This would match the base activity class and all activities that derive from it, meaning everything. You then add this information into the MatchingLocations collection of the ActivityTrackPoint to indicate that you want to track these events or into the ExcludedLocations collection to indicate that this data should not be collected.

For more granular control, you can specify conditions, in the form of property equality, that must be met in order to track the data. For example, you might choose to track e-mail messages only when the priority property on the SendEmail activity is set to High. Figure 3 shows an example of this.

Figure 3 ActivityTrackPoint

ActivityTrackPoint actTrack = new ActivityTrackPoint();

ActivityTrackingLocation actMatch = new ActivityTrackingLocation(
    typeof(SendEmail), false, new ActivityExecutionStatus[] { 
        ActivityExecutionStatus.Closed });

ActivityTrackingCondition condition = 
    new ActivityTrackingCondition("Priority", "High");
actMatch.Conditions.Add(condition);

actTrack.MatchingLocations.Add(actMatch);

profile.ActivityTrackPoints.Add(actTrack);

You can also extract data from the activity or workflow in the form of property values. For example, you can track the To and Subject properties on the SendEmail activity when your matching locations are reached in the workflow. You do this by adding ActivityDataTrackingExtract or WorkflowDataTrackingExtract items into the Extracts collection on the ActivityTrackPoint. Each ActivityDataTrackingExtract identifies the member to be extracted and sent to the tracking service. Adding data extracts to the activity track point so that the To and Subject properties tracked look like this:

actTrack.Extracts.Add(new ActivityDataTrackingExtract("To"));
actTrack.Extracts.Add(new ActivityDataTrackingExtract("Subject"));

UserTrackPoints, which identify the final point of tracking, are the most flexible of the three types. Sometimes you need to track data from your workflow but the data is not available as a property on an activity. Or you may need to use logic beyond the simple condition support in an ActivityTrackingLocation. In such cases, you can call the TrackData method on the ActivityExecutionContext or the Activity class. This lets you send data to the tracking interceptor along with an optional key or name for the data item. For a tracking service to receive this data, it must define a UserTrackPoint with the correct matching locations. You can also define workflow and activity data extracts to have properties on the workflow or the current activity tracked when the UserTrackPoint is matched.

SQL Tracking Service

Windows Workflow Foundation includes a SQL Server tracking service, which allows you to define tracking profiles for a workflow type and deploy them to SQL Server. The tracking service then uses the profile at run time to configure the tracking interceptor. The SQL Server tracking service includes several compelling features, including profile versioning, profile update notification, and data partitioning.

Tracking profiles are deployed to the SQL tracking service by calling the UpdateTrackingProfile stored procedure and passing in the serialized tracking profile along with type information for the workflow and a version number. You can use this stored procedure to insert the tracking profile specific to your workflow type, but you must use a unique version number each time you update the profile. TrackingProfileSerializer, which I mentioned earlier, lets you get the XML text of your profile so you can pass it to the stored procedure.

If you do not specify a tracking profile for your workflow type, SQL tracking service uses the UseDefaultProfile setting to determine whether it should supply a default profile to the tracking interceptor. The default profile tracks all status events for the workflow and all activities. It also includes matching locations to track any user tracking events that are submitted. If you do not want to have this default behavior, you need to set the UseDefaultProfile property to False when configuring the SQL tracking service. Then only those workflow types with profiles specified will be tracked.

To ensure it is current, the tracking interceptor periodically asks each tracking service for profiles that it has already received. To avoid the repeated loading or creation of tracking profiles, a tracking service can implement the IProfileNotification interface. This defines two events: ProfileUpdated and ProfileRemoved. When a tracking service implements this interface, the tracking interceptor does not query for updated profile information unless one of these events is raised. To support this model, the SQL Server tracking service periodically queries the database (every 60 seconds by default) to see if the tracking profile for a workflow type has been updated or removed. You can configure how often the service queries for updates by setting the ProfileChangeCheckInterval property to a value that represents the milliseconds between polling.

Since tracking data can accumulate quickly in a production environment, the SQL tracking service supports data partitioning. You can configure data to be segmented by day, week, month, or year; the service will create tracking tables for each time period. In addition, you can control whether data is moved on demand (so it can be scheduled for downtime) or automatically (when the workflow completes). To configure this setting, you set the PartitionOnCompletion property on the SQL Server tracking service to true to indicate that each workflow should be moved to the appropriate tracking partition when it completes. To manually partition the data, you call the PartitionCompletedWorkflowInstances stored procedure in the tracking database.

Querying Tracking Data

When you collect data using the SQL tracking service, the data is stored in SQL Server, which means you can use traditional reporting tools to view the data and do analysis. But in some scenarios, you'll want to query the workflow data in your application. Rather than forcing you to write SQL queries and determine the correct relationships in the database, Windows Workflow Foundation includes a tracking query programming interface that allows you to search for tracking information for workflow instances. The information is returned as a class, which provides access to all of the tracked events and data that was specified by your tracking profile.

The SqlTrackingQuery class allows you either to try to load a particular workflow instance's tracking data using the instance ID of the workflow or to query for a list of workflows using a discrete set of parameters. When loading a particular instance, you use the TryGetWorkflow method, which returns a Boolean value indicating whether the request was successful. To query for multiple workflows, you use the GetWorkflows method, which requires a SqlTrackingQueryOptions instance.

The query options allow you to define the parameters to use when searching for workflows in the tracking database. The simplest options include a minimum and maximum date value and the type and status of the workflow. For example, you can configure it to search for all workflows matching your workflow type which have a status of Completed and fall between today and a week ago. This lets you retrieve targeted information about a workflow type and how it turned out.

I already showed you a tracking profile that lets you indicate interest in failure conditions. Using the code shown in Figure 4, you can query the database for all instances of workflows that terminated and output the exception details. Note that the tracking information contains the exception details as the event arguments of the terminated event that were tracked.

Figure 4 Querying for Terminated Workflow Instance

SqlTrackingQueryOptions options = new SqlTrackingQueryOptions();
options.WorkflowStatus = WorkflowStatus.Terminated;

SqlTrackingQuery query = new SqlTrackingQuery(connectionString);
IList<SqlTrackingWorkflowInstance> workflows = 
    query.GetWorkflows(options);

foreach (SqlTrackingWorkflowInstance workflow in workflows)
{
    foreach(WorkflowTrackingRecord workflowEvent in 
        workflow.WorkflowEvents)
    {
        TrackingWorkflowTerminatedEventArgs args = 
            workflowEvent.EventArgs as 
                TrackingWorkflowTerminatedEventArgs;
        if (args != null)
        {
            Console.WriteLine(
                "workflow {0} terminated with exception: {1}", 
            workflow.WorkflowInstanceId, args.Exception.Message);
        }                  
    }
}

You can also search based on extracted values. For example, if you have a tracking profile configured to extract the To address of a SendEmail activity, then you can write a query to return the workflow instances where the To address matches a certain value, such as a particular employee or partner. The following code is an example of using TrackingDataItemValue items in the SqlTrackingQueryOptions to refine the search to particular items based on extracted values:

SqlTrackingQueryOptions options = new SqlTrackingQueryOptions();
TrackingDataItemValue item = new TrackingDataItemValue(
    "sendEmail1", "To", "highpriority@example.com");
options.TrackingDataItems.Add(item);

SqlTrackingQuery query = new SqlTrackingQuery(connectionString);
IList<SqlTrackingWorkflowInstance> workflows = 
    query.GetWorkflows(options);

If you choose to directly query the SQL Server database for reporting or other needs, you should use the views that are defined in the table rather than using the tables directly. This is a good practice in general, but it is very important here as the views will contain data from all of the tracking partitions that have been created. By using the views, you remove the burden of having to union the data from all of those tables together yourself.

Custom Tracking Services

While tracking data into a SQL Server database is an important use case, and likely the most common, it is not the only scenario for using tracking information. Because it is such a common requirement, Microsoft decided to include this rich service with Windows Workflow Foundation. However, a service such as this, with such wide use, must be as general as possible. This means that it may not be appropriate in many situations. In such cases, you can create a custom tracking service.

There are many requirements in workflow solutions that can be met using a custom tracking service. Here I'll discuss the basics of creating your own tracking service, as well as provide a few examples of custom services that meet common requirements. The important thing to remember is that tracking is your view into the workflow. Once you understand how to configure a tracking service with tracking profiles so you can get the results you need, it is then just a matter of how your custom service consumes or represents that tracked information.

In order to create a custom tracking service, you need to do two things. First, you have to determine how your tracking profiles for your service will be created. In some cases, the profile may be static and unchanging across all workflow types. In that case, you will be able to create your profile in code and simply return it to the runtime when the profile is requested. However, if you need your tracking service to support multiple profiles or versions of profiles, then you need a way to store that information and retrieve it from your tracking service.

Next, you have to implement two classes that provide the runtime behavior of your service: a tracking service and a tracking channel. The tracking service class acts as the workflow runtime service, and it is the direct interface for the host application and the runtime to interact with your tracking service. You will create an instance of this class and add it to the workflow runtime just like any other service, using code or configuration.

The tracking service class has two primary functions: to provide tracking profiles to the tracking interceptor and to provide tracking channels to which the listener can write data. The tracking channel is the class you implement that has the responsibility of receiving and handling the tracking data. It can write that data out to a store, such as SQL Server or the Windows event log; it can raise events, such as Windows Management Instrumentation (WMI) events for operations; or it can send the data back to the tracking service to hold it in memory and make it available to the host application. There are certainly other options, and what you do with the data will depend on your needs and requirements. Your custom tracking services will be differentiated by the tracking profiles they use and how they handle the data that they receive.

The tracking channel is the easiest place to start as it has two simple methods: Send and InstanceCompletedOrTerminated. Your tracking service is asked to supply a tracking channel for each workflow instance and it is common practice to supply a unique instance of your tracking channel for each request. You generally want to provide a constructor that takes TrackingParameters as a parameter-you may need this information in your implementation of the other methods. The TrackingParameters class has information about the workflow instance being tracked (such as the instance ID), access to the root activity definition, and information about parent activities and calling workflows (if this workflow was invoked from another workflow).

The InstanceCompletedOrTerminated method allows your tracking service to get notification when a workflow is complete so that it can clean up any resources related to tracking the workflow. This is a common place to need access to the workflow instance ID as supplied by the TrackingParameters because the InstanceCompletedOrTerminated method does not take any parameters.

The Send method on your tracking channel is called for every tracking record that was requested in the tracking profile. Mimicking the profile, there are workflow, activity, and user tracking records which all derive from the base class TrackingRecord. In the Send method, it is your responsibility to check what type of tracking record has been submitted and then handle it accordingly. If you know that your tracking profile does not indicate an interest in user tracking events, for example, then you do not need to handle user tracking records in your tracking channel.

The tracking service class itself is responsible for managing the tracking profiles. The tracking service base class provides several methods for retrieving the profile that allow you to manage profiles based on workflow type and workflow instance. This means that your service can return profiles that range from being the same for all workflows, to unique for each workflow instance.

There are two overloads for the GetProfile method. One takes an instance ID and the other takes the type and profile version number. Also, TryGetProfile takes the type information. In all of these cases, the job of your custom tracking service is to return a tracking profile that is appropriate for the type or instance.

There is also a method called TryReloadProfile, which the runtime calls to get an updated profile if one exists. The tracking interceptor, wanting to make sure it is up to date, will periodically ask your tracking service for an updated profile. Rather than having to create a new profile each time this request is made, or to look one up, you can implement the IProfileNotification interface in your custom tracking service. The IProfileNotifcation interface includes two events: ProfileRemoved and ProfileUpdated. When you implement this interface in your tracking service, the tracking interceptor no longer polls for updates. Instead, it waits for notification of any updates through the events on this interface.

Included in the code download for this column are several sample tracking services. The current state sample uses tracking to get the current state of a state machine workflow and make it available to the host application. The tracking profile for this service contains an activity track point for the executing status of State activities. The tracking channel then updates a dictionary in the tracking service class with the current state name when an activity tracking record is received. This is an example of a service that has the same profile for all workflows and writes data back to the service so it can be queried by the host application.

The other sample tracking service included in the code download is geared toward solving a common developer problem in Windows Workflow Foundation. Because of the nature of how workflows are executed and separated from the host, it is not a simple procedure to get a reference to the running workflow and extract property values. This is a common requirement, especially in ASP.NET scenarios. Once a ManualWorkflowScheduler has completed executing the steps in the workflow, it may be necessary to update the user interface with property values from the workflow.

While the information in the workflow can be passed out using the CallExternalMethod activity, this is not always the desired solution and can prove to be overwhelming in a large project. The workflow property tracking service sample uses tracking to solve this problem. In this tracking service, the profile returned is different for each type of workflow in order to reflect the different properties on the workflow. Reflection is used to build a collection of workflow data tracking extracts and add them to an activity track point. The location of the tracking is on the closed event of each activity. Therefore, after each activity closes, the property values from the workflow instance are extracted and sent to the tracking channel. The channel then updates a dictionary in the service making those values available to the host application.

These are two fairly simple examples of custom tracking services, but both address common developer scenarios and provide simple, reusable implementations taking advantage of the existing tracking infrastructure. Without the base to build on, each of these challenges would have been much larger, requiring more custom code, and the solutions likely not consistent.

As you can see, the tracking services in Windows Workflow Foundation are extremely flexible and can help in solving a number of different business requirements. Tracking is the key to workflow visibility and custom tracking services are your tool for tapping into that visibility in your applications.

Send your questions and comments to  mmnet30@microsoft.com.

Matt Milner is an independent software consultant and instructor for PluralSight specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. Matt lives in Minnesota with his wife Kristen and his two sons.