Monitoring Workflows with WF for Enhanced Application Data Visibility

Ganesan Krishnamurthy
Prashanth B. G.
Microsoft Technology Center, Infosys

Summary

Applications use workflows to control the order of execution of their individual activities. It is critical to monitor workflows to achieve data visibility within an application.

This white paper describes different approaches for monitoring workflows in Windows Workflow Foundation (WF) which provides the framework, tools and development models to build critical business applications. It also includes details of WF tracking framework support for Business Activity Monitoring (BAM) — a critical aspect of successful enterprise-level business applications.

The paper is aimed at readers who are familiar with WF.

Contents

The white paper contains the following topics:

• Introduction
• Overview of WF tracking Framework
• Using Out-of-box SqlTrackingService
• Customizing the WF Tracking Framework
  • Creating a Custom Tracking Profile
  • Creating a Custom Tracking Service
• Using End-to-end Tracing
• Using WF Tracking Performance Counters
• Displaying Tracked Data
• Performance Considerations for Tracking
• Comparing WF Tracking with BizTalk Server(BTS) BAM features

1. Introduction

In today's business scenario, it is essential for enterprises to improve the effectiveness and response time of their business operations and cope with strict service level agreements (SLAs).

Hence it is vital for enterprises to access business performance indicators in real time. The key to this lies in constant monitoring of workflows within applications.

1.1. Why Workflow Monitoring

Monitoring a workflow provides an insight into the progress of the workflow (required by administrators) and assists in the analysis of technical and business processes (required by business users).

Monitoring workflows solves the following data visibility issues:

1.2. How do you Monitor Workflows with WF

WF provides the following options to monitor workflows:

1. Tracking Infrastructure — Monitors and queries the execution of a workflow instance
2. Performance Counters — tracks the performance of the workflow runtime as it executes workflow instances
3. End-to-end Tracing — Monitors the application status without disturbing a running system

Note: WF does not support tracking through GUI (Graphic User Interface) wizards, portals / dashboards. However, customized tools can be built to visualize and analyze data. WF SDK (Software Development Kit) provides a sample administrative tool to visualize tracked data. This sample tool can be enhanced to provide richer capabilities that match the standards of BAM.

2. Overview of WF tracking Framework

The WF tracking infrastructure uses the following two approaches to monitor workflow instances:

1. Using the Out-of-box SqlTrackingService — Data is stored in SQL Server 2005 or SQL Server Express or SQL Server 2000.
2. Writing a custom tracking service — Data is stored in customized data sources like file system or database.

Note: System.Workflow.Runtime.Tracking namespace, available as part of .NET Framework 3.0 provides classes and Interfaces to implement either SqlTrackingService or customized tracking service.

The WF tracking framework consists of the following:

1. Runtime tracking Infrastructure
2. Tracking Services
3. Tracking Profile
4. Tracking Channel

The following sections provide details on the components of the tracking infrastructure.

2.1. Runtime tracking Infrastructure

Runtime tracking infrastructure is part of the Workflow Runtime Engine.

Following picture illustrates overview of tracking framework.

Figure 1: Runtime tracking Infrastructure

When a workflow is instantiated by the host application, tracking run time infrastructure does the following:

1. Instantiates the tracking service that is registered with the workflow runtime programmatically or through a configuration file.
2. Loads a tracking profile (one of the protected overloaded method — GetProfile() of TrackingService class) corresponding to the workflow instance type using TryGetProfile method.
3. Gets the tracking channel through GetTrackingChannel method, instantiates it and attaches the same to a workflow instance. While the workflow instance is running, the runtime engine captures the emitted tracking data such as Workflow events, Activity events and User events and passes them to the tracking channel that has been created.

The WF runtime generates the following events based on the way the workflow has been coded. Tracking Runtime Engine acts as an interceptor, it subscribes to these events and filters them based on the tracking profile associated with workflow instance. The attached tracking profile contains the list of events that are to be monitored. Those filtered events will then be forwarded to tracking channel. The tracking channel stores the filtered events' details into database or any storage depends on the implementation of it.

WF also provides a feature to track the output of rules evaluation. When workflow runtime evaluates the rules, it emits RuleActionTrackingEvent as a user track point to the runtime infrastructure.

RuleActionTrackingEvent object has the following two properties:

1. RuleName: Gets the name of the rule that caused RuleActionTrackingEvent to be raised.
2. ConditionResult: Gets the result of the condition evaluation (true/false).

2.2. Tracking Service

Tracking service is responsible for providing a tracking profile and a tracking channel to the tracking runtime. Using the tracking profile, runtime provides the events and data emitted from the workflow instances to the tracking service. Multiple tracking services can be registered with the Tracking Runtime Engine depending on the business scenario, for example, if it requires storing the information of the events and other data in different storage mediums for auditing purposes. Each Tracking Service requires a tracking profile and a tracking channel.

2.3. Tracking Profile

The tracking profile is used by the tracking runtime to filter specific events (that have already occurred and the ones to be monitored) and data (emanated from running workflow instances). Runtime tracking infrastructure uses tracking profiles (they contain a list of tracking point objects) to send information about specific events. The same tracking profile can be used for multiple workflow types or different profiles can be used for different workflow types.

A tracking profile is essentially collection of track points. Each track point provides the information on the type of data and when to collect the data. A tracking profile can have the following three types of track points:

2.4. Tracking Channel

The Tracking Runtime Engine requests Tracking Service for a Tracking Channel, to associate that with a workflow instance as part of initialization process, so that the tracked data can be stored in a storage medium. Tracking channel is a communication medium which transfers information about the tracked data from runtime tracking infrastructure to tracking service. The information is transferred in the form of tracking records.

When the runtime engine finds a match between the track points in tracking profiles and track events, it sends tracking records to the tracking service. These tracking records contain data associated with track points and track events of the tracking service. The tracking service invokes the Send method of the tracking channel object to store the data into a storage medium. tracking channel implementation stores these tracking records in a storage medium.

3. Using Out-of-box SqlTrackingService

WF provides a default out-of-box, SQL server based tracking service.

SqlTrackingService records information provided by the tracking infrastructure to the SQL server database. These records can then be retrieved directly through SQL or through sqltrackingQuery interface.

Note: When you install WF, the tracking database and corresponding schema objects are not created.

To install SqlTrackingService, you have to do the following:

1. Create a database, for example, tracking.
2. Execute the tracking_Schema.sql and tracking_Logic.sql scripts in the tracking database. This creates tables, stored procedures, and views required for the tracking service. You can pick up these scripts from:

   %windir%\Microsoft.NET\Framework\v3.0\Windows Workflow Foundation\SQL\EN\

Use the following code snippet to plug SqlTracking service into the workflow runtime:

SqlTrackingService	Copy Code
// Add the tracking service workflowRuntime.AddService(new SqlTrackingService(connectionString));

Attach SqlTracking service to workflow runtime through a configuration file as shown in the following code snippet:

WorkflowRuntime

Copy Code

<WorkflowRuntime Name="BusinessActivitytracking"> <Services> <add type="System.Workflow.Runtime.tracking.SqlTrackingService, System.Workflow.Runtime, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" ConnectionString = "Initial Catalog=tracking;Data Source=DBServer;Integrated Security=SSPI;" UseDefaultProfile = "true" IsTransactional = "true" ProfileChangeCheckInterval = "60000" PartitionOnCompletion = "false" /> </Services> </WorkflowRuntime>

The following diagram illustrates the SqlTracking Service class diagram.

Figure 2. SqlTrackingService Class Diagram

The following lists the attributes of the SqlTrackingService:

4. Customizing WF Tracking Framework

WF provides a feature to flexibly customize the tracking framework to suit the project requirements.

The following customizations are possible in WF:

1. Custom Tracking Profile: Used when customizing collected track events. Windows SDK also provides a tool to create tracking profiles that is described in the following sections.
2. Custom TrackingService: Used when a custom data store (apart from SQLServer 2005, SQL Server Express or SQL Server 2000) is needed

The following sections describe the processes involved in creating a custom tracking profile and a custom tracking service.

4.1. Creating a Custom Tracking Profile

A custom tracking profile applies only to the SqlTrackingService (which has the notion of a default profile). Write a custom tracking profile for the SqlTrackingService in order to change the events that are tracked or to track data from the workflow along with events.

A custom tracking profile is created when the tracking service needs to track only a subset of the workflow information, such as in the following scenarios:

• The amount of data tracked is too large to analyze. This may happen when the default tracking profile has tracked all events.
• The default tracking profile has a performance penalty because of the amount of data it collects.

The following sections describe three ways to create tracking profiles.

4.1.1. Using the Tracking Profile XML schema

The TrackingProfileSerializer class provides a property called Schema which gets the tracking profile schema definition (xsd).

• Use this schema to create a tracking profile XML file.
• The TrackingProfileSerializer class also provides methods to serialize and deserialize the tracking profile object into an xml document.

4.1.2. Using the Tracking Profile Object Model

The WF tracking profile object model creates tracking profiles, serializes them to XML form, and then deserializes the XML tracking profiles back to a tracking profile object.

• It provides WorkflowTrackPoint, ActivityTrackPoint and UserTrackPoint classes to create workflow track points, activity track points and user track points, respectively. Each track point contains a set of matching locations.
• The following displays class diagrams of the various classes used in tracking Profile Object Model.

Figure 3: TrackingProfile, WorkflowTrackPoint, ActivityTrackPoint and UserTrackpoint Class Diagrams

The following section explains the process of customizing the above classes:

1. Customizing WorkflowTrackPoint: It is used during runtime to match specific workflow status events from a workflow instance.
   
   WorkflowTrackingLocation is assigned to the MatchLocation property of WorkflowTrackPoint. It contains a list of configured workflow status events that belong to the runtime tracking engine.
   
   The following sample code creates a tracking profile that adds workflow track points to the tracking profile object using the object model.
   
   

   Workflow Track Point

   Copy Code

   trackingProfile trackingProfile = new trackingProfile(); WorkflowTrackPoint workflowTrackPoint = new WorkflowTrackPoint(); //Create instance of workflow tracking locations WorkflowTrackingLocation workflowTrackingLocation = new WorkflowTrackingLocation(); //Get the list of workflow status events. IEnumerable<trackingWorkflowEvent> workFlowEvents = Enum.GetValues(typeof(trackingWorkflowEvent)) as IEnumerable<trackingWorkflowEvent>; //Add list of workflow track points foreach (trackingWorkflowEvent workFlowEvent in workFlowEvents) { workflowTrackingLocation.Events.Add(workFlowEvent); } //assign workflow tracklocations to workflowtrackpoint workflowTrackPoint.MatchingLocation = workflowTrackingLocation; //Add workflow track points to tracking profile trackingProfile.WorkflowTrackPoints.Add(workflowTrackPoint);

2. Customizing ActivityTrackPoint: Like the WorkflowTrackPoint class, ActivityTrackPoint is used to match activity execution status events. ActvityTrackingLocation provides a list of interested execution status events raised from activities. Use the ActvityType and ActivityTypeName properties to program the runtime to extract data for a particular activity in the workflow instance.

Note: To extract data, at least one of the execution status events should be matched and all tracking conditions must be satisfied.

The following sample code adds code activity track points to the tracking profile object.

Add Activity Track Points

Copy Code

//Create instance of Activity Track points ActivityTrackPoint activityTrackPoint = new ActivityTrackPoint(); //Create instance of Activity tracking locations ActivityTrackingLocation activityTrackingLocation = new ActivityTrackingLocation(); //Specify the activity type to be tracked activityTrackingLocation.ActivityType = typeof(CodeActivity); //Set the MatchedDerivedTypes property to true to match the activities //derived from the reference activity type. activityTrackingLocation.MatchDerivedTypes = true; //Get the list of Activity Execution status events. IEnumerable<ActivityExecutionStatus> activityExecutionStatus = Enum.GetValues(typeof(ActivityExecutionStatus)) as IEnumerable<ActivityExecutionStatus>; foreach (ActivityExecutionStatus status in activityExecutionStatus) { activityTrackingLocation.ExecutionStatusEvents.Add(status); } activityTrackPoint.MatchingLocations.Add(activityTrackingLocation); //Add activity track points to tracking profile trackingProfile.ActivityTrackPoints.Add(activityTrackPoint);

4. Customizing UserTrackPoint: UserTrackingLocation defines the parameters to be matched for derived activity types by specifying the following:
   1. Type of user data
   2. Type of activity from which an event is emitted from a workflow instance
   3. Set of conditions for filtering emitted data from workflow instance
   4. Optional key name with which user data must be associated

Match the activities derived from the reference activity type by setting MatchedDerivedTypes property to true.

The following sample code creates and adds user track points to the tracking profile object.

Add User Track Points

Copy Code

//Create instance of User Track point UserTrackPoint userTrackPoint = new UserTrackPoint(); //Create instance of User Tracking Location UserTrackingLocation userTrackingLocation = new UserTrackingLocation(typeof(string), typeof(Activity)); //Set the MatchedDerivedTypes property to true to match //the activities derived from the reference activity type. userTrackingLocation.MatchDerivedActivityTypes = true; userTrackPoint.MatchingLocations.Add(userTrackingLocation); //Add user track points to tracking profile trackingProfile.UserTrackPoints.Add(userTrackPoint);

Note: You can modify the tracking profile during runtime without making any changes in the code since you only need to deploy the changed version of tracking profile.

4.1.3. Using Tracking Profile Designer.exe

Windows SDK provides Tracking Profile Designer.exe — a tool to create tracking profiles using the workflow designer surface. To create a tracking profile, select an activity and the required tracking event (like initialized, executing, closed, and so on).

Note: You can access the Tracking Profile Designer.exe from

C:/Program Files/Microsoft SDKs/Windows/v6.0/Samples/WFSamples/Applications/TrackingProfileDesigner/TrackingProfileDesigner.exe

Note: To create custom tracking profiles (using the Tracking Profile object Model) you need to first understand the profile object model.

The following diagram gives snapshot of tracking profile designer.

You can also add annotations and extract data and conditions from a tracking profile. After creating a tracking profile, you can save it in a database or in a file as an XML.

To view the generated tracking profile XML, use the Tracking Profile Markup tab in the application.

The following is a sample tracking profile (the object is serialized) in XML format which contains a list of all workflows and activity events:

4.1.4. Monitoring User Track Events

To monitor user track events, you have to program the workflow to emit required events or data. Use either public fields or properties to avoid additional coding to emit user track events. Tracking Extract is used to extract the fields and properties from a workflow instance.

Note: To use fields and properties in Tracking Extract, their access levels should be public.

ActivityDataTrackingExtract and WorkflowDataTrackingExtract classes specify the fields and properties to be extracted from a workflow instance during an event of an activity or a workflow instance respectively.

These classes can be included in either ActivityTrackPoint or UserTrackPoint to instruct the runtime tracking infrastructure to extract a field or property from the associated activity.

Note: The runtime must match an ActivityTrackPoint or UserTrackPoint to extract a field or property from the associated activity.

This extracted field or property is returned to tracking service as ActivityTrackingRecord or UserTrackingRecord.

The code snippet below does the following:

1. Creates and adds workflows and activity data track extracts to activity track points
2. Adds activity track points to a tracking profile

Add Activity Data Track

Copy Code

WorkflowDatatrackingExtract workflowDatatrackingExtract = new WorkflowDatatrackingExtract(); workflowDatatrackingExtract.Member = "Name"; ActivityDatatrackingExtract activityDatatrackingExtract = new ActivityDatatrackingExtract(); activityDatatrackingExtract.Member = "Name"; activityTrackPoint.Extracts.Add(workflowDatatrackingExtract); activityTrackPoint.Extracts.Add(activityDatatrackingExtract); trackingProfile.ActivityTrackPoints.Add(activityTrackPoint);

4.1.5. Activity tracking Conditions

Workflows can have more than one similar type of activity (CodeActivity, If-Else Activity, and so on).

To extract data from a particular activity, you can define conditions in the tracking profile. This provides granular level control over the information that needs to be tracked.

Use the ActivityTrackingCondition class to define conditions in tracking profiles.

During runtime, tracking infrastructure evaluates conditions and if they are satisfied, track points specified in the profile will be tracked.

The following sample code defines conditions for activity tracking such that data is tracked if the "Name" of the code activity is equal to "CodeActivity1".

"Name" = "CodeActivity1"	Copy Code
ActivityTrackingCondition activityTrackingCondition = new ActivityTrackingCondition(); activityTrackingCondition.Member = "Name"; activityTrackingCondition.Value = "CodeActivity1"; activityTrackingCondition.Operator = ComparisonOperator.Equals; activityTrackingLocation.Conditions.Add(activityTrackingCondition);

4.1.6. Annotating TrackPoints

If a considerable amount of data is tracked for a workflow instance then it becomes difficult to correlate the data with the business process. To overcome this problem, tracking profile provides annotations to attach extra information with the track points. This helps describe the track points in order to determine which activity in the workflow emitted the track events. Annotations can be attached to all three track points. Runtime tracking Infrastructure includes both emitted data from the workflow instance and annotation collection from the tracking profile object in the tracking record and sends it to the tracking service.

Runtime tracking infrastructure encapsulates each extracted field or property and its associated annotation in trackingDataItem and adds it to ActivitytrackingRecord.Body or UsertrackingRecord.Body.

The following sample code adds annotations to track points.

Add Track Point Annotations	Copy Code
//Workflow track point annotation workflowTrackPoint.Annotations.Add("WorkflowTrackPoint Annotation."); //Actvity track point annotation activityTrackPoint.Annotations.Add("ActivityTrackPoint Annotation."); //User track point annotation userTrackPoint.Annotations.Add("User track point annotation");

4.1.7. Tracking User Data

Activity status event and workflow status events are emitted only when there is a change in the status of a workflow instance. To track, any other data, other than the status changes, in a workflow, use User Event.

Activity.trackData and ActivityExecutionContext.trackData method overloads emit data, as a user event, to the runtime tracking infrastructure. These methods help monitor business data.

The UserTrackingLocation class defines the location (combination of specific type of user data and type of source activity from which it is emitted). It also defines a set of conditions for filtering tracking records. If there is more than one activity of same type and UserTrackingLocation is defined for that type of activity, the resultant tracking will be applicable to all those activities. Further filtering can be done at the activity's name level.

The following data needs to be matched by the runtime tracking infrastructure for the UserTrackingLocation class:

• The activity that emits a user event must match the set of activities specified by ActivityType, ActivityTypeName, Conditions and MatchDerivedActivityTypes.
• The key associated to user data must match keyName
• The type of user data emitted by the user event must match the set of user data types specified by ArgumentType, ArgumentTypeName and MatchDerivedArgumentType.

Typically a workflow is built using policies which consists of one or more rules. These policies define the business rules in workflows which represent critical business conditions. It is very important to track the outcome of the rules in a policy. Workflow instance emits these rules results to the runtime infrastructure.

To track the rule evaluation results, tracking profile should contain the following objects:

• UserTrackPoint
• ActivityType — Either PolicyActivity or Activity type
• ArgumentType — should be RuleActionTrackingEvent

A workflow can have more than one policy activity. By specifying the policy activity name, you can extract the rules results of a particular policy activity.

The following code snippet gets tracking events from the policy execution:

Get Tracking Events

Copy Code

//Create instance of User track point to track user track points UserTrackPoint userTrackPoint = new UserTrackPoint(); //Create instance of User Tracking Location UserTrackingLocation userTrackingLocation = new UserTrackingLocation(); //To get the tracking events from the policy execution specify the //activity type as PolicyActivity and Argument as RuleActionTrackingEvent userTrackingLocation.ActivityType = typeof(PolicyActivity); userTrackingLocation.ArgumentType = typeof(RuleActionTrackingEvent); userTrackingLocation.ActivityTypeName = "PolicyActivity1";

4.2. Creating a Custom Tracking Service

The out-of-box SqlTrackingService provides excellent tracking services. However, applications often require custom tracking services to work with custom data stores.

4.2.1. Pre-requisites for Creating Custom Tracking Services

The following are pre-requisites for creating custom tracking data:

• A custom tracking service class should derive from the TrackingService base class.
• A custom tracking channel class should derive from TrackingChannel base class.

4.2.2. Creating Custom Tracking Service Class

The TrackingService base class contains several abstract methods.

The following abstract methods of TrackingService base class must be overridden in the derived class. Runtime tracking infrastructure makes a call to these methods to retrieve custom tracking profile and custom tracking channel objects.

When the workflow instance is loaded for the first time, runtime tracking infrastructure gets the tracking profile from the storage medium like database or file and caches the profile in the memory. For subsequent workflow instances, runtime gets the profile from the cache avoiding database and file access. Tracking service class has to implement IProfileNotification to notify the runtime tracking infrastructure to reload the cache whenever the profile in the storage medium gets updated or removed.

4.2.3. Creating Custom Tracking Channel Class

To create a custom tracking channel for a tracking service, derive a class from TrackingChannel base class.

The following figure illustrates the Tracking Channel class diagram:

To transfer data to the tracking service, runtime tracking infrastructure calls GetTrackingChannel to get a channel object for a workflow instance. This class passes the TrackingParameters to the tracking channel object. To store tracking records, runtime tracking infrastructure uses the overridden Send method of the custom tracking channel class. To persist tracking data at the end of transaction scope, tracking channel has to implement IPendingWork interface. Send method adds the tracking data item to the current work batch using a method "WorkflowEnvironment.WorkBatch.Add". Work batch is a collection of work items associated with the workflow instance. When the workflow persists, Commit method of IPendingWork interface will be called and tracking data will be saved to the storage medium.

5. Using End-to-end Tracing

Tracing is a useful tool for monitoring the execution process of an application during runtime. Use information tracing to resolve technical issues while debugging the application.

End-to-end tracing records the following information:

• Errors
• Warnings
• Execution status of activities and workflow instances

5.1. Start an End-to-end trace

WF provides five trace switches to track information on component execution.

To enable tracing in the workflow, the following configurations need to be added to the application configuration file:

• Set the Value attribute — The amount of data that is traced is controlled by this attribute. The possible values are:
  • All
  • Off
  • Critical
  • Error
  • Warning Information and
  • Verbose.
• Set the value of LogToTraceListeners to 1 — Tracing information is written to the listener that is specified by the host application. For example, TextWriterTraceListener.
• Set the LogToTraceListeners to 0 — Tracing information is not logged. The default tracing mode for a workflow is off.

The following code snippet shows a configuration for End-to-end tracing which can be inserted into the application configuration file.

End-to-end Tracing

Copy Code

<?xml version="1.0" encoding="utf-8" ?> <configuration> <system.diagnostics> <switches> <add name="System.Workflow LogToTraceListeners" value="1" /> <add name="System.Workflow.Runtime" value="All" /> <add name="System.Workflow.Runtime.tracking" value="All" /> <add name="System.Workflow.Runtime.Hosting" value="All" /> <add name="System.Workflow.Activities" value="All" /> <add name="System.Workflow.Activities.Rules" value="All" /> </switches> <trace autoflush="true" indentsize="4"> <listeners> <add name="TraceListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="WFtrace.log" /> </listeners> </trace> </system.diagnostics> </configuration>

6. Using WF Tracking Performance Counters

Apart from tracking Infrastructure and End-to-end tracing, WF provides performance counters to monitor the status of workflow instances. Monitoring workflows through performance counters is done for administrative purposes.

WF performance counters are enabled by default.

You can monitor the following counters through performance monitor (perfmon.exe):

The following describes parameters of the code used while configuring performance counters:

1. Workflow performance counters can be switched on by setting the EnablePerformanceCounters attribute to true in the workflow configuration file.
2. Performance counters can be disabled by using EnablePerformanceCounters properties of the WorkflowRuntimeSection class.

The following code snippet is a section of the code for enabling performance counters:

Enabling Performance Counters

Copy Code

<?xml version="1.0" encoding="utf-8" ?> <configuration> <configSections> <section name="CustomWorkflowRuntimeSettings" type="System.Workflow.Runtime.Configuration.WorkflowRuntimeSection, System.Workflow.Runtime, Version=3.0.00000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" /> </configSections> <CustomWorkflowRuntimeSettings Name="Workflow1" EnablePerformanceCounters ="true"> </CustomWorkflowRuntimeSettings> </configuration>

7. Displaying Tracked Data

WF allows you to develop monitoring tools to view the following:

1. Workflow status events
2. Activity status events
3. User data

Use the following two ways to view tracked data:

1. Write a customized program
2. Use workflowmonitor.exe — a sample monitoring application provided by Windows SDK

7.1. Write a Customized Program

WF provides out-of-box SqlTrackingService and SqlTrackingQuery APIs to provide high level access to tracked data in SQL server. These APIs need to be called to get tracking data.

Use the out-of-box SqlTrackingQuery APIs for accessing tables, writing logic to retrieve data in stored procedures, handling database connections, and so on.

For detailed monitoring, build custom monitoring tools using ADO.NET to query the SqlTracking database.

The SqlTrackingQuery class provides GetWorkFlows and TryGetWorkFlow methods to retrieve data from the tracking database.

These methods retrieve information on tracked workflow instances and return SqlTrackingWorkFlowInstances object.

Use the SqlTrackingQueryOptions parameter of the GetWorkFlows method to specify filter conditions to retrieve information. For example, workflows which have been started, completed, suspended, and so on.

The following figure illustrates the classes needed for writing customized programs:

The following code snippet demonstrates the process of retrieving data using the SqlTrackingQuery class:

SqlTrackingQuery Class

Copy Code

//Create a typed Sql tracking Workflow Instance list object List<SqlTrackingWorkflowInstance> workFlows = new List<SqlTrackingWorkflowInstance>(); SqlTrackingQuery sqlTrackingQuery = new SqlTrackingQuery(connectionString); SqlTrackingQueryOptions sqlTrackingQueryOptions = new SqlTrackingQueryOptions(); sqlTrackingQueryOptions.StatusMinDateTime = DateTime.UtcNow.AddHours(-5); sqlTrackingQueryOptions.StatusMaxDateTime = DateTime.UtcNow; sqlTrackingQueryOptions.WorkflowStatus = WorkflowStatus.Completed; workFlows.AddRange(sqlTrackingQuery.GetWorkflows(sqlTrackingQueryOptions));

To get a particular instance of workflow, use TryGetWorkFlow. It takes workFlowInstanceId as the input parameter and returns an instance of SqlTrackingWorkflowInstance as the output parameter, as follows:

  sqltrackingQuery.tryGetWorkflows(workflowInstanceId, 
                                   out sqltrackingWorkflowInstance);

TrackingDataItemValue class is used to specify the "where clause" for retrieving data from the SQL database (refer to the code sample below).

TrackingDataItemValue object contains a set of filter conditions for extracting data from a workflow instance. This object is added to SqlTrackingQueryOptions.trackingDataItem.

SqlTrackingQueryOptions.TrackingDataItem

Copy Code

TrackingDataItemValue TrackingDataItemValue = new TrackingDataItemValue(string.Empty, string.Empty, string.Empty); //Set the Qualified name of the activity, activity member from which //data has to extracted and the value of the extracted data. TrackingDataItemValue.QualifiedName = "CodeActivity"; TrackingDataItemValue.FieldName = "Quantity"; TrackingDataItemValue.DataValue = "100"; sqlTrackingQueryOptions.trackingDataItems.Add(trackingDataItemValue);

Post accessing tracked data, use the following to view the tracked data in a meaningful way:

• Custom components
• Stored procedures
• Views provided by the SqlTrackingService

SQL Reporting Services can be used to analyze tracked data in the SQL Server Database.

7.2. Workflow Monitor — A Sample Monitoring Application provided by Windows SDK

Windows SDK provides Workflow Monitor — a sample monitoring application that is built using SqlTrackingQuery APIs.

Note: Access Workflow Monitor from: C:\Program Files\Microsoft SDKs\Windows\v6.0\Samples\WFSamples\Applications\WorkflowMonitor\WorkflowMonitor.exe

With Windows SDK tool, you can monitor workflow instances in real time by specifying the polling time interval in the settings options.

This tool polls the tracking database at specified time intervals and updates the tracking information in real time.

You can do the following in the sample monitoring application:

1. View the user tracked events.
2. View data for a particular date and time range.
3. Filter the search result by specifying the workflow instance ID or by specifying the activity name, property name and value.

Use the workflow designer control of this application to view all workflow instances tracked by SQL tracking database.

The display of this tool contains three panes as shown in the following figure.

• Top left pane: Lists workflows tracked in the tracking database and the status of the workflow.
• Lower left pane: Provides activities for the selected workflow.
• Right pane: Provides the workflow designer control.

Note: To display workflows and activities in the GUI, all workflows and activities should be placed in the same directory as the monitoring application executable.

Following diagram gives snapshot of Workflow Monitor:

8. Performance Considerations for Tracking

While it is extremely critical for an application to be monitored continuously, we have to understand the side effects of such constant monitoring.

Adding tracking services to workflow instances creates overheads on the performance of an application. The performance decreases with an increase in the amount of tracked data.

For each instance of the workflow, tracking runtime has to do the following:

• Evaluate corresponding tracking profile
• Filter emitted data from the workflow instance
• Call stored procedures to store the filtered data in the database.

Performance considerations for Transactions in WF monitoring: SqlTrackingService supports transaction in tracking data to the database. The default value of the property IsTransactional is set to true, that is workflow instance tracking events are written to the database in a single batch.

• This kind of tracking provides better performance since tracking data is flushed to the database when the workflow instance in finished or unloaded.
• This may also have negative impact on the performance in case of long running workflows (when coded without persisting) and if the workflow contains lots of tracking events, since all the tracking events are flushed to the database at the end. This consumes more memory and time to make entry in the database. The tracking data is not available to view until the workflow instance is finished and hence cannot make business decisions based on current executing or active workflow instance. In order to avoid such penalty it is recommended to use persistent points at logical completion of the work in the workflows.

If the IsTransactional property is set to false, data is written to the database for each track point. In this case for each and every track point, database call is made and data will be written to the database. This may impact the performance of the workflow depending on the number of track points in the workflow

Performance considerations for data growth: The volume of data tracked will have significant impact on the disk consumption and growth of the database. It is important to monitor tracking database for the disk consumption. The tracking database needs to be regularly cleaned or archived.

WF provides a data maintenance feature for SqlTrackingService. In this case, completed workflow instance data is moved to the corresponding partitioned tables.

SqlTrackingService provides the following two schemas for partitioning:

• Partition on Completion
• On Demand Partition

8.1. Partition on Completion

This schema is used when an application has no downtime or cannot afford a downtime.

The following lists the characteristics of this schema:

1. When the workflow is active, data is tracked in regular tracking tables.
2. When workflow instance gets completed, tracked data from the regular tables is moved to the partitioned tables without disrupting the other workflow instances. This will add up extra work at the end of every workflow instance completion.

To use this schema, set the property PartitionOnCompletion ofSqlTrackingService class to true (it is set to false, by default). A new set of partitioned tables are created periodically. This can be programmatically set using a stored procedure SetPartitionInterval.

The default partition interval is monthly, but it can be changed to daily or yearly.

The following code snippet creates a tracking service and sets the PartitionOnCompletion property.

PartitionOnCompletion Property	Copy Code
SqlTrackingService sqlTrackingService = new SqlTrackingService(connectionString); sqlTrackingService.PartitionOnCompletion = true;

Whenever partitioned tables are created or dropped, tracking views will be updated. By using tracking views, user can query tracked data from all the portioned tables.

8.2. On Demand Partition

This can be used when an application have a scheduled downtime and partitioned can be done during the downtime. By executing the PartitionCompletedWorkflowInstances stored procedure, all completed workflow instance tracked data can be moved to the partitioned tables.

	Copy Code
EXEC PartitionCompletedWorkflowInstances

Users can drop the old partitioned tables by executing the stored procedure DropPartition.

9. Comparing WF Monitoring with BizTalk Server (BTS) 2006 BAM features

WF provides a robust tracking infrastructure to trace various technical and business data. However, unlike BTS, there is no integrated support to flexibly configure the required data using GUI wizards or analyze the retrieved data. However, it is always possible to use SQL Reporting Service to create reports on tracked information and analyze them.

BizTalk Server, which belongs to the Microsoft server family, provides the following set of tools for monitoring business processes:

• BAM Definition Excel workbook: Helps business analysts to list business data that needs to be extracted
• A deployment utility: Creates the database infrastructure using an XML file that is generated by the BAM definition excel workbook.
• Portal/dashboard kind of utility for end users: Analysis of business data.
• BizTalk Server 2006 R2 release provides BAM tracking interceptor for Workflow Foundation. BizTalk BAM features like aggregation, tracking and alert can be extended to Workflow Foundation by implementing BAM interceptor in workflows.

WF however, does not provide tools to achieve BAM, but developers can build custom components and services to achieve BAM. WF SDK provides the following set of sample applications:

• Workflow tracking profile designer: Creates custom tracking profiles.
• Workflow monitoring tool: Displays workflows and provides information on the activity status.

You can modify these applications according to business requirements.

10. Conclusion

As we have seen, WF already provides an extensible mechanism to track down the data of workflow applications but still requires some effort from developers to extend that further to dig out the critical business details of the applications. This paper detailed the overview of existing mechanism and how it can be extended will help the developers to go beyond further to create true, robust BAM tool in future. This paper covers the details of customizing Workflow tracking mechanism and details various useful tools/samples provided by WF SDK, It also discusses performance considerations for tracking.

Author Profiles

Ganesan Krishnamurthy is a Senior Technical Architect with Microsoft Technology Center, Infosys. He works on understanding and building expertise in .Net 3.0 technologies.

Prashanth B. G. is an Senior Technical Specialist with Microsoft Technology Center (Infosys Technologies Ltd). He is currently working on the creation of integration solutions on the Microsoft platform using technologies like BizTalk Server, .NET Framework 3.0, Windows SharePoint Services and SQL Server 2005.

Acknowledgements

Joel West is a Senior Software Development Engineer in the Connected Systems Division at Microsoft focusing on the hosting and runtime aspects of workflow.

Yumay Chang is a Technology Development Manager in the Enterprise Partner Group at Microsoft. She leads the partner strategy on SOA and Business Process Management and supports partner in their adoption of Microsoft technologies in this space.

Related Information

1. MSDN WF developer site
   http://msdn2.microsoft.com/en-us/netframework/aa663328.aspx
2. Community Site
   http://wf.netfx3.com/
3. MSDN sample codes
   http://msdn2.microsoft.com/en-us/library/ms741723.aspx
4. Performance white paper
   http://msdn2.microsoft.com/en-us/library/bb264458(vs.80).aspx
5. General Reference
   http://msdn2.microsoft.com/en-us/library/ms732093.aspx
6. WF introduction
   http://msdn2.microsoft.com/en-us/library/bb264459(vs.80).aspx
7. WF Deep dive
   http://msdn2.microsoft.com/en-us/library/bb264458(vs.80).aspx