Microsoft BizTalk Server 2004 is a server application that enables businesses to integrate disparate, heterogeneous applications, and to automate business processes. BizTalk Server 2004 handles all data communication between the underlying Microsoft Windows Server operating system and Microsoft SQL Server database on behalf of all external applications, services, processes, and systems.

To enable the integration of disparate applications and the coordination of logic between business processes, BizTalk Server 2004 transforms and persists all the messages in the MessageBox database on SQL Server. BizTalk Server uses the Publish / Subscribe metaphor for this integration.

The following figure shows the message flow in BizTalk Server 2004.

Figure 1 Message flow in BizTalk Server 2004

1. BizTalk Server receives messages from external applications, services, processes, and systems. BizTalk Server uses receive pipelines to convert messages from their external format to XML data and persists these messages to the MessageBox database. This completes the "Publishing" of messages.
   
2. BizTalk Server may then process the messages with orchestrations by using the Business Rule Engine with these processes. These processes are considered as "Subscribers" to the messages that were published to the MessageBox database. The processes may have a need to send the message to other applications or to trading partners, which results in the orchestration processes "publishing" a new message to the MessageBox database.
   
3. BizTalk Server uses send pipelines to convert the XML data to their external format. The Send adapter then sends the messages to their external applications, services, processes, and systems.
   

The document explains how the various BizTalk Server adapters handle these transactional semantics.

It will also describe how and when to use atomic and long running transactions associated with business processes.

Adapters in BizTalk Server are the first and last components that typically touch messages received and sent through BizTalk Server. Out of the box, BizTalk provides File, FTP, SQL, HTTP, SOAP, SMTP, Message Queuing, and Base EDI adapters.

This section covers how the transactional semantics apply to messages when they are processed by the following BizTalk adapters. You can use all the adapters to send and receive messages.

Transactions guarantee one-time delivery at a cost. For non-transactional adapters, in a single MessageBox database scenario, the messaging engine uses SQL transactions versus creating COM+ transactions using MSDTC. In these scenarios, however, if there were any system failure while processing, the system could receive or send the same message twice. With transactional adapters (SQL/mySAP/MQ Series/MSMQ), the engine will always use COM+ DTC transactions, since the transactions span multiple systems—the message box(es) and the source or destination of the message. The adapter always owns the transactions and is responsible for the commit or rollback of the transaction; the engine simply votes.

In high throughput scenarios there might be a need to scale out the MessageBox database layer that BizTalk uses. In a multi-message box topology the agent/engine automatically uses DTC and any performance boost achieved by adding the second message box is offset by distributed transactions. Only when a third message box is added does it really improve performance. Hence, the general recommendation is to go from a single message box topology to three message boxes (not to two).

There are special cases when the messaging engine automatically uses a DTC. This is when the engine is processing large files or large batches (batches containing multiple messages whose aggregate datasize is greater than a configured threshold). This is a configured threshold that defaults to 100K.

The following sections cover how the transactional semantics apply to messages when they are processed by the following BizTalk adapters. See BizTalk Server 2004 Help for more information about configuring and using the adapters.

There are several native adapters for BizTalk Server. These include:

• File adapter
  
• SQL adapter
  
• Message Queuing adapter
  
• FTP adapter
  
• HTTP adapter
  

File Adapter

SQL Adapter

<UpdateGramRootElement>
   <sync>
      <before>
         <TableName col1='somevalue' col2='somevalue' />
      </before>
      <after>
         <TableName col1='somevalue' col2='somevalue' />
      </after>
   </sync>
</UpdateGramRootElement>

Message Queuing Adapter

FTP Adapter

HTTP Adapter

There are other Microsoft adapters and custom adapters in addition to the native adapters. These include:

• MQSeries adapter 2.0
  
• MSMQ adapter
  
• mySAP 2.0 adapter
  
• Custom adapters
  

MQ Series Adapter 2.0

MSMQ Adapter

mySAP 2.0 Adapter

Custom Adapters

There are scenarios where pipeline components need to enlist in transactions, and either commit the results or roll back depending on whether the messages are successfully posted to the MessageBox database.

The support is for MSDTC transactions and applies to transactional and non-transactional receive adapters. Send adapters are currently not supported.

For a non-transactional adapter, the DTC transaction is initiated by the messaging engine. For transactional adapters, the DTC transaction is initiated by the receive adapter and passed to the engine and then to the component. The transaction applies to the transport batch. Any other component from any receive pipeline that processes the messages from this transport batch can avail of this transaction. If all the messages in the batch are successfully submitted, the transaction is committed by the engine. The messaging engine rolls back the transaction if submission of any of the messages in the batch fail. For non-transactional adapters, if the pipeline component throws an exception, the engine rolls back the batch submission and reports a submission failure to the receive adapter. For a transactional adapter, the engine reports the batch failure to the receive adapter and the receive adapter is responsible for rolling back the batch. A transactional pipeline component has to query the IPipelineContextEx interface and use the GetTransaction method to get the transaction context as detailed in the following example.

The following code example demonstrates a transactional pipeline component that gets the transaction from the EPM and then updates the database within that transaction. This sample assumes that the schema elements that are needed in the database call are Promoted Properties:

public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inMsg)
{
string customerId;
string companyName;

// Get the customer Id and companyName from the property schema
IBaseMessageContext ctx = inMsg.Context;
customerId = ctx.Read("customerID", 
"http://somepropertyschema").ToString();

companyName = ctx.Read("companyName", "http://somepropertyschema")).ToString();

IPipelineContextEx pcx = pc as IPipelineContextEx;

if (pcx != null)
{
ITransaction trans = null;
trans = (ITransaction)pcx.GetTransaction();

if (trans != null)
{
SqlConnection nwindConn = 
new SqlConnection(
"Data Source=localhost;Integrated Security=SSPI;" +
"Initial Catalog=Northwind;Enlist=false;");
SqlCommand cmd = 
new SqlCommand(
"INSERT INTO Customers (CustomerID, CompanyName) " +
"Values(@CustomerId, @CompanyName)", nwindConn);
    
cmd.Parameters.Add("@CustomerId", SqlDbType.VarChar, 5).Value = customerId;
cmd.Parameters.Add("@CompanyName", SqlDbType.VarChar, 
40).Value = companyName;
nwindConn.Open();
pc.ResourceTracker.AddResource(nwindConn);
nwindConn.EnlistDistributedTransaction(trans);
try
{
cmd.ExecuteNonQuery();
}
catch (SqlException e)
{
throw(e);
}
}
else
throw new ArgumentNullException("IPipelineContextEx:GetTransaction();
}
return inMsg;
}

BizTalk Server provides a process execution engine that manages the state, applies business logic, and invokes the supporting applications of complex processes and/or transaction sets. Business processes can be composed as discrete pieces of work using atomic transactions that automatically roll back all changes in case of errors or long running, which can contain nested transactions and use custom exception handling to recover from error scenarios. These transactional semantics are typically managed through the Scope construct in the Orchestration Designer. The long running processes can span days, weeks, and longer time durations. The long running processes typically utilize correlation to correlate messages that are received to the messages that may be sent. The orchestration engine typically dehydrates these instances to conserve system resources and re-hydrates the process upon receipt of these correlated messages. The orchestration engine persists the orchestration state to the MessageBox database at known checkpoints to recover from any application or system exceptions.

The orchestration engine saves the entire state of an orchestration instance at various persistence points to allow rehydration of the orchestration instance. The state includes any .NET-based components that may be used in the orchestration, in addition to messages and variables. The engine stores state at the following persistence points:

• End of a transactional scope (atomic or long running)
  
• At debugging breakpoints
  
• At the execution of other orchestrations through the Start Orchestration shape
  
• At the Send shape (except in an atomic transaction)
  
• When an Orchestration Instance is suspended
  
• When the system shutdowns in a controlled manner
  
• When the engine determines it wants to dehydrate
  
• When an orchestration instance is finished
  

The engine optimizes the number of persistence points as they are expensive, especially when dealing with large message sizes. As shown in the two orchestration instances below, in the orchestration with the Send Shapes in an atomic scope, the engine determines a single persistence point between the end of the transaction scope and the end of the orchestration. In the other orchestration, there will be two persistence points, one for the first Send shape and the second for the Send shape plus end of the orchestration.

Figure 2 Orchestration persistence

Any .NET-based objects you use in orchestrations, either directly or indirectly, must be marked as serializable unless they are invoked in atomic scopes, or if the objects are stateless and are invoked only through static methods. System.Xml.XmlDocument is a special case and does not need to be marked as serializable regardless of the transaction property for a scope.

How does the special handling for System.Xml.XmlDocument work:

When the user defines a variable X of type T, where T is System.Xml.XmlDocument or a class derived from System.Xml.XmlDocument then the compiler will treat X as a serializable object.

When serializing X, the runtime will keep the following pieces of information: (a) the actual type Tr of the object X is referring to (b) the OuterXml string of the document.

When de-serializing X, the runtime will create an instance of Tr (this assumes a constructor that does not take any parameters) and will call LoadXml providing the instance with the saved OuterXml. X will then be set to point to the newly created instance of Tr.

The orchestration engine uses the concept of correlation to match incoming message to an orchestration instance. The engine may spawn different orchestration instances, each of which deals with instance-specific data. Correlation needs to be used in scenarios where there might be a need to send/receive messages asynchronously, especially in the case of long running processes. The engine uses implicit correlation when dealing with request/response scenarios.

Correlation is achieved by creating correlation types and correlation sets based on the correlation type. A correlation type is a list of properties that can be picked from either the application-specific property schema or the GlobalPropertySchema used by BizTalk. It is important to note that the application-specific properties have to be promoted properties and will not appear in the Correlation Properties dialog box if they are identified as distinguished fields. The following figure shows the Correlation Properties dialog box. As is shown, the available properties by default include all the properties from the GlobalPropertySchema.

Figure 3 - Correlation Properties dialog box

A correlation set is a collection of promoted message property values that you use to route an incoming message to an executing instance. The message is evaluated for the message type and every property in the set in order to correctly route it to the instance.

Sample Scenario:

The following orchestration displays a scenario where a message is:

• Received by an Activating Receive shape that starts the orchestrations
  
• Transforming and sending the message out through a certain transport. The correlation set will be initialized in this shape using some promoted properties that can uniquely identify the message.
  
• Receiving a response message back from a transport other than the send transport. The shape is defined as "Following Correlation Sets" that was initialized in the preceding Send shape.
  

Figure 4 Orchestration scenario

It is important to note that if the same scenario had been constructed using a request-response send port, the engine would create an implicit "Initializing Correlation Sets" for the Send, and "Following Correlation Sets" for the Receive shape.

Compensation can be thought of as a process to do a logical undo of the work that has been successfully committed in response to some error condition. This is handled in orchestrations through the use of compensation handlers. Compensation handlers can only be associated with transactional scopes (long running or atomic). It is important to understand that the compensation handler of a scope is invoked only if the scope completed successfully, but then is required to be undone because a surrounding scope has decided to abort (due to failures that may occur later in the process).

The orchestration engine defines clear semantics about the information that is available to a compensation handler:

• It gets the state of the running instance exactly as it existed at the point at which the associated scope was successfully completed. This includes the state of messages, variables, objects, and so on.
  
• A compensation handler cannot influence the global scope of the executing process. Any modification to the global state is only visible locally.
  

A compensation handler for a particular scope is always invoked by either the compensation or exception handler of the surrounding scope. The default compensation process for nested transactions has the effect of running the compensation handler for each of the nested transactions in the reverse order of how they were committed.

The orchestration engine provides a Compensate shape for invoking specific compensation handlers. If the Compensate shape targets itself (also known as self compensation). The effect is the same as the default compensation. After the compensation handler completes, the control transfers back to the statement immediately following the Compensate shape taht triggered the original compensation. In a nested scope scenario, if the Compensate shape targets a child compensation handler, and if the child scope does not have a user-specified compensation handler, a default system-provided one is run. A particular scope can be compensated only once. The compiler will prevent more than one path leading to the compensation of a particular scope.

The entire orchestration can have a custom compensation handler. It is set by setting the Compensation property for the orchestration to Custom. This has the effect of an additional Compensation tab next to the default Orchestration tab in the Orchestration Designer. An empty sheet is created with all the Port shapes as they exist in the orchestration being compensated. The custom compensation handler is useful to compensate completed orchestration instances and applies primarily for called orchestrations. The custom compensation does not run for stand alone orchestrations.

Samples for compensation usage are provided with scenarios for long running transactions as they typically are used with long running processes where there is a need to undo logical units of work that have completed due to errors in the process.

BizTalk orchestrations can be designed to run discrete pieces of work, following the classic 'ACID' concept of a transaction. These discrete or atomic units of work, when performed, move the business process from one consistent state to a new, consistent and durable state, and that are isolated from other units of work. This is typically done by using the Scope construct that encapsulates the units of work with the transactional semantics. The entire orchestration can also be defined as an atomic transaction without the use of scopes. The scopes, however, cannot be marked as transactional unless the orchestration itself is marked as a long running or atomic transaction type.

Atomic transactions behave like classic MSDTC transactions, but they are not explicit DTC transactions by default. The following isolation levels are supported by the atomic transactions:

• Read Committed
  Shared locks are held while the data is being read to avoid dirty reads, but the data can be changed before the end of the transaction, resulting in non-repeatable reads or phantom data.
  
• Repeatable Read
  Locks are placed on all data that is used in a query, preventing other users from updating the data. This prevents non-repeatable reads, but phantom rows are still possible.
  
• Serializable
  A range lock is placed preventing other users from updating or inserting rows into the database until the transaction is complete.
  

When the atomic transaction fails, all states are reset as if the orchestration instance never entered the scope. The rule BizTalk has for atomic transactions is that all variables (not just local to the scope) participate in the transaction. All non-serializable variables and messages used in an atomic transaction should be declared local to the scope; otherwise, the compiler will give the "variable….is not marked as serializable" error. All atomic scopes are assumed to be "synchronized" and the orchestration compiler will provide a warning for redundant usage, if indeed the synchronized keyword is used for atomic scopes. The synchronization for the shared data extends from the beginning of the scope until the successful completion of the scope (including the state persistence at the end of the scope) or to the completion of the exception handler (in case of errors). The synchronization domain does not extend to the compensation handler.

Atomic transactions can be associated with timeout values at which point the orchestration will stop the transaction and the instance will be suspended. If an atomic transaction contains a Receive shape, a Send shape, or a Start Orchestration shape, the corresponding actions will not take place until the transaction has committed.

An atomic transaction retries when the user deliberately throws a RetryTransactionException or if a PersistenceException is raised at the time that the atomic transaction tries to commit. The latter could happen if for instance, the atomic transaction was part of a distributed DTC transaction and some other participant in that transaction stopped the transaction. Likewise, if there were database connectivity problems at the time when the transaction was trying to commit, there would also be a PersistenceException raised. If this happens for an atomic scope that has Retry=True, then it will retry up to 21 times. The delay between each retry is two seconds by default (but that is modifiable). After 21 retries are exhausted, if the transaction is still unable to commit, the whole orchestration instance is suspended. It can be manually resumed and it will start over again, with a fresh counter, from the beginning of the offending atomic scope.

Note

Only the RetryTransactionException and PersistenceException can cause an atomic scope to retry. Any other exception raised in an atomic scope cannot cause it to retry, even if the Retry property is set to True. The two-second default delay between each two consecutive retries can be overridden by using a public property on RetryTransactionException (if that is the exception that is being used to cause the retries).

The atomic scopes have restrictions on nesting other transactions that cannot nest other transactions or include Catch - Exception blocks. The following scenarios describe the use of atomic transactions.

Scenario 1 - An Atomic Transaction with COM+ ServicedComponent

The following orchestration shows how to use the RetryTransactionException with atomic transactions. Although exception handlers cannot be directly included for an atomic scope, the scope can include a non-transactional scope that can have an exception handler. The ServicedComponent enlists in the same DTC transaction and any exception raised by the component is caught and re-thrown as RetryTransactionException. (This assumes that the Retry property is set to True for the atomic scope).

Note that the orchestration would have been suspended and the action in the MessageAssignment shape would have been rolled back even if the RetryTransactionException is not thrown. This pattern, however, allows building resilience in the application where the retries occur automatically.

Figure 5 An atomic transaction with COM+ ServicedComponent

Scenario 2 - Using Transacted Adapters with Atomic Transactions

The following orchestration shows how to use the atomic transactions with the SQL adapter. The whole orchestration is marked as long running with individual atomic transactions for the two logical pieces of work, Inserting a new Customer and Insert Order details for the customer. If for whatever reason, the Order Insert fails, the Customer Insert should be rolled back. The sample uses the SQL adapter to do the database work. As mentioned earlier, the scope associated with an atomic transaction completes when the message is sent to the MessageBox database. This implies that after the engine is successful in sending the message in the Scope_InsertCustomer and Scope_InsertOrder scopes, each one of the scopes commits. The SQL adapter creates a new transaction for the actual Insert of the Customer or the order. The Ports have a property “Delivery Notification” for validating that the message has been successfully sent via the Sent Port. When the Delivery Notification property is set to “Transmitted”, a receive subscription is placed before the Transactional commit point of the Send Operation. However in case of Atomic Scopes, the receive subscription is placed in the enclosing Parent scope. In the scenario where the InsertOrder SQL transaction fails, a "Nack" will be sent back and the "Scope_InsertOrder" commits. After the Sent Port exhausts the configured retries, a DeliveryFailureException will be raised This exception will be caught by the default exception handler, which will run the default compensation process. This will invoke the compensation handler associated with the Scope_InsertCustomer, causing the undo operation of inserting the customer information.

Note
Nesting the two scopes in a long running scope, and invoking the Compensate shape (targeting the long running transaction) from the exception handler for the long running scope will result in the same behavior as described above. The whole orchestration could not be marked atomic as atomic transactions do not allow nested transactions.

Figure 6 Transacted adapters with atomic transactions

Atomic transactions are usually unsuitable for ensuring transactional semantics for long running work because of all the resources that are consumed in preserving the ACID semantics. Long running transactions assist in the implementation of such scenarios by providing a framework that provides some of the ACID semantics inherently (both atomicity and isolation is lost), but provide a programmable way to achieve the rest.

Long-running transactions are similar to atomic transactions in regard to guaranteeing durability and consistency of state at the end of the transaction.

They are also fundamentally different from atomic transactions because:

• Exception handlers are permitted on long-running scopes.
  
• There is no "auto" rollback of state in case of a transaction abort. You can achieve this programmatically through the exception and compensation handlers.
  
• Long running transactions permit nesting of other transactions (long running or atomic).
  

Long running scopes can be associated with a timeout, which is a logical time within which the long-running work must complete. If the scope does not complete within the specified time, a pre-defined system exception TimeoutException is raised.

You can create long running processes by either marking the entire orchestration as long running or having an outer long running scope nest any other scopes. In the former scenario, a system-provided exception handler runs, whereas the latter allows associating specific exception handlers to the outer scope. The default system-provided exception handler will run the compensation handler for each of the successfully completed nested transactional scopes, if any, in reverse order of their completion. You can achieve the same through self compensating by using the Compensate shape in the exception handler for a long running transaction.

Scenario 1 - Using Long Running Transactions with Timeouts

The following orchestration is a representation of how to associate timeouts with long running transactions. Sometime, you may need to interface with legacy systems that operate in a batch fashion. The scenario shows a purchase order being received and sent to the legacy system. The legacy system processes the purchase order and sends back a purchase order acknowledgement. The send operation initializes a correlation set using the purchase order number and the receive operation follows that correlation set. The receive operation is also in a long running scope with a timeout value. The orchestration engine will dehydrate the orchestration instance waiting for the receive. The correlation will ensure that the same orchestration instance is invoked after the message is received. If the purchase order acknowledgement does not arrive within the time interval specified by the timeout values, a TimeoutException will be thrown.

Figure 7 Long running transactions with timeouts

Scenario 2 - Using Long Running Transactions with Custom Compensation

The following two orchestrations demonstrate how to associate and invoke custom compensations associated with entire orchestrations. The scenario is similar to Scenario 2 for Atomic Transactions; inserting a new customer and inserting the order details for the customer. The process remains the same, if the order insert fails, you should roll back the customer insert. The customer insert could be done by a legacy system, and is therefore, demonstrated in a separate callable orchestration. The called orchestration has the Custom property set for compensation, which provides a separate sheet to do the compensation process. The compensation is to delete the newly inserted customer.

The calling orchestration has a long running scope to do the order insert. This scope is nested within an outer long running scope. The outer scope has an exception handler associated to catch any exceptions. The handler uses the Compensate shape to invoke the custom exception associated with the called orchestration to roll back any changes that might have occurred in the call to the orchestration.

Figure 8 Long running transactions with custom compensation

Figure 9 Called orchestration (main)

Figure 10 Called orchestration (compensation)

Technical Contributors

Ruslan Yakushev, Suri Narayanan, Kartik Paramasivam, John Taylor, Gregory Vandewiele