BizTalk Adapters

Integrate E-Mail Processing Into Your Business Solutions

Alex Starykh

Contents

First, Some Background
Building the Solution
Orchestration
Create a Custom Adapter

It's a common problem everywhere. Data arrives at your doorstep in all sorts of formats, typically initiating some sort of customer service, order fulfillment, or other business process. Often the data arrives in an e-mail message accompanied by other documents. Once the e-mail is received, an automated reply must be sent and the e-mail must be saved to a database or to a line-of-business (LOB) application. Depending on the kind of data received, different e-mail notifications may be needed.

Dealing with the variety of data formats and work processes is no easy task. Fortunately, BizTalk® Server can help. In this article I'll develop a solution that shows how. My solution uses BizTalk Server to monitor a specified e-mail account for new messages, trigger a confirmation when a new e-mail message is received, and then persist the entire contents of the e-mail, including all attachments, to a custom database. It also includes some conditional logic that will send notifications based on the data in the claim. New features in BizTalk Server 2006, including the new POP3 adapter and the enhanced SMTP adapter and variable multipart message support will glue the solution together.

If you want to further refresh your BizTalk knowledge, you may want to look at the useful introductions by Aaron Skonnard in the November and December 2005 issues.

First, Some Background

BizTalk Server is an integration server designed to work with messages. Figure 1 shows a typical message flow. A message is received by one of the locations belonging to a receive port. While one port can have multiple receive locations, there is always one and only one port for each location. Each receive location has a specific configuration for one adapter. For example, a receive location may have a configuration set for a file adapter that indicates what type of files (file mask) to poll from what folder. When the adapter receives a new message, the message moves from the receive location to the receive port, then to a pipeline. The pipeline is simply a sequence of components that is executed for every message. Usually the pipeline performs tasks such as decoding, decrypting, verifying messages, and generating the XML representation.

Figure 1** Typical Message Flow in BizTalk **(Click the image for a larger view)

Once work in the pipeline is complete, the message is submitted to a SQL Server™ database called the MessageBox, which contains information about subscriptions, in-flight and suspended messages, message properties, tracking, queuing, and other important services provided by BizTalk.

BizTalk is built on a publish/subscribe model. Every single message received by BizTalk is automatically routed to subscribers whose subscriptions match, say, all messages received from a certain port. A more complex subscription might target messages of a given type with values promoted from inside the messages.

When a message is published to the MessageBox database, if there is not at least one subscriber for that message, it is immediately suspended. Most suspended messages can be manually resumed for processing later.

Subscribers in BizTalk are either send ports or orchestrations. Each send port's subscription determines what messages will be routed to that port. When a message is destined for a send port it goes through a send pipeline. Usually the send pipeline is responsible for assembling the message into the appropriate format, signing and encrypting the message, and so on. From the pipeline the message goes to the adapter configured on the send port.

Orchestrations represent a business process workflow that executes when an appropriate message is received by BizTalk. When orchestrations send messages, those messages are published again to the MessageBox to be routed to the matching send port.

BizTalk ships with more than 25 different adapters that cover most transport protocols and many common applications, including file, FTP, HTTP, SOAP, SMTP, POP3, SharePoint®, SQL Server, Oracle, SAP and EDI. Most can be used for receive and send ports. Refer to the BizTalk documentation for a complete list of available adapters.

If no built-in adapter matches your needs, BizTalk lets you create a custom adapter. Later in this article I'll show you how.

The new BizTalk Server 2006 POP3 adapter is responsible for polling a message from a specified mailbox, parsing the message, and separating attachments. The adapter then creates a new multipart message with one message part designated as the body and other parts corresponding to each attachment.

The body part represents a default part that is used by the engine for processing because its type is known to BizTalk. But BizTalk doesn't have to understand the other parts of the message, unless we explicitly declare them. In fact one of the new features of BizTalk Server 2006 is the ability to declare only the body of the multipart message at design time, leaving the number of other parts and their types undeclared. This comes in handy when the number of attachments is unknown beforehand.

Building the Solution

Now let's start developing our solution. First we'll set up the SMTP and POP3 servers. If you have them already available for development purposes, use them; otherwise install the SMTP and POP3 services that ship with Windows Server® 2003. You can add those services in Control Panel by selecting Add/Remove programs, then Windows Components, then SMTP and POP3 services. When that's done, add mailboxes for a couple of users to test mail delivery functionality between those users.

Now go to the BizTalk Server Administration console and create a new application called ClaimsProcessingApp.Note that applications in BizTalk represent a logical grouping of different BizTalk artifacts.

You can create the first receive port with one receive location that will use the POP3 adapter by right-clicking on the Receive Ports folder and selecting new one-way receive port, specifying the port name RcvEmailPort. Next, click on the ReceiveLocation tab and create one receive location with a transport type of POP3. Then select the POP3 adapter and click Configure to get the POP3 adapter configuration screen.

The POP3 adapter lets you specify how the message should be parsed when it's received by the adapter. Figure 2 shows some of the most important configuration settings for the POP3 handler.

Figure 2** Configuration Settings for the POP3 Handler **(Click the image for a larger view)

In our scenario, the received e-mails have one XML attachment and any number of other attachments, so we'll specify that mime decoding should be used and that the body part content type is text/xml. Don't forget to specify the mail server, user name, and password used for retrieving e-mail messages, and make sure the poll frequency is set to something appropriate for development purposes-let's say once every 15 seconds.

Now let's get ready to test our receive port. We'll use the messaging-only aspect of BizTalk by creating a send port that subscribes to any message received by our receive port. Click on send port, selecting a new static one-way port, and File as the transport adapter for the port. Then specify a physical location where the test messages will be placed. Now specify a subscription on that port: select Filters from the Send Port Properties list (as in Figure 3) and specify the following filtering expression to indicate that all messages received by RcvEmailPort will go to this send port:

BTS.ReceivePortName == RcvEmailPort

Figure 3** Send Port Filter **(Click the image for a larger view)

Let's test what we've done so far. Go to the receive location configuration, enable it, and start the send port we just created. Then send e-mail with a couple of attachments including an XML file (any XML file will do) to the monitored account. A few moments later you should see the new file created with the body of the message you sent. If you don't, first verify that the BizTalk host is running and then check the event log for additional troubleshooting steps.

At this point you should be able to retrieve a message from the POP3 server and save the XML portion of that message to a test folder.

Orchestration

Now we'll build the business process that begins as a new message arrives. Again in our scenario, the entire content of the message, including its sender, is saved to a database. Then an automated acknowledgment is sent back to the sender. The last step is to decide if additional e-mail notification is needed based on the claim amount.

Typically this sort of business process is implemented via orchestrations. To build one, start Visual Studio® and create a new project based on an empty BizTalk project type.

To design a new schema that represents the claim data, right-click on the project and select Add New Item | Schema and name it Claim. Add several fields to the claim, including the claim amount as a decimal type. Right-click on the amount and select Show Promotions, then promote that field as a distinguished field (see Figure 4). I've also created a Submitter node in the schema with promoted from, to, and subject elements in that node. We will need those promotions later when we capture details of the original message and decide about additional notification.

Figure 4** The Claim Schema **(Click the image for a larger view)

We will also need a sample of the instance that complies with the schema. BizTalk can generate one if you right-click on the schema and select Generate Instance.

Now we're ready to design an orchestration. Much as you added the schema, add a new orchestration to the project, then create a new multipart message type, called ClaimMessageType, with one part-a body part called xmlBody defined by the schema we just designed. Add two messages-incomingClaimMsg and finalClaimMsg-of the same ClaimMessageType. Now add a receive shape, which will receive incomingClaimMsg and will activate a new instance of the orchestration (activate property of the shape needs to be set to true).

The next step is to construct a new, finalClaimMsg, of the same message type but with additional values persisted to the fields in the body of the message. Those fields will be populated based on information from the SMTP header of the message. For example, we will add e-mail Subject, From, and To addresses to the original XML body of the message.

To achieve this we assign messages first using the following expression in a newly added message assignment shape:

finalClaimMsg = incomingClaimMsg

In the same construct message shape, we also assign the additional fields:

finalClaimMsg.xmlBody.Submitter.from = incomingClaimMsg(POP3.From); finalClaimMsg.xmlBody.Submitter.subject = incomingClaimMsg(POP3.Subject); finalClaimMsg.xmlBody.Submitter.to = incomingClaimMsg (POP3.To);

and

finalClaimMsg(BTS.DestinationParty) = "mailto:" + incomingClaimMsg(POP3.ReplyTo);

The last line sends an acknowledgment back to the sender.

Now we are ready to send finalClaimMsg to the file or database and send an acknowledgment back. To do so, add two send shapes and indicate that finalClaimMsg should be used on both of the shapes. Finally, let's add a decision shape to indicate that with a claim amount of more than 100, an additional notification will be sent to a predefined user. The rule in the decision shape might look like the following:

If (finalClaimMsg.Amount > 100)

The Then branch would consist of only a new send shape with finalClaimMsg.

Now we need to create four logical ports and connect them to the appropriate send or receive shapes. Drag and drop a port shape from the toolbox to the orchestration port surface and go through the port configuration wizard, accepting all default values except port name and the communication's direction. Repeat that process to create one receive and three send ports. Connect the receive and send shapes to newly created logical ports. The final orchestration should look similar to Figure 5.

Figure 5** The Complete Orchestration **(Click the image for a larger view)

Now adjust the project properties to use a key file for assembly signing and then compile the project. Go to the deployment tab and change the Application Name property to ClaimsProcessingApp. Deploy the solution by right-clicking the project and selecting deploy menu. At this point the project with all schemas, maps, and orchestration is compiled into a .NET assembly that gets placed in the Global Assembly Cache (GAC) as well as registered with the BizTalk management database.

Assuming deployment is successful, let's go back to the BizTalk Management console to bind the orchestration's logical ports to the physical ports we'll create next.

There are two additional ports that use the SMTP adapter to send an e-mail acknowledgment and notification. These ports are very similar except for the actual message text, which can be specified on the port level as opposed to using the dynamic port with the body specified in the orchestration.

The first port, named SendAcknowledgmentBack, uses the SMTP transport with the To address set to %DestinationParty%, which will be replaced by the destination party information we've already assigned in the message assignment shape (finalClaimMsg(BTS.DestinationParty)=...).

The second port, SendAdditionalNotification, uses SMTP transport with the delivery address set to a fixed value.

For the database port, let's temporarily use the file adapter port that was created in the previous part of the solution, but with the subscription filter completely removed. We'll change the file transport to the database transport in the next section. Select the SendResultsOut send port and remove any subscription filters.

Now we need to change a pipeline on the ReceiveEmailMsg from Passthrough to XmlReceive, since BizTalk needs to know what's inside the message's body part. Also, since we are using promoted fields the actual promotion is also done via an XmlAssembler that resides in XmlReceive pipeline. Go to the properties of the deployed orchestration and specify the bindings as shown in Figure 6.

Figure 6** Orchestration Bindings **(Click the image for a larger view)

Now we're ready to test the solution. Start the application (right-click on the application and select Start) and send a new e-mail message to the account monitored by BizTalk with the sample message generated earlier as an attachment.

If everything is set correctly, we'll receive an acknowledgment back and one additional file will be created in the destination folder with the XML body of the message. If the sample we attached had a claim amount greater than 100, an additional e-mail notification will be sent as well.

Create a Custom Adapter

There's still one major piece of the solution missing. The file that BizTalk creates in the destination folder represents only the XML body of the message. Where are all the attachments and what about storing the entire message in the database? As it turns out, the majority of the BizTalk adapters, including the file adapter, use only the body of the message, completely ignoring the rest of the parts. While this makes sense for many BizTalk implementations, it's not exactly what we need here. Enter custom adapters.

Custom adapters are .NET components registered with BizTalk Server that are responsible for receiving and/or sending messages. While building a generic custom adapter can be fairly complex, Microsoft has created a base library called the Adapter Framework, which makes creating a custom adapter much easier. The framework ships with the BizTalk SDK; its complete source code is available in C#.

Figure 7 highlights the relationship between a custom adapter and the BizTalk Server environment. Not only does the framework enable you to create a run-time environment for the adapter, it also allows custom configuration of settings that are rendered to the user by BizTalk Server Explorer or BizTalk Server Administration console.

Figure 7** Adapter Framework **

There are multiple variables that influence design requirements for the adapter. Among the most critical are:

• Communication direction—whether the adapter is a send or receive adapter.
• Hosting-whether the adapter is hosted by a BizTalk process (in-process host) or outside of the BizTalk environment.
• Message exchange pattern—which involves three possibilities: one-way asynchronous, in which messages flow in one direction with no responses or other dependencies from previous or subsequent messages; request-response, in which each message flowing into the adapter should generate a response back to the message box (used, for example, when an orchestration initiates a call to a Web service using a SOAP adapter); solicit-response, which is similar to request-response except the direction of the initial message (as when an orchestratin is exposed as a Web service using a SOAP adapter).
• Batch-supported or batch-unsupported—whether the adapters can send multiple messages simultaneusly, which reduces chattiness between the adapter and messagebox database as well as between the adapter and the outside world.
• Transaction-aware—whether the adapter supports transactional sending and receiving of messages.

There are several mandatory .NET interfaces that all adapters must implement. These interfaces include IBaseComponent, which returns name and version of the adapter, IBaseTransportType, which details transport type, and IBatchCallback, which receives the status of a submitted batch to the MessageBox database. Depending on the selected variables, the adapter is likely to require implementation of additional interfaces. For example, a batch-aware transactional send adapter is represented by a class that must implement IBTBatchTransmitter. The BizTalk documentation outlines specific requirements for each adapter type implementation, so refer to this resource often.

The send adapter we need to create here has to support transactional interactions with a custom database, relying on a one-way asynchronous pattern. Our adapter will be generic, meaning it won't know much about message content. It will call a stored procedure for each message part, passing message part data and supporting information-contentType, messageId, and a flag indicating if the message part passed is a bodyPart. So each message part will be persisted to the database in a context of a single atomic transaction initiated on a message level. This means that if some part can't be persisted to the database (let's say the call to the stored procedure fails), the transaction will roll back and the entire message will not be saved. In that case, the adapter will repeat the process until the limit set in the adapter's advanced properties is reached. When the limit (by default up to three times every five minutes) is reached, the message will be suspended.

From the adapter point of view, each message is represented by the IBaseMessage interface. Since each message can have multiple message parts, those parts are represented by the IBaseMessagePart interface. The actual body of the message part is accessible via the IStream interface, as shown in Figure 8.

The IBaseMessageContext interface represents a dictionary indexed on property name and namespace. Such an interface-centric design allows easier interoperability with COM-based adapters as well as implementation-specific substitution (for example, based on size of the message) of actual instances.

Figure 8** The IBaseMessagePart Interface **(Click the image for a larger view)

While we could proceed along that route and build upon the Adapter Framework, let's take an easier route. The BizTalk SDK ships with several examples of custom adapters. One is a transactional send adapter that does almost exactly what we need. So we will build on that sample instead of starting from scratch.

Let's copy the sample from BizTalk Installation Folder\SDK\Samples\AdaptersDevelopment\TransactionalAdapter to a new folder so it can be safely modified. For my installation BizTalk Installation Folder refers to C:\Program Files\Microsoft BizTalk Server 2006.That folder contains two projects: one represents design-time features of the adapter and the second represents run-time features. We also need to copy the base adapter project located in the BaseAdapter subfolder of the directory I just cited.

We can easily add those three projects to the same solution we used to design the orchestration (though you can create a new solution if you prefer) using File | Add | Existing Project in Visual Studio. Check for any missing strong name key files and replace them with your own if needed.

Since we're building a send adapter, we can safely delete several files designed to support the receiving side of the adapter. Those files include TransactionReceiver.cs and TransactionReceiverEndpoint.cs.

Let's quickly review the code. The entry point for our send adapter is in the TransactionalTransmitter class. In fact, that class is exactly what we will be using later on when we create a .reg file for our adapter registration.

The code of the TransactionalTransmitter class shows that the CreateAsyncTransmitterBatch method is called by BizTalk when there is a new batch to send out. The code creates a new instance of the TransactionalAsyncBatch class, which implements the IBTTransmitterBatch interface that is needed to inform BizTalk that the adapter sends messages asynchronously.

The TransmitMessage method is called for each message in the batch and then the Done method runs. For each batch of messages, the Done method creates a new worker thread where the SendMessage method is eventually executed.

Let's make some changes to the projects. First, to be good citizens we need to modify a unique GUID associated with the adapter in case you need to reuse the original source code again. The GUID is located in TransactionalTransmitter.cs and it's passed in as a parameter in the base constructor of TransactionalTransmitter class.

Also, in TransactionAsyncBatch.cs, the Send method must be changed. That method is called when the adapter is ready to send out a message or message batch. The new code for that method is located in Figure 9.

Figure 9 New Send Method

private void SendMessage( IBaseMessage message, TransactionalTransmitProperties properties) { string storedProcedureName = properties.CmdText; string connectionString = properties.ConnectionString; Guid messageId = message.MessageID; using (SqlConnection connection = new SqlConnection( connectionString)) { connection.Open(); for (int i = 0; i < message.PartCount; i++) { //get one part string partName; IBaseMessagePart part = message.GetPartByIndex(i, out partName); //read the part body to an array byte[] msgData = new byte[part.Data.Length]; part.Data.Read(msgData, 0, msgData.Length); string contentType = part.ContentType; bool isBody = (part.PartID == message.BodyPart.PartID); SqlCommand command = new SqlCommand(storedProcedureName, connection); command.CommandType = CommandType.StoredProcedure; command.Parameters.Add( new SqlParameter("@MessageID", messageId)); SqlParameter bodyParam = new SqlParameter("@Data", SqlDbType.VarBinary); bodyParam.Value = msgData; command.Parameters.Add(bodyParam); command.Parameters.Add( new SqlParameter("@ContentType", contentType)); command.Parameters.Add( new SqlParameter("@IsBody", isBody)); //execute command.ExecuteNonQuery(); } } }

As you can see from Figure 9, we open a new connection to SQL Server, accepting a connection string as a parameter that can be customized at deployment time, then iterate through each message part of the outgoing message. After reading message content, content type, and a flag indicating whether the part is a body, we create and execute a new SQL command. If you check the code that calls the Send method (the BatchWorker method), you'll find that it creates and controls transactions using the Transaction class new to the .NET Framework 2.0-that's exactly what allows us to run code in a single atomic transaction.

Figure 9 has the complete text of the stored procedure that's invoked by the adapter. As you can see, it's a simple insert statement against a custom table. Be careful not to use GUID as a primary key for your table when you perform many inserts as this may cause future performance problems due to data page fragmentation:

CREATE PROCEDURE [dbo].[MakeMessage] @messageId uniqueidentifier, @data varbinary(max) = null, @contentType varchar(50), @isbody bit = null AS BEGIN INSERT INTO Messages(messageID, data, contentType, IsBody) VALUES (@messageId, @data, @contentType, @isBody) END

Figure 10** AdapterRegistryWizard **(Click the image for a larger view)

Now build the solution. When the build succeeds, we need to create a registry file (*.reg) that summarizes changes needed to register our adapter within BizTalk. Fortunately, that task can be easily done by running the AdapterRegistryWizard from BizTalk Installation Folder\SDK\Utilities\AdapterRegistryWizard. When the wizard starts, type a name for the adapter and the properties namespace, and check the box indicating that the adapter will only send messages (see Figure 10).

On the next screen, select the assembly with the run-time implementation of the adapter-usually it's in the project's bin folder-and select the Transmitter class as an implementation entry point for the adapter. The adapter alias can be any string uniquely identifying the adapter and used for dynamic ports in the orchestration, but it's not used for our solution.

The next screen asks about management interface implementation. It needs to point to the assembly generated by the second project we've copied from the original location. Make sure that the box representing send handler configuration is checked. Specify a location output for the registry file and the wizard will create the .reg file there. I recommend that you review the registry file as it contains useful information for better understanding the adapter registration procedure. Double-click on the .reg file and confirm the registry merge.

The final step is to make the adapter available to the BizTalk management database. We'll do so by expanding the Platform Settings | Adapters folder in BizTalk Administration console and right-clicking and selecting the New | Adapter menu. Our adapter should be available from the dropdown selection; type a name and confirm the changes. Don't worry about restarting the host now as that can be done later.

Now it's time to test the adapter. Stop the application (right-click and select the Stop command), then go to the SendResultOut port and specify a new transport to use: TransactionDbAdapter. Click configuration and specify a connection string and stored procedure name. Restart the host via Platform Settings | Host Instances, select BizTalkApplication, and click Restart.

Now send an e-mail message with several attachments, including an XML form, to the mailbox monitored by BizTalk. As soon as an acknowledgment is received, check the table in the database-you should see several records, one corresponding to each attachment and one for the message body.

As you've seen, triggering a business process when e-mail is received can be easily accomplished with BizTalk Server. In many scenarios there is no need to write any code, but if you must, use the Adapter Framework from the BizTalk Server SDK and take advantage of the samples shipped with the product to jump-start your development.

Alex Starykh is a Technology Architect at Microsoft Technology Center in Virginia. He has more than 12 years of experience in architecture, development, and consulting for commercial companies and government agencies with focus in integration, databases, and custom development. Alex can be reached at alex.starykh@microsoft.com.