Workflow in Application Integration
by Kevin Francis
Summary: One of the greatest challenges facing the architect today is the integration of applications. Let's look at a framework for application integration that moves beyond the common, one-off integration approaches and toward a cohesive structure. The requirements for successful integration are outlined along with the presentation of an architectural approach for meeting these requirements by using tools such as workflow technologies.
Contents
Enterprise Architecture Backbone
Integration Layer Components
Extending the Integration Scenario
A Successful Integration Mindset
About the Author
Integrating applications has continued to become more common, and the growing availability of tools and standards (such as the WS-I Web services standards) and service-oriented architecture (SOA) appear to hold a promise of easier integration. Many articles exist that promote the simplicity of linking applications using Web services, or even SOA. Two approaches for integrating applications are commonly used today: point-to-point and services bus integration (see Figure 1).
In the point-to-point scenario direct links are created among applications through a direct application program interface (API) link, file transfer protocol (FTP), or batch interfaces. Transformation (translation) of data may take place as data is transferred across the link. Generally, point-to-point interfaces are implemented without the use of an integration product, with translation of data taking place using code at the point of integration at one or both ends of the interface.
.gif "Click here for larger image")
Figure 1. Point-to-point and service bus integration (Click on the picture for a larger image)
Service bus integration makes use of a technology solution to provide a bus, upon which applications are able to place messages to have the bus itself manage the routing of messages among applications. The bus will also generally manage the transformation of message formats among applications.
As organizations strive to bring a greater range of services online, aim to integrate product lines together, and aim to streamline call center experiences, an integration solution requires more back-end systems. It is not uncommon, for example, for call center operators to switch among upwards of ten applications in handling customer calls. The replacement of this type of scenario, where the user is effectively integrating the applications by copying and pasting or retyping information into various applications to complete a transaction, is a common driver for the integration of applications.
Table 1. The requirements of an integration layer
| Layer | Requirement | Description |
|---|---|---|
| Data | Connectivity | Basic connectivity in which applications are able to communicate with each other |
| Transformation | Translation of data format, and so on among applications | |
| Information | Data aggregation | Aggregated view of data across multiple systems |
| Business rules | Development of business rules across multiple systems | |
| Transaction management | Ability to perform ACID transactions across multiple systems | |
| Information model | A cohesive data model across all systems where a common understanding of data entities and structure is achieved | |
| Reference-data management | Management of commonly used reference data from multiple systems in a single location | |
| Session management | Management of session information across interactions and across systems | |
| Instrumentation | Common point for the logging of operational information | |
| Error management | Consistent approach to error management from a single rule set | |
| Configuration management | Ability to configure the run-time operation of the entire system, configure its communication with the various systems that make up the environment, and deploy new versions of the various components | |
| Workflow | Workflow processes that cross multiple systems | |
| Process | Complex business rules | Shared, reusable business rules that cross multiple systems |
| Business process modeling | Modeling of business processes across systems for optimization and integration | |
| Business activity monitoring | Monitoring the speed and efficiency of end-to-end business processes for optimization and issue tracking |
Enterprise Architecture Backbone
There are a number of methods that can be used to integrate applications behind a Web site or call center application, and the most common solution is to construct a number of point-to-point or service bus interfaces in the first instance.
.gif "Click here for larger image")
Figure 2. Three integration layers (Click on the picture for a larger image)
What happens, however, when a call center solution and a Web application need to access the same information? The ideal scenario is to reuse the interfaces, right? Well, in most cases where development teams (and architects) are disconnected, this is not the common approach, given the complexities of taking an existing interface with its own complex set of code and allowing it to be called by something else. This issue grows exponentially with the number of interfaces that need to exist among systems and is relative to the size of the organization; the experiences of larger organizations are obviously worse than smaller organizations.
Failure to implement a well-architected, end-to-end solution results in duplicated code across the enterprise, inconsistent architectural approaches in each system, and an inability to respond to business needs in a timely manner. These issues are a by-product of a project-centric approach to solution architecture.
SOA is commonly heralded today as the solution for the integration issues among applications, but a number of additional capabilities are required for a truly efficient integration solution. These capabilities are listed in Table 1 and grouped into three layers of integration:
- Data integration—The most basic layer, data integration generally is achieved in even the most basic integration scenarios. In this layer data is moved among applications with transformation taking place to allow data to be translated among applications.
- Information integration—In this second layer, data and calls to applications are aggregated to enable single calls to access multiple applications, with the basic business rules in place to allow single calls to bridge applications. The use of these techniques provides service aggregation and meets the minimal requirements to achieve an SOA implementation.
- Process integration—The third layer of integration builds on top of data integration by aggregating and integrating the processes and data that are involved with executing a business process that operates across application boundaries.
Table 2. Technical requirements for the data layer
| Requirement | Description |
|---|---|
| Connectivity | Connectivity is the ability to transfer information using a variety of protocols or methods, and it includes the provision of a Web services interface. It also includes specific protocols required by specific scenarios, which will change from one organization to another. For example, many organizations that retain mainframes for core business systems would require IBM WebSphere MQ and/or SNA connectivity using Microsoft Host Integration Server (HIS). Other systems may require COM+ interfaces or even HTTP, raw sockets, or FTP.
All protocols should be executed through a common interface, with pluggable specific implementations for the specific scenarios. All specific technical requirements of the particular protocols or methods should be handled by the implementation, and external systems should not be required to contain any logic to handle specific cases. |
| Transformation | Transformation is the rule-based transformation of data from one structure to another. Once again, the transformation engine should be contained within the integration layer. |
Table 3. Technical requirements for the information layer
| Requirement | Description |
|---|---|
| Data aggregation | The provision of services that wrap multiple calls brings with it data aggregation. It is important to consider data aggregation as a separate requirement, as it brings a requirement for greater care on the design of the data and the use of careful data design and business rules to allow the data to be aggregated rather than simply accumulated. |
| Business rules | It is necessary to be careful in the development of business rules in an integration scenario to ensure that only those rules that form part of each application reside within the application, and those rules that are related to the integration of the application are encapsulated within the integration layer. This approach provides the greatest opportunity for reuse, simplifies the maintenance of the applications, and provides a single, centralized point for development and execution of the rules. |
| Transaction management | Transaction management is both a necessary and complex requirement for application integration. It is necessary because there is a need to commit or roll back transactions across all applications that are being accessed, driven from the calling application, which is critical. It is complex, however, because rollback for many systems may involve providing reversing entries or some other code-based method, which is an ideal example of the need to centralize this complex logic. |
| Reference data management | Reference data is used commonly by application user interfaces to provide lists of choices or for data validation. It is common for reference data to be common across applications (such as lists of countries, postal codes, products, office locations, and so on). Accessing reference data from a shared location in the integration layer provides better performance, consistent results, and less development effort. |
| Session management | Session management is used to ensure that all systems in the integration scenario have a current understanding of who is accessing the data, and to ensure that the current actions are synchronized across the systems. A complete use of session management can be used to allow a session to be captured from one access point, such as a customer accessing an Internet portal, and reused in another, such as to allow a consultant to fix and complete the order internally. |
| Instrumentation | In a similar way to reference data management, the centralization of instrumentation (logging) into an integration layer is logical in that it is the point from which the majority of calls with other systems are made, as well as again allowing the instrumentation code to exist in a single location. |
| Error management | Error management is similar to reference data management, in that the use of an integration layer for error management allows error processing code to be shared and allows a single set of error responses to be used. |
| Configuration management | Configuration management of the type of environment outlined here is a complex matter and worthy of its own content. Configuration management should, however, provide a central point where the addresses of systems can be altered, access to systems can be controlled (preferably with functionality gracefully deprecated), and where overall configuration settings can be stored and loaded once by a shared configuration manager. |
As you move through the three layers of integration, the focus changes from technology to business. The end result, however, allows for greater ability to quickly add value to the business.
Many views of SOA provide a common connectivity model and assume to provide transformation, but these views, in reality, only provide data integration. The challenges faced by an architect in integrating applications in an efficient and repeatable manner are far more complex, particularly when an integration scenario includes applications that are already in existence.
Why does all of the foregoing matter? When you build the first interface between two applications, simple data integration can be sufficient. As you build more interfaces the design becomes more important, however. Failure to design and manage a complete integration scenario leads to an exponential increase in the cost of each interface in turn. Conversely, as the functionality that is provided in the integration layer grows, the code that is needed in each layer is able to be reduced.
Integration Layer Components
The components of an integration layer can be split into three layers: data, information, and process (see Table 1). Furthermore, they can be separated into components that make-up a technical framework; the components appear in blue in Figure 2. The technical framework provides plumbing and engineering requirements. A business framework provides business enablers, which are shown in gold in Figure 2. Let's take a closer look at the components that make up the three layers of an integration layer.
Table 4. Business requirements for the information layer
| Requirement | Description |
|---|---|
| Information model | As the number of applications that are linked into the integration layer grows the amount of data that is available grows. This situation provides the ability to begin to build an information model across the systems that are in scope, which can be closer to building an information model for an organization than most are able to achieve.
When implemented correctly, the information model that sits within the integration layer can provide a single point of truth for key data entities. An example of the single source of truth is an understanding of a customer that covers all systems with addresses, products, order history, and support history. Two commonplace methods of creating a single point of truth are to utilize one key system and attempt to synchronize across other systems, or to use a database that is updated with changes to multiple systems. The use of a shared information model in the integration layer is particularly powerful as the result is a shared single point of truth that is available through the combining of data from multiple sources without impacting data or code in those sources. Also, given that no synchronization of data is required the data is more likely to be current. |
Table 5. Business requirements for the process layer
| Requirement | Description |
|---|---|
| Workflow | Workflow systems can be implemented as user (human) driven or system driven. Both can be utilized in an integration layer; although, it is important to separate the two. System-driven workflow includes the execution of the steps involved in interacting with the various systems in the environment. |
| Complex business rules | Complex business rules provide an additional value as the integration layer becomes complete and as it becomes a place where business rules begin to reside there in their own right. Given the central role that an integration layer begins to provide, it can become the host for business rules that provide new or extended functionality beyond the integration of applications, which could be new products, combining products together, additional discounts, better credit check, and so on. |
| Business-Process Modeling | Workflow systems can also be used for the execution of human-facing workflows, and this reason is predominant for their existence. The provision of an integration layer provides the ideal host for a business-process modeling environment to exist, allowing human workflows to be automated. The use of workflow tools allows processes to be documented and executed in a graphical form. The use of the integration layer behind the workflow tools allows access to multiple applications through the workflow tool more easily and readily, particularly when the capabilities outlined here, such as the information model and transactional support, have been implemented. Thus, business-process modeling is allowed to facilitate greater improvements than would otherwise be possible. |
| Business Activity Monitoring | Business Activity Monitoring (BAM) is the tracking of each transaction through the end-to-end system to identify roadblocks, slow performance, issues with particular transactions, and opportunities for overall performance. Based on the instrumentation capabilities provided by the information layer, the capabilities provided by end-to-end BAM are considerable. |
Data layer. The two components of the basic data layer are connectivity and transformation, both of which form the foundation of the technical framework (see Table 2). Single point-to-point interfaces using Web services are frequently used as examples of SOA, but many principles of SOA refer to the provision of services that wrap multiple calls across systems. As mentioned previously, however, Web services exist in the data layer, which does not provide for these principles. In this context, Web services provide for the connectivity, but we need to explore the information layer for service-related integration.
Information layer. The information layer is made up of components that build on the technical framework (see Table 3). The business framework is built on the base of the information model (see Table 4).
Process layer. As the focus moves from technical requirements to business value, the focus of the functionality that should be delivered through the process layer shifts to providing the true value of the integration layer concept—quickly configurable business processes (see Table 5).
In bringing it all together, there are a number of ways that an integration layer can be implemented: custom development; the use of frameworks, toolkits, open source, or operating system-provided components; and the use of a more complete integration package. Custom development could be applied completely to the integration layer but is not recommended given the availability of toolkits and frameworks.
Some frameworks that are available provide an opportunity to quickly integrate systems at the user interface layer, such as the Customer Care Framework from Microsoft. This type of integration solution provides an excellent solution to the rapid integration of back-end systems to provide a consolidated user interface, but it is important to understand the relative merits of this type of solution. Frameworks that enable the integration of systems at the user interface layer provide rapid solutions for certain issues encountered by businesses, such as improving call times for call center operators. An integration layer can provide further benefits, at an additional initial implementation cost.
Extending the Integration Scenario
An integration layer allows key functions to be hosted centrally, shared among applications, allowing more than the integration of the user interfaces. The range of data, information, and process integration capabilities can be used across applications to lower the cost of each development or integration in turn. Experience has shown that the use of an integration layer as described here produces growing cost savings across a range of applications over time, with the range of creating new applications beyond the original integration scenario.
For example, while billing, trouble-ticketing, customer-management, and ERP systems may be integrated behind a CRM user interface in the first instance, the fact that these systems are already linked provides simple solutions for Web self-help, and then extending to online ordering and a B2B channel. Should there be a purchase of a new application, such as a supply-chain application, the integration of that application into the environment is still not a trivial process. However, the integration is much easier than it would be without an integration layer because data integration can be achieved more easily by hooking up a single set of known interfaces into the integration layer, not into each application. Also, the process of integration can be added by extending the integration workflows in the workflow engine, just as BAM integration, data aggregation, inclusion in the information model, and so on are more easily achieved by simply extending that which already exists.
The workflow component is a key part of any integration solution being used to provide both human-focused workflow processing through a business process management capability and in playing the central role of executing the processes involved in linking systems. These two tasks are quite separate and quite different, but they are both suited perfectly to the use of workflow tools. A system-integration workflow can include such steps as retrieving data from each system and aggregating it together, validating user entries, and updating systems in order with entered or updated data.
Workflow tools provide the ideal method of performing these activities, are vastly superior to using code, and they can therefore provide a significant business benefit from an integration layer, given these points:
- Working across multiple systems is inherently process driven, executing tasks in a step-by-step way with decision points, branching, and other core workflow steps.
- System integration workflows sit centrally among applications and are therefore more often impacted by changes to the entire environment, given that each change to each system will be reflected in changes to the integration layer. The use of workflow tools allows processes to be more easily understood by a wider group of people than the original developers, given the graphical and somewhat self-documenting nature of the workflows; processes to be more easily and quickly changed, and more easily debugged than code; and the use of less code, which results in both greater reliability and easier debugging.
- Given the diagrammatic form that workflow tools provide an easier understanding of the business processes, ownership can be extended easily beyond the development teams to business analysts and even core users in the business community.
A Successful Integration Mindset
Having the tool and using it correctly are two very different things, and this discussion outlines a mindset that is needed to achieve truly successful integration. Regardless of the approach that is taken, remember that an enterprise-wide view of integration will lead to far greater business benefits; integration itself is nontrivial and should be considered in an enterprise context; and while considering the enterprise context and designing for maximum business benefits it is still possible—and necessary—to begin to implement the solution with a single project.
The enterprise view of integration should be taken as early as possible. While there is clearly a place for point-to-point integration where further integration is unlikely, a more capable integration environment providing the workflow and other capabilities outlined here will provide advantages as the number of systems and integration grows. Assessment outside the scope of each single project is therefore needed to uncover whatever opportunities may exist.
As with all architectural decisions, there are business drivers, costs, capabilities, and so on that are taken into account when making architectural decisions. The approach to integration architecture recommended here is well suited to the capabilities of today's tools and today's architectural concepts. It will hopefully help you in understanding some of the decisions that can be made in the area.
About the Author
Kevin Francis is an IT professional with 19 years of industry experience in a range of positions—from CIO in a manufacturing company, to security consultant, to global architect—and he also operated his own successful software development organization. Kevin has been architecting and managing leading-edge, e-commerce projects since the birth of e-commerce. As an Infosys principal architect, Kevin is responsible for the success of the company's technical implementation for its customers. He consults to large organizations; works with Infosys's architects, development teams, and customers to ensure that each technical solution adds to the overall quality and strategic direction of the customer; and is a member of The Infosys Australia Technology Council. Kevin was awarded a Most Valuable Professional (MVP) award by Microsoft in July 2005 for his knowledge and experience in the Visual Developer Solutions Architect area.
This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal website.