Composite Applications: The New Paradigm

by Chris Keyser

Summary: Composite applications offer a long-sought-after business nirvana whereby empowered technical business users can stitch together componentized business capabilities. In many ways, composite applications are the business users' equivalent of Web 2.0 and "mash-ups." While there has been a lot of hype around composites, many vendors have been slow to deliver real value in this area. Technologies are emerging, however, that will change this game, and composition will become an increasingly important aspect of constructing business logic. In this article, we'll discuss some of the fundamentals and advantages of using composite applications for today's business challenges.

Contents

Business Motivation
Composition: A Walk-Through
Layers of Composition
The Emerging Application Paradigm
Conclusion
Acknowledgments
About the Author

Business Motivation

For many enterprises, empowering business users to react quickly to rapidly changing business environments and gain a competitive edge is a high priority. Business users have long been beholden to IT to create processes and logic. Composite applications have the potential, however, to move the reuse discussion from the technical domain to the business domain, freeing enterprises from the confines of stove-piped host applications and developers, and enabling them to define behaviors optimized for their business through process flows and metadata.

The recent phenomenon with Web 2.0 in the consumer marketplace indicates that a fundamental shift is taking place and is a leading indicator of users' increasing expectations. More users have increased influence in driving their own experience and are bringing these same expectations to the workplace. Based on working with Web 2.0, users will increasingly expect to exert more control over their work experiences and to participate in them. They will expect business applications to adjust to the way they work, rather than accept a suboptimal experience. These Web 2.0 capabilities will mature as they drive into the enterprise. Ultimately, Web 2.0 is not really about the technology. It's about social networks and users' control of their experience; disposable applications created by end users represent the ultimate ability to respond to quickly changing landscapes. In the enterprise, the way to achieve this movement of power to the end user is through a composite solution that considers the business user as well as the developer.

A composition platform provides a mechanism for multiple technology vendors to participate in a solution without hardwired and fragile dependences. Greater value can be realized from a much broader spectrum of application vendors with much greater ease; a composition framework provides an interaction model that allows components to be effectively decoupled and abstracted from dependencies on other components. As systems increasingly move to composition, they will also be increasingly metadata-driven. The movement to metadata will open opportunities for new sets of capabilities, such as self-healing and version resiliency to applications. The composition model therefore brings many aspects of service orientation to the entire landscape.

Composition: A Walk-Through

Composition means different things to different people. Today, line-of-business (LOB) vendors' systems predominantly manage well-known steps that represent discrete points in time when moving a lead to a sale. This is often referred to as a structured process.

In recent years, vendors have started to use familiar applications—for example, Microsoft Office—to front a better user interface on top of this process for interaction at specific points within an existing process. These improvements enhance the productivity and usability of the process and extend the reach of application logic to a much broader set of users.

These capabilities alone should motivate businesses to adopt them. While these applications are a major improvement in extending reach, they are generally just surfacing existing functionality in a more usable way. Even with these applications, though, traditional LOB applications may be inhibited from progressing beyond this level of control. The inhibitors include:

• Legacy System Construction—Partitioning and componentizing these complex applications interlaced with many implicit dependencies are daunting tasks, and it will take a long time to truly unwind hardwired logic. LOB vendors eventually intend to move beyond their tight-coupled legacy systems into loosely joined capabilities based on service-oriented concepts that enable participation at a deeper level in the "composition" space.
• Loss of Control—It is scary for traditional LOB vendors to give up some control of data and process during the course of decomposing and surfacing capabilities through services and configurable user interfaces.
• Pervasive Horizontal Nature of Collaboration—Customers want one collaboration infrastructure. LOB vendors may not be well positioned to both scale up/down, as necessary, or provide pervasive generic, easily managed capabilities.

New user interfaces in front of existing processes have not really given businesses any additional insight or control. Although the application may appear easier to use, the result doesn't improve an employee's ability to accomplish the task. As an example, this simplified process in Figure 1 describes turning a lead into an order for a custom-engineered product. Taking a lead to a sale actually involves much more work between elements than is represented in the initial structured process. This work is often collaborative in nature. Innovation frequently takes place outside structured processes in the collaboration interactions, where imagination predominates, where the creativity of information workers is truly tapped, and where the critical business differentiation occurs.

Figure 1. Example of a structured process for Customer Relationship Management (CRM) (Click on the picture for a larger image)

Although using a new interface, such as Microsoft Office, improves the view of the data, it does not in itself solve the collaboration problem. Integration of LOB systems into document-centric applications offers a particularly rich opportunity to compose information worker-oriented workflow extensions to the LOB process. With many portal solutions now incorporating workflow technologies (for example, the integration of Microsoft Office SharePoint and Microsoft Windows Workflow), it will increasingly become the collaborative workflow solution for LOB applications—blurring the line between managing documents and managing business processes. The example in Figure 2 illustrates how an account team would assemble a proposal in Microsoft Word to respond to an RFP. This is an example of using composition to extend the process across multiple-user experience channels (Microsoft Word, Microsoft Excel, and Microsoft Outlook), and have cooperating processes, one ad hoc and one structured, running on different systems to accomplish one logical business task.

Figure 2. Using Microsoft Office for composition (Click on the picture for a larger image)

A composite can also bridge processes running in independent LOB systems. For a case where some value-added processing needs to be injected, a composite service makes perfect sense, as is commonly the case today through integration middleware. This is illustrated by an information worker taking a confirmation response for a quote e-mailed to a customer from within Microsoft Outlook, and invoking a composite service to bridge CRM and ERP systems, creating the sales order in the CRM system and the internal order for processing in the ERP system.

Layers of Composition

Now that we have a good idea of what composition means, it's worth understanding other types of composition that can take place. I like to classify these compositions by the layer at which composition occurs.

• User Interface and Client Logic Composition—Views can be assembled by loosely coupled user-interface parts that cooperate through a surrounding framework. These views can be assembled into sequences that have compatible interfaces through triggered events and information to compose higher-level functions. Such composition may take place through portals on the desktop or on a Web server. Examples today can include the Microsoft Patterns and Practices Composite Application UI Block, Microsoft Office SharePoint's composite application framework, Eclipse Rich Client Platform, and a number of other portal frameworks. Web parts, a term used to define the parts that make up a page, are enabling technologies for building a composite framework. Web 2.0 mash-ups typically compose at this layer, generally composing directly in the browser using Asynchronous Java and XML (AJAX) and represent informal and unanticipated compositions.
• Services Composition—Multiple services can be stitched together with process flow, metadata, and rules to provide value-added services. A somewhat trivial but common example is the addition of third-party credit checking to a purchase order service. Service Composition is often performed by integration engine capabilities—for example, Microsoft BizTalk Server.
• Data/Information Composition—A third layer of data/information composition is often one of the hardest problems to solve, and most vendors don't solve this problem outside of well-defined niches. There are several techniques, however, that can help overcome this challenge. For example, Master Data Management (MDM) aggregates, scrubs, and distributes (through a syndication model) specific types of enterprise information. Entity Aggregation has also been discussed for several years and represents a more generalized approach based on ETL/Data Warehouses. Enterprise Information Integration (EII) can also be used as a straight-through processing mechanism to achieve a centralized view of shared information necessary to achieve harmonization of capability execution across a set of disparate services.2 These approaches are the reality today because legacy systems do not have notions of strong-partitioned data boundaries along service-capabilities lines. Nor do systems today make it easy to build services that manage information along these partition boundaries. Pat Helland wrote extensively about the notions of reference data, resource data, activity data, and message data in his Metropolis series. Maarten Mullender also took this up in his series on service agents. Frameworks like the Information Bridge Framework (IBF) allow developers to navigate entity views and relationships described in metadata to enable composition locally by the application rather than through a mid-tier server.
• End-to-End Process Composition—In this case, process is a higher-level composition that stitches together distinct interactions through different mechanisms, typically over an extended period, often across a number of business systems. An example would be Employee Provisioning. The walk-through in Figure 3 illustrates interactions using multiple client applications accessing a back-end process across time. Even in cases today where LOB systems capture structured processes like quote-to-cash, there are many opportunities to take advantage of interactions occurring around the LOB structured process using process composition. Process is also important to other areas of composition within the boundaries of the composition, like user interface and service mentioned above, but use of process in those cases is distinct from this case. In those cases, the processes are internal to the boundary of the composed surface—for instance, a workflow defined within a composed client for a task-driven UI (wizard-like), or a workflow internal to a composed service that orchestrates a number of secondary service invocations and business-rule evaluations to provide a single unit of business functionality.

Figure 3. Composition across independent LOB systems (Click on the picture for a larger image)

The Emerging Application Paradigm

Composite application frameworks have the potential to change the way that applications are constructed, delivered, and experienced by end users. At some levels, however, this complicates the life of application developers, since now more thought needs to be given to which parts of the experience should be surfaced through which vehicles. It also puts pressure on vendors to think harder about developing true service-oriented applications, carefully considering service boundaries for capabilities in composite environments. As composition frameworks become easier to use, and operational and support models evolve to the point that composite applications are as easy to run, then the power of drawing on many vendors will be a strong incentive to accelerate change.

A successful framework needs to significantly reduce the friction across every level of composite boundaries. Collaborative scenarios are not a prerequisite for a composite application but are a natural place for a lighter-weight solution that is relatively simple yet high in value. The ability to compose elements of an application will move upstream to technical business users as these platforms increasingly move toward model, workflow, and rules-driven approaches, and bring to the surface more configurable attributes from business logic.

To realize this, we need to achieve interoperability at both the syntactic and semantic levels with line-of-business applications. Solutions to a number of difficult problems need to become accessible to everyday developers to achieve a full-blown composition infrastructure. Each of these areas warrants an article on its own, and several have already been explored in depth by others. These include:

• Identity—In particular, to have mechanisms to universally recognize a common user identity either through single store, or through federation. As cloud-based services mature, identity will be an important service that emerges. To realize a model of Software as a Service (SaaS), cloud-based identity stores will need to federate with local identity stores. The identity and authorization credentials must seamlessly pass between composite clients, composite services, and back-end service logic. Even on the desktop, experiences that cross applications need to be seamless to be useful in the long run. This will require broad adoption of mechanisms for federating and propagating identity.
• Context—For the parts of a composite application to work together to deliver a rich experience for sophisticated applications, they must have a shared notion of context. Mechanisms for passing context between cooperating composite applications and within a composite application need to be addressed.
• Process Integration Between Composition Engines—Projecting a view of the process interface defined within a particular service facilitates wiring complementary logic. As an example, I could design a document workflow that exposes a user interface and manages a Microsoft Excel document for sales-pipeline management and integrates into a multistep, roll-up approval process for the forecasts running in my sales force automation (SFA) business application. By projecting that roll-up process interface into a design environment, I can more easily build the cooperative workflow that triggers events or actions in a back-end system. Additionally, I could use the projected process interface to understand and interpret the current state of the corresponding process in another system.
• Entity Definition—For composition to cross boundaries of applications, and for entities to be used in many formats, we need a central notion that separates the conceptual entity from the physical representation. Once the notion of entity is defined, it can be reused through different physical representations. A centrally managed definition of entity, and the relationships between entities, will extend across a composite experience. Nontrivial problems remain to be solved: How will entities be managed across organizational boundaries? How will platforms make it easier to transform entities from one form to another? Clearly standardization approaches in the past have had mixed results at best. Microformats are an interesting demand-driven approach to creating de facto standardization. Entity Data Management and .NET Language Integrated Query (LINQ) are the first steps in this direction for the Microsoft platform.
• Data/Information Management—This relates to the entity problem, but is slightly different. As mentioned in the previous section, today services and applications often don't have strong notions of data boundaries, or the mechanisms to make data boundaries enforceable while still delivering high performance. Notions of resource data, reference data, activity data (related to context), and message data need to be strongly identified and supported by design patterns, tooling, and infrastructure. Dealing with complexities like data ownership, data versioning, reference data syndication, and multiple valid versions is typically not considered when building applications today, nor does infrastructure make solving these issues easy.
• Eventing Infrastructure—In reality today, most Web services are request/response in nature. More sophisticated approaches supported by the WS-* protocol stack are needed for richer interaction patterns required by sophisticated applications.
• Repository/Discovery Mechanisms—UDDI gives one level of discoverability but has limitations. WS-Policy and WSMetaDataExchange add additional layers to discover information for services and capabilities. This area needs to continue to mature.
• Modeling and Metadata Frameworks—Developers need to increasingly be aware of, and surface, developed functionality through models and metadata to pass more control through configuration to technical business users—and, ultimately, end users—to control the way applications behave and are assembled. The model components will be assembled using workflow and rules-based systems. Many of the concepts within software factories reflect this shift to both build better domain-specific solutions for developers, and to also pass control up the organization away from developers where warranted. This will require careful analysis by application architects and developers to determine whether to give up some level of behavior control and where to constrain configurability to ensure that correctness and consistency are not violated.

Figure 4. Layers of the Composition Stack

Conclusion

Composability is a paradigm shift in computing from brittle, monolithic, developer-centric applications solving one particular problem, to agile, contextual, user-driven applications to support particular business processes. As with the shift to service orientation, this will require not only refactoring existing code, but also rethinking new code. It will also require new infrastructure to host the applications. The Microsoft 2007 Office System, especially with the inclusion of SharePoint server support and support of Workflow, is a first step toward these applications. In the future, the trend will continue so that users can effectively configure and control their own work environment, passing data seamlessly among applications, and perhaps across enterprise boundaries, to complete their tasks. Much still needs to be done to accomplish this vision, but spurred on by user expectations from advances in Web technology, it will be hard to turn back now.

Acknowledgments

I would like to give special thanks to Daz Wilkin and Jon Tobey for helping crystallize and refine this paper.

About the Author

Chris Keyser is the lead architect for Microsoft's Global ISV team. Chris likes to write code and has a strong interest in Web services, and distributed and real-time systems. He spent the decade prior to joining Microsoft working for a series of start-up companies (B2B/e-commerce, voice biometrics, and physical layer network switching), using a variety of technologies in real-time and business-system development. For the first five years out of college, Chris raised havoc in the United States Navy as a nuclear engineering officer on board the USS Virginia. That was a total immersion experience in systems engineering, an education that is very useful to this day. He graduated from college with a double major in computer science and history in 1984.

This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal website.

© Microsoft Corporation. All rights reserved.