Secrets of Great Architects

By Don Awalt and Rick McUmber
RDA Corporation

July 2004

Summary: All great architects have mastered the ability to conceptualize a solution at distinct levels of abstraction. By organizing the solution into discrete levels, architects are able to focus on a single aspect of the solution while ignoring all remaining complexities. Presents techniques for applying levels of abstraction to IT solutions, and compares this to other engineering disciplines. (16 printed pages)

Contents

Applying Levels of Abstraction to IT Solutions
Levels of Abstraction: A Powerful Weapon for All Engineers
Core Principles When Applying Levels of Abstraction
Applying Levels of Abstraction to IT Systems
Simple Framework: Four Levels of Abstraction
Evolve the Levels Through Iterations
Revisit the Levels of Abstraction Core Principles
Scaling Levels to Support Enterprise Solutions
Benefits
Conclusion
Self Assessment

Applying Levels of Abstraction to IT Solutions

Enterprise architects are challenged by the vast amount of complexity they face. It's one thing to develop an isolated, departmental application that automates a business task. However, it is quite another to design and assemble a worldwide network of IT labs filled with applications, servers, and databases that support tens of thousands of IT users, all supporting various business activities. Compounding the complexity, the IT network must always be available, responsive, and protect the corporation's precious information assets. In addition to all of this, the IT network must be flexible enough to support the ever-changing needs of the business and to adopt new technologies as they emerge.

Some architects clearly stand out and thrive in this complexity. We've been fortunate to work side by side with some truly great analysts and architects over our careers. Reflecting on these experiences, we've analyzed what makes an exceptional architect.

Without exception, all great architects have mastered the ability to conceptualize a solution at distinct levels of abstraction. By organizing the solution into discrete levels, architects are able to focus on a single aspect of the solution while ignoring for the moment all remaining complexities. Once they stabilize that part of the solution, they can then proceed to other aspects, continuously evolving and refining the layers into a cohesive model, one that can ultimately be implemented.

Most software developers know that they should decompose the solution into levels of abstraction. However, this is very difficult to apply in practice on actual projects. Upon encountering the first hurdle, it's easy to abandon the levels in the rush to start coding. Great architects work through the challenges and are disciplined to maintain the levels throughout the entire project lifecycle. They realize that if they don't, they'll eventually drown in the complexity.

This article presents techniques for applying levels of abstraction to IT solutions. We first demonstrate the approach through a simple example, and then propose a structure of system artifacts based on formal levels of abstraction.

Levels of Abstraction: A Powerful Weapon for All Engineers

Other engineering disciplines, such as civil engineers, have been coping with complexity by leveraging levels of abstraction for centuries. Let's study how other, more mature, engineering disciplines apply levels of abstraction, starting with the electrical engineers who design computer systems that grow increasingly complex with each new generation.

Hardware Engineers

System designers model computer systems using levels of abstraction. Each level is well defined and provides a distinct perspective of the system. Many systems are designed at three primary levels: system, subsystem, and component, as shown in Figure 1.

Layering enables the engineers to integrate an enormous amount of complexity into a single, working computer system. It's impossible to comprehend a computer strictly at the level of its atomic parts. There are approximately 25,000,000 transistors on a single Intel Itanium® chip alone.

This approach of breaking complexity down into layers of abstraction is of course not unique to IT-related disciplines. A similar approach is used in countless other disciplines, from aeronautical engineering to microbiology.

Core Principles When Applying Levels of Abstraction

All engineers follow this core set of principles when applying levels of abstraction. These principles also hold true when applying levels of abstraction to software.

The number and scope of the levels are well defined; in order for engineers to collaborate on a complex system, all team members must share the same understanding of the levels. So as designers make design decisions, they must file those decisions at the appropriate level of detail.

The three levels of abstraction are defined as follows:

Figure i. The three levels of abstraction defined

Figure ii. A simple framework for the levels of abstraction

Multiple views within each level

The complexity within a single level can become too much to grasp all at once. In this case, engineers present the design within a single level through multiple views. Each view presents a particular aspect of the design, yet remains at the same level of abstraction. For example, the motherboard engineer creates a view for each layer of the board, modeling the design of the connection pathways for that layer.

Figure 1. Levels of abstraction for a computer system

Must maintain consistency among the levels

In order for the system to function as intended, each subsequent layer must be a proper refinement of its parent layer. If the computer system designer switches from an IDE bus to a SCSI bus, then the interface specifications for all devices must also switch to SCSI. If the levels are not synchronized, the system won't perform as envisioned at the top level.

Applying Levels of Abstraction to IT Systems

Now that we've examined how other disciplines apply levels of abstraction, let's apply the technique to IT solutions1. The following sections present techniques for applying levels of abstraction to model the requirements, design, and implementation of a typical IT application. The techniques are presented through a simple, instructional example of an online order system for a hypothetical retailer. In our example, we not only include the architecture, but expand the scope to include the system requirements and the business context as defined by the retail industry.

Simple Framework: Four Levels of Abstraction

Our simple example defines the following four levels of abstraction for an IT solution:

• Domain
• Business process
• Logical
• Physical

Within each level, we present both the dynamic view and static view of the behavior for that particular level. Whereas the dynamic view models the messaging among the objects, the static view models the structure and relationships among the objects.

Domain Level of Abstraction

Applying the scoping rules above, the retailer serves as the black box central actor in the domain level. The customer serves as the external actor. The domain level is modeled from the perspective of the customer. Only the purchase interactions are modeled. The communication mediums used to complete the purchase are not included at this level but are introduced at the business process level.

Figure 2. Domain-level dynamic view for purchasing items from a retailer

Figure 3. Domain level static view for purchasing items from a retailer

Dynamic view

The dynamic view within the domain level models the interactions between the customer and the retailer. The figure below summarizes the domain context and contains a simple use case narration of the business interactions.

Figure 4. Business process level dynamic view for purchasing items online from a retailer

Static view

The static view of the domain level models the class structure and their relationships of the objects witnessed in the use case. In other words, it tells what objects the customer needs to understand in order to accomplish the purchase transaction at this level of abstraction. Figure 5 presents the class diagram for the static view of the domain level.

Figure 5. Business process level static view for purchasing items online from a retailer

The customer is an instance of Person. The relationship between the customer and the retailer is embodied as an Account. All Purchases are associated with the customer's Account. The Purchase is associated with each of the Items being purchased. Each Item is of a specific Product, where Product follows the meta-class pattern. Instances of Product are in effect classes themselves. Modeling Product as a meta-class makes our model more flexible in that adding additional Products to the Catalog is purely a data-driven process and doesn't impact the class model. Rounding out the classes, each Payment is associated with its Purchase.

As you can see, the model at this level is representative for most any retailer—online or brick-and-mortar, large or small. This illustrates why the [Industry] Domain Model should indeed define the company as the black box central actor. Companies in the same industry tend to support the same set of business interactions with their external actors. Moreover, domain models exclude the specific business processes of the company, as they can vary widely among companies in the same industry.

The domain level focuses strictly on business interactions viewed from the perspective of the external actor. We must be careful to not include implementation mechanisms for accomplishing the interactions. These details belong at the next level of abstraction. So in this case, we only model browse, select, purchase, and pay. We do not model how these interactions are accomplished—through telephone, US Mail, e-mail, Web application, in person, check, credit card, or cash.

Business Process Level of Abstraction

The next level of abstraction models the company's business processes for realizing the interactions that were captured at the domain level. The system level "zooms inside" the company black box and identifies all employees and systems that collaborate in order to perform the business transaction. At this level, the system to be developed serves as the black box central actor.

Applying the scoping rules for the system level, the online order system serves as the black box central actor. The customer and the employee serve as external actors. The system level is modeled from the perspective of the customer and the employee. The customer performs the purchase online. Payment is made by credit card. The order is fulfilled by shipping the items to the customer's shipping address. Notification of shipment is sent by e-mail.

Dynamic view

The dynamic view replays the domain level purchase transaction, this time exposing the internal business processes of the retailer. Figure 4 summarizes the business process context and contains a simple use case narration of the interactions among the system and its actors.

Static view

The static view at this level refines the class model to capture the objects witnessed in the business process level use cases. In other words, "What objects do the customer and employee need to understand in order to create an order online and fulfill the order?" Figure 5 presents the class diagram for the business process static view. We refine the domain class model to capture the perspective at this level of abstraction. The Person, Account, and Company abstractions remain the same, as well as Catalog and Product. However, the abstract Purchase event from the domain model is replaced with an Order.

Orders contain LineItems, which are associated with the Product in the Catalog. Since this level models the company's internal business process, we need to capture the inventory on hand (an attribute of a stock-keeping unit (SKU), which represents an inventory of items at a particular location). We also model the customer's UserAccount, which provides access to the online system. Payment is accomplished by using a CreditCardAccount. Location represents a geopoint within the US and serves as the billing address, as well as the Order's shipping address. Shipment contains Items included in the Shipment.

The system level of abstraction usually requires a great deal of creativity because it is at this level that we invent ways to streamline business processes. In doing so, it is common to realize a single domain level transaction using several different mediums at the business process level. For example, a purchase can be accomplished online, over the phone, by mailing or faxing an order form, or in-person at the retail store. Each of these would need to be modeled at the business process level. Notice that even though the Credit Authorizer is an external actor to the retailer, it is introduced at this level because it is only needed to implement a business process that first appears at this level.

Lastly, notice that the system is technology independent. Our online purchase system could be implemented with any Web technology. Selecting technologies inside the system black box is an architecture decision.

Logical Level of Abstraction

The logical level zooms inside the system black box, exposing the high-level design of the system. The architect selects the technology and defines the high-level system structure. In our simple example, the system is comprised of a Microsoft IIS/Microsoft ASP.NET server that hosts the presentation, business, and data access layers and a Microsoft SQL Server database server that hosts the persistent data.

Dynamic view

The dynamic view at the logical level traces the message flows through the major components of the system. As an example, Figure 6 traces the flow when submitting the ConfirmOrder Web form.

Figure 6. Logical level dynamic view for purchasing items online from a retailer

Static view

The static view at this level also switches our perspective to the system internals. Whereas the business process level modeled the real-world abstractions that appear in the business processes, this level models the abstractions as they are to be represented inside the system. In an actual system, the architect would design the classes for each software layer (presentation, business, and data access). To keep this article brief, Figure 7 presents just the static design for the business layer to show how the system-level abstractions are refined to the design.

Figure 7. Logical level static view for purchasing items online from a retailer

The architect refines the system-level classes to design the business-layer interface.

All accounts and customers in the system are the retailer's, so it's not practical to create a single instance of Company and associate it with all accounts, so Company is omitted at this level. Rather than creating a separate instance for each CreditCardAccount, we simply store the credit card number and billing address with the Payment. Further, it is not practical for the system to create an instance for every Item sold, so Item is removed from the model and instead the model tracks the quantity of items ordered in the LineItem and the quantity of items shipped in the new ShippedItems class.

The architect also defines the granularity of the services that the business layer exposes. For this example, the business layer exports Create, Read, Update, and Delete (CRUD) services for Account, UserAccount, Order, Shipment, and Catalog. The ellipses indicate the CRUDing granularity.

Note that even though the classes at this level are not a proper superset of the business process classes, the architect arrives at this design by directly refining the business process classes, changing the perspective from outside the system to inside the system.

Physical Level of Abstraction

The physical level of abstraction captures the structure of the system implementation. The system is implemented as a network of nodes, each node configured with hardware and software. The three software layers in the logical view (presentation, business, and data) are physically implemented as code and deployed onto these nodes. Persistent classes in the logical view are physically stored in relational tables in a SQL Server database.

Dynamic view

The dynamic view traces the message flows through the nodes of the physical configuration. The ConfirmOrder HTTP post flows from the customer's browser through the Internet through the retailer's firewall to the Web server where Microsoft Windows forwards it to IIS, which passes it to Microsoft ASP.NET, which then dispatches ConfirmOrder.aspx. Fortunately, modern development tools insulate us from the majority of the physical network. Architects, however, need to understand the physical layer in order to avoid network bottlenecks and security exposures.

Static view

The static view (Figure 8) refines the persistent classes in the logical view to their physical representation. In our retail example, the business-layer classes are stored in the following SQL Server tables.

Figure 8. Physical level static view for purchasing items online from a retailer

Classes map to relational tables and attributes are implemented as columns. One-to-one relationships and one-to-many relationships are implemented using a foreign key. Optimistic concurrency is implemented by assigning a datetime field to each parent class being CRUDed.

When designing the logical level, the architect focuses mainly on implementing system functionality. Confident that the system functionality is covered, the architect can focus on optimizing the implementation at the physical level.

Evolve the Levels Through Iterations

Having established this framework, the architect evolves the solution over several iterations. Each iteration incorporates additional functionality—invoices, back orders, order in person, order by phone, and so on. In each case, the architect updates the appropriate level of abstraction, and then refines the updates to the physical implementation level.

Revisit the Levels of Abstraction Core Principles

Let's test our example against the core level of abstraction principles.

• The number and scope of the levels are well defined: We have four distinct levels; company black box, system black box, logical design inside the system, physical implementation
• Multiple views within each level: In this simple example, we presented a dynamic view and static view at each level.
• Must maintain consistency among the levels: If a change is made to the domain model, the impact of the changes must flow down to the lower levels. For example, if the retailer decides to offer maintenance contracts for its products, the analyst would add MaintenanceContract to the domain model and refine it to its physical representation. Synchronizing all levels is critical for maintaining large systems. As enhancement requests are submitted, the analyst performs an impact assessment to the appropriate level of detail. Some enhancements impact the domain level (and therefore all subsequent levels). Others merely impact the physical level.

Scaling Levels to Support Enterprise Solutions

Now that we've presented a simple example with four levels of abstraction, let's scale the approach to support solutions for IT enterprises. Figure 9 presents a Rational Unified Process (RUP) configuration that organizes project artifacts in to well defined levels of abstraction.

The levels in the table are described below.

Figure 9. RUP configuration organizing project artifacts into well defined levels of abstraction

• Domain. The domain level captures the business context for a project.
• Project vision. The project vision communicates the business impact the system will have on the business. It quantifies this impact in a return on investment analysis. The project vision represents the highest level of abstraction of the project.
• Business process. The system level models the business processes within the company. For extremely complex organizations, this level can be subdivided into sublevels: division, interdepartmental, and intradepartmental.
• UI specification. The UI specification designs the user interface that realizes the business processes. It is comprised of a UI-design document and a functional UI prototype.
• Detailed requirements. The detailed requirements specify the lowest level abstraction of the system requirements. It includes details such as data type formats and detailed business rules. It also contains the proficiency requirements such as performance, availability, security, internationalization, configuration, scalability, and flexibility requirements.
• Architecture. The system architecture is organized into six views:
  • Logical. Defines the software layers and major abstractions that perform the system functionality.
  • Concurrency. Captures the parallel aspects of the system, including transactions, server farms, and resource contention.
  • Security. Defines the approach for authentication, authorization, protecting secrets, and journaling.
  • Deployment. Defines the network topology and deployment configuration of the system.
  • Component. Defines the system components, their interfaces and dependencies.
  • Data. Defines the design structure of the persistent data.

Benefits

Organizing system artifacts into discrete levels of abstraction delivers several benefits:

• It separates the system requirements into three distinct levels of abstraction: Business Processes, UI specification, and detailed requirements. No longer do we overwhelm the business users with a single, monolithic functional specification. Instead, we communicate the system requirements in three refined levels of detail.
• Analysts and architects are able to harness the complexity into a single, integrated model of the system.
• Architects can focus on a single aspect of the system and integrate those decisions into the overall solution.
• The levels of abstraction form the structure of the system artifacts. For example, the software architecture document dedicates a subsection for each view.
• The levels of abstraction provide direct traceability from requirements to design to implementation. Traceability enables a team to perform an accurate impact assessment when evaluating change requests.
• After developing several systems using the same framework, patterns emerge at each level of abstraction. Organizations can catalog these patterns and other best practices within each level of abstraction. This catalog of best practices serves as the foundation of a process improvement program.

Conclusion

All engineering disciplines apply formal levels of abstraction in order to cope with complexity. Software is no exception. In order to realize the benefits of levels of abstraction, projects must:

• Formally identify the layers, each with a well-defined scope.
• Split complexity within a level into multiple views.
• Maintain consistency among the levels.

This article demonstrated how to apply levels of abstraction through a simple example, then scaled the approach to support enterprise IT solutions. It offered a RUP configuration framework that organizes system artifacts into well-defined levels of abstraction.

Self Assessment

• Does your current project apply levels of abstraction?
• Are the levels well-defined?
• Are the levels well understood by the project team?
• If the complexity becomes too great within a level, does the team split it into views?
• Does the team maintain consistency among the levels?
• Would your project benefit from levels of abstraction?

Great architects follow these principles instinctively. The rest of us must consciously apply levels of abstraction and exercise discipline to maintain the levels throughout the project lifecycle.

Resources

Cockburn, Alistair. Writing Effective Use Cases. New Jersey: Addison-Wesley, 2001

Kroll, Per and Kruchten, Philippe. The Rational Unified Process Made Easy: A Practitioner's Guide to the RUP. Boston MA: Pearson Education and Addison-Wesley, 2003

DeMarco, Tom and Plauger, P J Structured Analysis and System Specification. Prentice Hall PTR, 1979

Online copy of DoD standard 2167A can be found at https://www2.umassd.edu/SWPI/DOD/MIL-STD-2167A/DOD2167A.html.

Footnotes

1   Many have successfully applied levels of abstraction to software. Ed Yourdon and Tom DeMarco proposed structured analysis and structured system design in 1979. Many branches of the US Government standardized on DoD's 2167A standard, which requires systems to be composed of a hierarchy of hardware and software configuration items. The DBA community frequently applies levels of detail to model relational databases. In particular, the Bachman toolset and James Martin's Information Engineering Methodology (IEM) model databases logically, then physically. A Google search of "software levels of abstraction" returns several results; however, most are from the academic community and seem to focus on formal computer languages.

About the author

Don Awalt is the CEO and founder of RDA Corporation, founded in 1988 as a custom software engineering firm with offices in Washington DC, Baltimore, Atlanta, Philadelphia, and Chicago. The firm, a Microsoft Gold Certified Partner, is focused on the development of enterprise Web and rich-client systems using the .NET Framework. Don currently serves as the Microsoft Regional Director for Washington DC, and was formerly the founding Regional Director for Philadelphia. Don is a frequent speaker at industry events, including Tech Ed, Developer Days, MSDN events, and various SQL Server and Windows events. He has served as a contributing editor for SQL Server Magazine and PC Tech Journal Magazine, and has written for other publications as well. Don's particular areas of expertise include Web services, SQL Server, the evolution of modern programming languages, and much of the architecture and process work seen in Microsoft's Prescriptive Architecture Group (PAG). Don can be reached at AWALT@rdacorp.com.

Rick McUmber is Director of Quality and Best Practices for RDA. For 11 years, he worked for IBM and Rational Software Corporation respectively, developing systems for the Department of Transportation, Department of Defense, NASA, and Canada's Department of National Defense. Since 1994, he has worked with RDA developing business solutions for its customers. Rick can be reached at McUmber@rdacorp.com.

This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal website.

© Microsoft Corporation. All rights reserved.