Service Compatibility

Microsoft Corporation

February 2008

Summary: This article describes what a tester should consider when updating services in a set of connected services. (6 printed pages)

Contents

Introduction

Testing Goals

Background

Test Requirements

Conclusion

More Information

Introduction

In a continuously running Internet portal (for example, Live.com, Hotmail.com, or Msn.com), multiple services typically are involved. As new versions of services are released, they are introduced into production via a rollout, which is the replacement of the current version on a computer with a new version. Similarly, "rollback" changes to the previous version from the current version. Before we roll out a new version, it must undergo testing to verify that it is a safe release. Beyond all of the functionality, performance, security, stress, and other tests, we have to look at compatibility.

Testing Goals

Service-compatibility testing verifies that:

• Rollout of the service has no negative effects.
• Use of the new service has no negative effects.
• Service can be rolled back without negative effects.

Background

Most services have multiple instances running in production. The load-balancing mechanisms might direct a call for service to any instance. Thus, during a rollout, two versions of the service must coexist—say, vN and vN+1. Callers to vN must be able to call vN +1 and get equivalent behavior. Similarly, vN +1 calls to other (older) services must demand no more than what vN did.

So, assuming a steady state system with no rollouts currently underway, we might have the following:

Note that client and server are just to indicate who initiates the call (caller) and who receives the call (callee). Furthermore, the client or server might be a partner that is not part of your feature team or part of your product. Note that the same is true when the communication is more indirect via shared data stores (for example, log writers/readers)—only even more complicated, as Y can "communicate with itself" via a data store from which it reads and writes (for example, persistent state). So, think also about your partner consumers and producers.

During rollout of service Y from vN to vN +1 (denoted as Y+), we have for the service-message protocols the following:

Typically, changing only Y means that there is no protocol change, because neither the caller (X) nor the callee (Z) of Y has changed. However, Y+ might rollout with a protocol change in anticipation of future or concurrent (as shown later) rollouts of X or Z.

Protocols can be updated in myriad ways. Two common update methods are transparent extension (compatible evolution) and explicit versioning (big bang). Transparent extension can occur by adding optional parameters. For example, say that X can send only three parameters to Y, and Y can return only two. Y+ must return only the same two parameters when dealing with X, but it can allow an optional fourth parameter (typically, coming only from X+). If it receives an optional fourth parameter, it can then return three parameters.

Note  Some protocols are built to "ignore" extra parameters—allowing Y+ to return excess parameters to X. However, with such "lazy" protocols, there are security implications.

Similarly, Y might be allowed to send only two parameters to Z, but Y+ might be built to accept an optional third parameter from Z (typically, a Z+1 version). Explicit versioning means that the collection of messages between two services is collected under a single version umbrella (for example, a versioned namespace). Changing any message means creation of a new version of the collection of messages. Typically, the start of a conversation between services identifies the version of the messages that are to be exchanged. It is possible also just to version every message.

For more coverage about this topic, see "5.2 Web Service Versioning" (http://www.w3.org/TR/wsdl20-primer/#adv-versioning) in the W3 working group WSDL 2.0 primer.

With data sharing, typically, Y+ might be rolled out able to "understand" new XY or ZY formats, but the only format to which it can change immediately might be the vN+1 format of its own data store.

How the versioning is done depends on the data-flow relationship. If Y merely is writing a log file, Y+ might intermix new (versioned) log records (assuming that the log reader is updated before or during the rollout!) into the same data store. If each instance of Y stores some independent state (for example, a cache), the Y would read/write to its store in vN format, while Y+ could read/write to its sore in vN+1 format. In rarer cases, the format is extensible, such that Y ignores extra or changed data in vN+1 formats. Y+ also might be rolled out with the vN+1 format disabled; then, after all instances are upgraded to Y+, a configuration change is used to allow vN+1 writing.

However, the preceding is just too easy. Frequently, along with your service Y, the surrounding services X and Z might be rolled out also as part of the release train. In this case, the situation looks something more like the following:

In that case, the explicit versioning model, X/Y, X/Y+, and X+/Y all would use version N of the protocol, while X+/Y+ could use a totally different N+1 version of the protocol. Similarly, Y/Z, Y+/Z, and Y/Z+ would use version M of their protocol, while Y+/Z+ could use a totally different M+1 version of their protocol.

Additionally, as alluded to earlier, it is possible for data formats to change as a release train rolls out. You must think about the producer and consumers of data in the same rollout.

A phased rollout should make sure that all data consumers are rolled out prior to data producers.

Test Requirements

We conduct a service-compatibility test prior to each release, to ensure compatibility across versions. We also take this opportunity to ensure that, for each of the services, we are able to rollback to the prior version.

Test should be done on a final release candidate (RC). A final RC is a build that has no crashes (or dumps) or active bugs, and on in which resolved bugs are triaged for any risk/impact that is pertinent to the backward-compatibility.

The italicized rows are only for services that use the additional step of turning on functionality via configuration, instead of just having it available in the new version (at rollout).

If new functionality becomes available just due to rollout, all interactions (protocols and data formats) that are due to old and new versions can be verified by rolling out just half when there are at least four computers total. Besides old and new interactions, we need two old versions to talk to each other and two new versions to talk to each other.

As the preceding table shows, tests for each service that verify that all of the callers and callees of the service behave normally must be provided and available. These tests will be run repeatedly for the various test bed states that were indicated earlier, so that the tests must be automated and relatively quick.

Make sure to test all new and old callee and caller scenarios.

Conclusion

It takes a lot of planning and testing to provide the confidence that updating services in a set of connected services can be done while providing site reliability. Testers must consider both direct message interactions and shared (for example, file) data or state information.

More Information

5.2 Web Service Versioning:

http://www.w3.org/TR/wsdl20-primer/#adv-versioning