Designing Extensible, Versionable XML Formats

 

Dare Obasanjo
Microsoft Corporation

July 15, 2004

Summary: Dare Obasanjo discusses what considerations you should make when versioning XML formats, as well as covering some approaches to designing extensible XML formats in a manner compatible with existing XML technologies. (23 printed pages)

An XML vocabulary should be designed in such a way that the applications that process it do not break when it is inevitably changed. One of the primary benefits of using XML for building data interchange formats is that the APIs and technologies for processing XML are quite resilient additions to vocabularies. If I write an application that loads RSS feeds looking for item elements, then processes their link and title elements using any one of the various technologies and APIs for processing, such as SAX, the DOM or XSLT, it is straightforward to build the application so that it is unaffected by changes in the RSS specification or extensions as long as the link and title elements always appear in a feed.

However, this approach gives the false impression that there are no versioning issues to consider when designing XML formats because you can always add elements and attributes to the format without harm. Experience has shown that this assumption is false and a lot of the same problems that face developers when versioning non-XML data formats now affect XML-based formats.

This article explores some of the points to consider when versioning XML formats, as well as some approaches to designing extensible XML formats in a manner compatible with existing XML technologies, such as W3C XML Schema.

Message Transfer Negotiation vs. Versioning Message Payloads

On wide area networks, such as the World Wide Web or intranets of global corporations, parties that exchange XML documents typically do so without apriori agreements. In such situations, it is difficult to ensure that clients and servers support the same versions of an XML format because upgrading applications at the various end points is not controlled by a central authority. This means that the XML formats that are used in such scenarios should implement a versioning policy that allows applications to support multiple versions of a format, and provide a mechanism for determining when the versions of a format supported by both parties are incompatible. The payloads of the XML messages exchanged by the parties must contain versioning information that can be used to determine if both parties can communicate successfully or not.

An alternative to addressing the communication problem directly is for both parties to agree upon which versions of a format they are using, either by some agreement before communication begins or through an automated negotiation process, such as WS-Policy. In an environment where an XML format is only used by parties that agree upon which versions of the format they support in out-of-band communications, it is not necessary for the format to contain any versioning constructs besides an identifier indicating which version of the format is being represented.

This article will focus on versioning of XML formats where versioning constructs are placed in the actual message payload.

Version Numbers vs. Namespace Names

Many designers of XML formats often have to struggle with determining the relationship between namespace names and version numbers when creating a versioning policy for an XML format. The namespace name of an element or attribute is part of its identity. The name of an element or attribute is syntactically in the form of a qualified name, also known as a QName.

The QName is an XML name called the local name, optionally preceded by another XML name called the prefix and a colon (':') character. The prefix of a qualified name must be mapped to a namespace URI through an in-scope namespace declaration, mapping the prefix to the namespace URI. Although prefixes are important mnemonic guides to determining what namespace the elements and attributes within a document are derived from, they are rarely important to XML aware processors. For example, the following three XML documents would be treated identically by a range of XML technologies including, XML schema validators.

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
        <xs:complexType id="123" name="fooType"/>
</xs:schema>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <xsd:complexType id="123" name="fooType"/>
</xsd:schema>
<schema xmlns="http://www.w3.org/2001/XMLSchema">
        <complexType id="123" name="fooType"/>
</schema>

The W3C XML Path Language recommendation describes an expanded name as a pair consisting of a namespace name and a local name. A universal name is an alternate term coined by James Clark to describe the same concept. To many XML applications, the universal name of the elements and attributes in an XML document are what is important, and not the values of the prefixes used in specific QNames.

This means that changing the namespace name of an XML vocabulary renames all the elements and global attributes in the vocabulary to namespace-aware XML application, such as XPath, XSLT, XML parsers, and a host of other technologies.

Traditionally, version numbers in the software world have at least two parts—major version number and a minor version number. A release that updates the major version number of a software system contains changes that make it incompatible with the prior major release. This means that applications dependent on an earlier major release may need to be changed to work with the new release. On the other hand, a release that updates the minor version number of the software is a backwards-compatible change. This means that an application written against version 1.0 will work fine when targeted against versions 1.1 and 1.2, but may fail if moved to version 2.0.

An XML format should use version numbers to differentiate between different versions of the format, preferably in a version attribute on the root element of the format. The major and minor versioning scheme from traditional software development practices should be used to help users identify compatibility between versions in a straightforward manner. If the XML format is namespace qualified, then the namespace name should be the same between minor versions and should be changed with major versions.

It should be noted that this practice is actually not followed much in the XML world. XSLT 2.0 uses the same namespace as XSLT 1.0, while SOAP 1.1 uses a different namespace name from SOAP 1.2. However, despite this indifference I believe following the aforementioned guidelines provides a rational way to reason about what the relationship between version numbers and namespace names should be in an XML format.

The Difference Between Versioning and Extensibility

Enabling extension of the format by parties other than the designers of the format, thus decentralizing the evolution of the format, is a desirable feature and one that is often touted as a benefit of XML. A good example of the enabling nature of extensible XML formats is RSS and the various RSS modules that extend the functionality provided in the base specifications. Designing a format so that it supports third party extensions is the extensibility problem.

An XML format should be both backward and forward compatible. It should be backward compatible in that the new versions of the format should also be valid instances of older versions of the format, and thus not break any consumers of the original format. It should be forward compatible in that older versions of the format should also be valid instances of newer versions of the format, so old producers can work with consumers of the new format.

There are a four broad classes of changes that could occur in the process of transitioning from one version of a format to another.

  1. New concepts are added (new elements or attributes added to format or new values for enumerations).
  2. Existing concepts are changed (existing elements and attributes should be interpreted differently, added elements or attributes alter semantics of their parent/owning element).
  3. Existing concepts are deprecated (existing elements and attributes should now issue warning when consumed by an application).
  4. Existing concepts are removed (existing elements and attributes should no longer work when consumed by an application).

Some of these changes are backward incompatible. Specifically, changing the semantics of existing concepts and removing existing concepts from the format are backward-incompatible changes.

Designing XML formats so that changes can occur between versions of a format, yet the format remains backward and forward compatible is the versioning problem. In additio,n a versioning scheme must also provide a mechanism for communicating incompatibility.

Versioning differs from extensibility in two broad ways. The first major difference is that versioning mechanisms must support change in a linear fashion, while extensibility mechanisms support change in a concurrent fashion. A versioning mechanism must define a mechanism to create a version 1, a backward compatible version 2, a backwards compatible version 3, and so on.

On the other hand, rather than defining an evolutionary process, an extensibility mechanism provides a way to allow new data to show up side by side (or concurrently) with data for a given format. Secondly, changing an XML format by creating subsequent versions is usually done by the entity that controls the format, while extensions are typically added by third parties. In practice, this tends to mean that versioning is done by the owner of the namespace of the XML format.

An ideal XML vocabulary is both extensible and versionable.

Guidelines for Designing Extensible XML Formats

In the article Versioning XML Vocabularies, David Orchard provided some guidelines for designing extensible XML formats. The guidelines below are slightly modified from some of the guidelines in Orchard's article

  • XML formats should be designed to be extensible.
  • Extensions must not use the namespace of the XML format.
  • All XML elements in the format should allow any extension attributes, and elements with complex content should allow for extension elements as children.
  • Formats that support extensibility must specify a processing model for dealing with extensions.

The guidelines above differ from those in Orchard's article in a few key ways. The major difference is that the guideline about specifying a processing model for extensions is upgraded from a should to a must. The reason I changed this rule is that without explicit and consistent rules for consumers of the format when dealing with extensions, then interoperability across implementations will suffer.

Another difference is that the guideline about allowing extension elements has specifically called out that elements with simple content (text content) shouldn't be allowed to have extension elements. The following discussions explore each of the guidelines in more detail.

Why XML Formats Should Be Designed to Be Extensible

The primary benefit of allowing extensibility in a format is that it enables a format to evolve without requiring central control of the format. A secondary benefit is that it allows the format to stay focused and simple by pushing specialized or niche-use cases and complex solutions into optionally supported extensions.

An example of the benefit of designing a format to be extensible is RSS 2.0 and the various RSS modules. The core RSS specification defines how to provide basic information, such as the title, description, and publication date of one or more entries in a syndication feed. Various extensions have been designed which enable much richer functionality, such as providing information about the number of comments posted in response to an entry and mechanisms for retrieving the comments to an entry as a separate RSS feed. These extensions have proliferated without the need to modify the RSS specification yet are not mandatory, so Web sites and news aggregators do not have to bear the cost of implementing complex features if they just need simple content syndication.

Another showcase of the benefit of extensibility in XML formats is the W3C XML Schema recommendation. The XML Schema recommendation allows one to place extensions to the specification, either as namespace qualified attributes on elements in the XML schema namespace or as element children of the xs:appinfo element.

Extensions to XML schema have been used to augment its functionality in a number of ways. One example is the ability to embed Schematron assertions in an XML schema to check constraints that are beyond the capability of W3C XML Schema. Another example, are the XML schema annotations used by Microsoft's SQLXML 3.0, which are used for mapping between XML and relational schemas.

Annotated schemas used by SQLXML 3.0 can still be used for validating XML documents, but have the added benefit of also being able to be used for shredding XML data into relational tables and vice versa. These extensions increase the utility of XML Schema without increasing the complexity for all consumers of XML Schema documents given that they are primarily of benefit in specialized scenarios.

It should be noted that extensibility is a double edged sword. The fact that evolution of an extensible format is decentralized may harm interoperability in certain cases because not all clients will support the same extensions.

Why Extensions Mustn't Use the Namespace of the XML Format

There are two main reasons extensions must use their own namespace name. The first reason is that it is important that each family of extensions be distinguishable from the core components of the XML format and other extensions. Without providing such identification there is potential for naming conflicts between different extensions or between extensions and future additions to the core specification.

Secondly, there should be a straightforward way to go from identifying an extension to learning more about it. If the namespace name of the extensions is an HTTP URI that points to human- and machine- readable information about the extensions, then it allows consumers of the format the opportunity to learn about the extensions they encounter.

Why all XML elements in the format should allow any extension attributes and extension elements

Once the decision has been made to make a format extensible, the next question is where one should allow extensibility. One could restrict the elements whose content model can be extended or annotated using attributes, but this may end up unnecessarily restricting the usefulness of extensions. Consider the following XML fragment:

 <books xmlns='http://www.example.com/books'>
   <book publisher="Addison Wesley">
    <title>Mythical Man Month</title>
    <author>Frederick Brooks</author>
    <publication-date>1995-06-30</publication-date>
  </book>
   <book publisher="Apress">
    <title>Programmer's Introduction to C#</title>
    <author>Eric Gunnerson</author>
    <publication-date>2001-06-30</publication-date>
  </book>
 </books

in the above document there are multiple areas one could augment the information that is provided. The book element could be extended with an ext:edition attribute that indicates what edition of the book is being described. An ext:price element could be added to the content model of book elements describing the price of the book. The author element could be augmented with an ext:is-editor, which is used on collected works such as anthologies to indicate that the specified author was actually an editor. And so on.

The point is that in an XML format it is likely that a lot of the structured data (elements) in the document can be extended or annotated in a way that adds more value to the data being transmitted. Given this situation, it is likely that designers of XML formats may not be able to anticipate the various ways the data in a format may be augmented or annotated. Thus, if the author(s) of an XML format restrict which elements can be augmented by extensions, there is the possibility that useful extensions may be prohibited by such restrictions. Allowing all elements in the XML format to be extended ensures that no useful extensions are prohibited.

When authoring a schema for an XML format using W3C XML Schema, one uses xs:anyAttribute to allow extension attributes to appear on an element and xs:any to allow extension elements to appear as children of an element.

Why formats that support extensibility must specify a processing model for dealing with extensions

Another key decision that must be made once it has been decided to use extensibility is what the processing model should be for handling extensions in consumers of the format. As pointed out by David Orchard in his article Designing XML Vocabularies, the most popular processing model for dealing with extensions has been the use of Must Ignore rules in combination with mustUnderstand constructs.

With Must Ignore rules in place, consumers of the format are expected to ignore extensions they do not understand. In the case of extension elements, this could take one of two forms. Presentation formats such as XHTML only ignore the unknown start and end tags for the extension elements, but still process their contents. Orchard calls this the Must Ignore Container rule. In most other situations, XML formats apply what Orchard called the Must Ignore All rule where the extension element and its children are ignored by the consumer if they are not understood. This implies that ignoring extensions is not fatal to the application and the data they contain is not significant.

In certain cases, an extension could be introduced to an XML format containing data that is significant to the application and should not be ignored by consumers of the format. This situation is especially likely in the case of XML formats that mainly act as containers or envelopes for more specialized data, such as SOAP. If such scenarios are intended to be supported then designers of the XML format should consider adding mustUnderstand constructs to the vocabulary. An example of a mustUnderstand construct is the SOAP mustUnderstand attribute. The rules for the mustUnderstand attribute in SOAP are given below

The SOAP mustUnderstand global attribute can be used to indicate whether a header entry is mandatory or optional for the recipient to process. The recipient of a header entry is defined by the SOAP actor attribute (see section 4.2.2). The value of the mustUnderstand attribute is either "1" or "0".

The absence of the SOAP mustUnderstand attribute is semantically equivalent to its presence with the value 0. If a header element is tagged with a SOAP mustUnderstand attribute with a value of 1, the recipient of that header entry either MUST obey the semantics (as conveyed by the fully qualified name of the element) and process correctly to those semantics, or MUST fail processing the message

XML formats that intend to allow extensions that may be required to be understood by consumers of the vocabulary should use a mustUnderstand construct such as namespaced attribute, which must appear on the extension element.

Other processing models for extensions are possible. One approach could be making the Must Understand rule the default rule for the format, meaning that a consumer must always fail if it does not understand an extension. Another approach is restricting the structure of extensions so that they can be provided to the end user in a consistent manner by consumers of the format. For example, restricting extensions to key-value pairs in XML configuration files.

Using XML Schema to Design an Extensible XML Format

W3C XML Schema provides a number of features that promote extensibility in XML vocabularies, such as wildcards, substitution groups, and xsi:type. I've written about a number of techniques for adding extensibility to XML formats using W3C XML Schema in my article W3C XML Schema Design Patterns: Dealing With Change. If you have read that article, then I suggest skipping this section.

  1. Using wildcards to create open content models: The wildcards xs:any and xs:anyAttribute are used to allow the occurrence of elements and attributes from specified namespaces into a content model. Wildcards allow schema authors to enable extensibility of the content model while maintaining a degree of control over the occurrence of elements and attributes. The most important attributes for wildcards are namespace and processContents. The namespace attribute is used to specify the namespace from which elements or attributes the wildcard matches can come from, while the processContents attribute is used to specify if and how the XML content matched by the wildcard should be validated.

  2. Gaining flexibility from substitution groups and abstract elements: A substitution group contains elements that can appear interchangeably in an XML instance document in a manner reminiscent of subtype polymorphism in OOP languages. One or more elements can be marked as being substitutable for a global element (also called the head element), which means that members of this substitution group are interchangeable with the head element in a content model. The only requirement is that the members of the substitution group must be of the same type or be in the same type hierarchy as the head element.

    The following is an example schema and the instance that it validates:

    example.xsd:
     <xs:schema   
     xmlns:xs="http://www.w3.org/2001/XMLSchema"
           targetNamespace="http://www.example.com"
           xmlns:ex="http://www.example.com"
           elementFormDefault="qualified">
    
      <xs:element name="book" type="xs:string" />
    
      <xs:element name="magazine" type="xs:string" substitutionGroup="ex:book" />
    
     <xs:element name="library">
     <xs:complexType>
      <xs:sequence>
       <xs:element ref="ex:book" maxOccurs="unbounded"/>
      </xs:sequence>
     </xs:complexType>
     </xs:element>
    
    
    </xs:schema>
    
    example.xml:
    <library xmlns="http://www.example.com">
     <magazine>MSDN Magazine</magazine>
     <book>Professional XML Databases</book>
    </library>
    

    The content model of the library element says that it can hold one or more book elements. Since magazine elements are in the book substitution group, it's valid for magazine elements to appear in the instance XML where book elements are expected.

    Substitution groups make content models more flexible and allow extensibility in directions that the schema author may not have anticipated. This flexibility is a double-edged sword. Although it allows greater extensibility, it makes processing documents based on such schemas more difficult. For instance, the code that processes the library element must not only handle its child book elements, but magazine elements as well. If the instance document specified additional schemas using the xsi:schemaLocation attribute, the processing application could have to deal with even more members of the book substitution group as children of the library element.

    Another complication is that members of a substitution group can be of a type derived from the substitution group's head. Writing code to properly handle any derived type generically is difficult, especially since there are two opposing notions of derivation. The first, restriction, restricts the range or values in the content model. The second, extension, adds elements or attributes to the content model.

    Certain attributes on element declarations can be used to give schema authors more control over element substitutions in instance documents and reduce the likelihood of unexpected substitutions in XML instance documents. The block attribute is used to specify whether elements whose types use a certain derivation method can substitute for the element in an instance document, while the final attribute is used to specify whether elements whose types use a certain derivation method can declare themselves to be part of the target element's substitution group. The default values of the block and final attributes for all element declarations in a schema can be specified using the blockDefault and finalDefault attributes of the root xs:schema element. By default all substitutions are allowed without limitation.

    It is a good practice to set the value of the blockDefault attribute to #all on the xs:schema element, thus preventing unwanted type substitutions, and then only overriding this value using the block attribute of specific elements. This controls where extensibility using type substitution is allowed. Typically, such extensibility is done using abstract elements. An element declaration that is marked abstract indicates that a member of its substitution group must appear in its place in the instance document.

    A schema designer can build an extensibility point into a schema by defining an abstract element which must be replaced by subtypes defined as extensions that are members of the abstract element's substitution group. For example, one could create an Address substitution group with members USAddress and UKAddress. The abstract element Address is used in the schema as a placeholder but it must be substituted by a USAddress or a UKAddress in instance documents.

  3. Runtime polymorphism via xsi:type and abstract types: The xsi:type attribute can be placed on an element in an XML instance document to change its type as long as the new type is in the same type hierarchy as the original type of the element. The effect of xsi:type is similar to substitution groups except that instead of changing the name and content model of the element, it only changes the content model of the element. Below is a sample schema and an instance document that utilizes xsi:type.

    example.xsd:
      <xs:schema   
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
            targetNamespace="http://www.example.com"
            xmlns:ex="http://www.example.com"
            elementFormDefault="qualified">
    
      <xs:element name="book" type="xs:string" />
    
      <xs:complexType name="bookWithIsbnType">
        <xs:simpleContent>
          <xs:extension base="xs:string">
           <xs:attribute name="isbn" />
          </xs:extension>
        </xs:simpleContent>
      </xs:complexType>
    
     <xs:element name="library">
      <xs:complexType>
       <xs:sequence>
        <xs:element ref="ex:book" maxOccurs="unbounded"/>
       </xs:sequence>
      </xs:complexType>
     </xs:element>
    
    </xs:schema>
    
    example.xml:
       <ex:library xmlns:ex=http://www.example.com
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 
     <ex:book>Mort</ex:book>
     <ex:book xsi:type="ex:bookWithIsbnType" 
               isbn="0-06-105764-9">Feet of Clay</ex:book>
    </ex:library>
    

    The most common usage of xsi:type in the practice is to name the specific type of a primitive value that was specified as xs:anySimpleType in a schema.

    Abstract types are complex type definitions that have true as the value of their abstract attribute, which indicates elements in an instance document cannot be of that type, but instead must be replaced by another type derived either by restriction or extension. Although it is not necessary to use abstract types in conjunction with xsi:type, if a generic format is being created for which most users will create domain specific extensions, then they provide some benefit.

  4. Using xs:redefine to update type definitions: The types in a schema can be redefined in a process whereby the type effectively derives from itself. xs:redefine, used for redefinition, performs two tasks. The first is to act as an xs:include element by bringing in declarations and definitions from another schema document and making them available as part of the current target namespace. The included declarations and types must be from a schema with the same target namespace, or it must have no namespace. Second, types can be redefined in a manner similar to type derivation with the new definition replacing the old one.

    Type redefinition is pervasive because it not only affects elements in the including schema, but also those in the included schema as well. Thus, all references to the original type in both schemas refer to the redefined type, while the original type definition is overshadowed. Using xs:redefine doesn't provide extensibility in the traditional sense, but instead allows one to effectively alter the definitions of types in a given schema.

    The expected usage of the xs:redefine element by the W3C XML Schema working group was for versioning XML schemas. It was assumed that when creating subsequent versions of an XML format, schema authors would use xs:redefine in later versions of the schema to alter the definition of types from previous versions. However, this has not seen widespread usage in practice.

Guidelines for Designing Versionable XML Formats

The following are guidelines for designing XML formats in a way that makes them resilient in the face of changes that occur in subsequent versions, and these guidelines are also modified from those in David Orchard's Versioning XML Vocabularies article

  • If the next version of a format is backward compatible with previous versions, then the old namespace name must be used in conjunction with XML's extensibility model.
  • A new namespace name must be used when backward compatibility is not permitted. That is, software must break if it does not understand the new language components.
  • Formats should specify a mustUnderstand model for dealing with backward incompatible changes to the format that don't change the namespace name.

The following discussions explore each of the above guidelines in more detail.

Why the same namespace name should be used for backward compatible versions of a format

The namespace of an element or attribute is part of its identity. Changing the namespace name of an XML vocabulary renames all the elements and global attributes in the vocabulary to namespace-aware XML application such as XPath, XSLT, XML parsers, and a host of other technologies. If the new version of the format is backwards compatible with the original version of the format then elements and global attributes should retain the same names so as not to break namespace aware applications that consume the format.

This means that the namespace name of a format should not be the primary mechanism for indicating what version of the format is being represented. Instead, an alternate means such as a version attribute on the root element of the format should be used.

Why a new namespace name must be used when backward compatibility is not permitted

As mentioned in the previous section, changing the namespace name of an XML vocabulary renames all the elements and global attributes in the vocabulary to namespace-aware XML application such as XPath, XSLT, XML parsers, and a host of other technologies. In certain cases, changes to an XML format can make it differ drastically from one version to the next in a backwards incompatible manner. In such cases, it is best to change the namespace name so namespace aware XML applications rightly fail to identify the new version of the format as being the same as the original and reject documents in the new format.

Why XML formats should specify a mustUnderstand model for dealing with backward incompatible changes to the format.

If a newer version of an XML format is not backward compatible with its predecessor but does not use a new namespace name, then there should exist a way to tell consumers of the format to error on changed or new constructs that they do not understand. A simple solution is for the format to provide a version number that can be tested by consumers on its root element before processing the XML document. In this case, the mustUnderstand model is that the consumer must understand all elements from the target namespace of the format if it supports the version number specified on the root element.

In cases where new elements are added to the format that are not backward compatible with older versions of the format, it may be best for such elements to be tagged with a mustUnderstand attribute. Doing this ensures that there is still some degree of interoperability because as long as the producer generates documents in the new format that do not contain the new constructs, then all is well.

For example, imagine an XML-based query language that adds update constructs in a newer version (create, replace, update, delete, and so on). In such a situation, a producer of the format that has upgraded to the newer version can still generate documents that contain the original query constructs in the language without worrying about compatibility. However, if the producer is generating documents using the new constructs, it adorns them with mustUnderstand attributes whose value is true, which indicates to older clients that they are to fail if they don't understand how to perform a delete (for example).

It should be noted that the mustUnderstand construct does not have to be an attribute. A limitation of using an attribute is that it isn't straightforward to use it to mark a new attribute as having to be understood. Another drawback of using an attribute is that it has to be repeated on each occurence of an element that must be understood, which is needlessly repetitive if that element appears multiple times in a document. Another approach could be specifying a mustUnderstand element that identifies which new items must be understood.

Using XML Schema to Design a Versionable XML Format

The most straightforward mechanism for versioning an XML format would be to eschew the use of namespaces or XML schemas and simply utilize a major/minor version number scheme for communicating backward compatible versions of the format. However, many XML formats have to contend with using namespaces and interacting with XML schemas, especially XML formats that will be used in XML Web Service scenarios. This section describes some general approaches to providing a versioning policy for an XML format that are compatible with W3C XML Schema. The following approaches provide mechanisms for describing XML formats using W3C XML Schema in a way that enables evolution in a backward compatible way.

  1. New constructs in a new namespace: The most straightforward versioning mechanism is to specify that additions to the format should be in a different namespace from the core components of the format. To make this backward compatible, the XML format should have an extensibility model with default Must Ignore rules for items outside the namespaces the consumer understands in combination with mustUnderstand constructs.

    The following examples show version 1 of XML schemas that describe a collection of books and an XML document that conforms to the schema.

    BOOKS-CORE.XSD    
    <xs:schema blockDefault="#all"
     xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.example.com/books-core">
    
      <xs:attribute name="mustUnderstand" type="xs:boolean" />
    
    </xs:schema>
    
    BOOKS-V1.XSD
     <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/books/v1" 
          xmlns:b1="http://www.example.com/books/v1"> 
    
      <xs:element name="books">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="book" type="b1:bookType" 
              maxOccurs="unbounded" />
          </xs:sequence>
          <xs:attribute name="version" type="xs:string" />
        </xs:complexType>
      </xs:element>
    
      <xs:complexType name="bookType">
        <xs:sequence>
          <xs:element name="title" type="xs:string" />
          <xs:element name="author" type="xs:string" />
          <xs:any namespace="##other" minOccurs="0" maxOccurs="unbounded" 
             processContents="lax" />
        </xs:sequence> <xs:attribute name="publisher" type="xs:string" />
       </xs:complexType>
    
     </xs:schema>
    
    BOOKS.XML
    <books version="1.0" xmlns="http://www.example.com/books/v1">
      <book publisher="IDG books">
        <title>XML Bible</title>
        <author>Elliotte Rusty Harold</author>
      </book>
      <book publisher="Addison-Wesley">
        <title>The Mythical Man Month</title>
        <author>Frederick Brooks</author>
      </book>
      <book publisher="WROX">
        <title>Professional XSLT 2nd Edition</title>
        <author>Michael Kay</author>
        <price xmlns="http://www.example.com/book/extensions">24.99</price>
      </book>
    </books>
    
    

    The schema for the http://www.example.com/books/v1 namespace describes the books element, which can contain one or more book elements, which have an author and title element as well as a publisher attribute. The content model of the book element allows for zero or more elements from any namespace besides the target namespace of the schema to appear after the author and title elements. The schema for the http://www.example.com/books-core namespace contains a mustUnderstand attribute, which must be added on extension elements or elements from a future version of the format that a consumer must know how to process.

    In the next version of the format, it is decided that an isbn element should be added to the content model of the book element. Since all consumers and producers of the Example.com XML Book Format won't upgrade at the same time, there will be times when a party consuming version 2 of the format will not understand the isbn element. Because the importance of the isbn element is dependent on the application, it is decided that the isbn element can appear with a mustUnderstand attribute indicating that applications consuming the format do not need to know how to process ISBNs.

    In general, it is a good practice for the value of the mustUnderstand attribute to be fixed for elements introduced in subsequent versions of a format. This ensures that the new constructs are treated consistently across various applications that consume that format.

    Below is the schema for version 2 of the format along with a sample document:

    BOOKS-V1.XSD
    <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/books/v1" 
          xmlns:b1="http://www.example.com/books/v1"
          xmlns:b2="http://www.example.com/books/v2"> 
    
      <xs:import namespace="http://www.example.com/books/v2" 
          schemaLocation="books-v2.xsd" />
    
      <xs:element name="books">
        <xs:complexType>
          <xs:sequence>
           <xs:element name="book" type="b1:bookType" 
              maxOccurs="unbounded" />
          </xs:sequence>
          <xs:attribute name="version" type="xs:string" />
        </xs:complexType>
      </xs:element>
    
      <xs:complexType name="bookType">
        <xs:sequence>
          <xs:element name="title" type="xs:string" />
          <xs:element name="author" type="xs:string" />
          <xs:element ref="b2:isbn"  /> 
          <xs:any namespace="##other" minOccurs="0" maxOccurs="unbounded"  processContents="lax" />
        </xs:sequence> <xs:attribute name="publisher" type="xs:string" />
       </xs:complexType>
    
     </xs:schema>
    
    BOOKS-V2.XSD
    <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/books/v2" 
          xmlns:core="http://www.example.com/books-core">
    
      <xs:import namespace="http://www.example.com/books-core" 
          schemaLocation="books-core.xsd" />
    
     <xs:element name="isbn">
      <xs:complexType>
       <xs:simpleContent>
         <xs:extension base="xs:string">
           <xs:attribute ref="core:mustUnderstand" fixed="false"/>
         </xs:extension>
       </xs:simpleContent>
      </xs:complexType>
     </xs:element>
    
    </xs:schema>
    
    BOOKS.XML
    <books version="2.0" xmlns="http://www.example.com/books/v1"
                         xmlns:p="http://www.example.com/book/extensions"
                         xmlns:v2="http://www.example.com/books/v2"
               xmlns:bc="http://www.example.com/books-core">
      <book publisher="HCI">
        <title>A Child Called It</title>
        <author>Dave Pelzer</author>
        <v2:isbn bc:mustUnderstand="false">1-55874-766-9</v2:isbn>
        <p:price>9.95</p:price>
      </book>
    </books>
    

    The primary drawback of this approach is that core components of the format are not in the same namespace. This makes it tricky for applications or human readers of the format to differentiate between extensions and core aspects of the format that show up in a later version.

    A secondary drawback is that although this approach is backward compatible (v2 documents can be consumed by v1 clients) it is not forward compatible. The v2 schema states that an isbn is mandatory, which is not the case in v1, meaning that a v1 document will be rejected by a v2 client. Switching the isbn element to being optional doesn't work because it makes the schema non-deterministic. It is non-deterministic because when an isbn element is seen, the validator cannot tell whether the sequence is over because the element may be validated as the optional isbn element that follows an author or against the wildcard, which allows any element in a namespace other than the target namespace to appear.

    Both of these drawbacks are tackled by the approach described next.

  2. Using extensibility points with sentries: Conceptually, a data format is made versionable by providing a well defined extensibility point where additions to the format are expected to appear. This functionality is provided in W3C XML Schema using wildcards. However, in practice simply placing a wildcard at a particular point in a content model often leads to non-deterministic content models. The following example shows a non-deterministic schema that seems like it should work.

    <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/incorrect" 
    
    <!-- THIS TYPE IS NON-DETERMINISTIC --> 
     <xs:complexType name="bookType">
        <xs:sequence>
          <xs:element name="title" type="xs:string" />
          <xs:element name="author" type="xs:string" />
          <xs:element name="isbn" type="xs:string" minOccurs="0" />
          <xs:any namespace="##targetNamespace ##other" minOccurs="0" maxOccurs="unbounded" />
        </xs:sequence>
        <xs:attribute name="publisher" type="xs:string" />
       </xs:complexType>
    
     </xs:schema>
    

    As mentioned earlier, the problem with the above schema is that when an isbn element is seen, the validator cannot tell whether the sequence is over because the element may be validated as the optional isbn element that follows an author or against the wildcard that allows any element in a namespace other than the target namespace to appear. This limitation is due to the Unique Particle Attribution Constraint of XML Schema. To make use of wildcards deterministic in such situations one can provide delimiters or sentry elements around the wildcard that helps the validator determine when the elements to validate against the wildcard begin and when it end.

    The following examples show version 1 of XML schemas that describe a collection of books and an XML document that conforms to the schema.

    BOOKS-CORE.XSD
    <xs:schema blockDefault="#all"  
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.example.com/books-core">
    
      <xs:element name="delimiter">
        <xs:complexType /> 
      </xs:element>
    
      <xs:element name="end">
        <xs:complexType /> 
      </xs:element>
    
      <xs:attribute name="mustUnderstand" type="xs:boolean" />
    </xs:schema>
    
    BOOKS.XSD
     <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/books"
          xmlns:b="http://www.example.com/books"
          xmlns:bc="http://www.example.com/books-core">
    
      <xs:import namespace="http://www.example.com/books-core" 
    schemaLocation="books-core.xsd" />
    
      <xs:element name="books">
        <xs:complexType>
          <xs:sequence>
       <xs:element name="book" type="b:bookType" 
              maxOccurs="unbounded" />
          </xs:sequence>
          <xs:attribute name="version" type="xs:string" />
        </xs:complexType>
      </xs:element>
    
      <xs:complexType name="bookType">
        <xs:sequence>
          <xs:element name="title" type="xs:string" />
          <xs:element name="author" type="xs:string" />
          <xs:element name="isbn" type="xs:string"  minOccurs="0" />
          <xs:sequence minOccurs="0" maxOccurs="1">
       <xs:sequence minOccurs="0" maxOccurs="unbounded">    
         <xs:element ref="bc:delimiter" /> 
         <xs:any namespace="##targetNamespace ##local" minOccurs="0" maxOccurs= "unbounded"/> 
       </xs:sequence>
       <xs:element ref="bc:end" />
          </xs:sequence>
          <xs:group ref="b:extensionGroup" minOccurs="0" />
        </xs:sequence>
          <xs:attribute name="publisher" type="xs:string" />
       </xs:complexType>
    
     <xs:group name="extensionGroup">
        <xs:sequence>
          <xs:element name="extensions">
             <xs:complexType>
            <xs:sequence>
              <xs:any namespace="##other" minOccurs="0" 
    maxOccurs="unbounded"  processContents="lax" /> 
             </xs:sequence>
            </xs:complexType>
          </xs:element>
        </xs:sequence>
      </xs:group>
    
     </xs:schema>
    
    BOOKS.XML
    <books version="1.0" xmlns="http://www.example.com/books">
      <book publisher="IDG books">
        <title>XML Bible</title>
        <author>Elliotte Rusty Harold</author>
      </book>
      <book publisher="Addison-Wesley">
        <title>The Mythical Man Month</title>
        <author>Frederick Brooks</author>
        <isbn>0-373-70708-8</isbn>
      </book>
      <book publisher="WROX">
        <title>Professional XSLT 2nd Edition</title>
        <author>Michael Kay</author>
        <extensions>
         <price xmlns="http://www.example.com/book/extensions">24.99</price>
        </extensions>
      </book>
    </books>
    

    The schema for the http://www.example.com/books/ namespace describes the books element, which can contain one or more book elements that have required author and title elements, an optional isbn element, as well as a publisher attribute. Each book element also has an extensibility point within which a delimiter element followed by zero or more elements from the target nampespace can occur multiple times. The end of the extensibility point is bounded by an end element. The content model of the book element also allows for zero or more elements from any namespace besides the target namespace of the schema to appear after the end element. The schema for the http://www.example.com/books-core namespace contains a mustUnderstand attribute that must be added on extension elements or elements from a future version of the format that a consumer must know how to process. The delimiter and end elements are also defined in this schema.

    In the next version of the format, it is decided to add an additional edition-number element to content model of the book element. Below is the schema for version 2 of the format, along with a sample document

    BOOKS.XSD
     <xs:schema blockDefault="#all"  elementFormDefault="qualified" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema"
          targetNamespace="http://www.example.com/books"
          xmlns:b="http://www.example.com/books"
          xmlns:bc="http://www.example.com/books-core">
    
      <xs:import namespace="http://www.example.com/books-core" 
          schemaLocation="books-core.xsd" />
    
      <xs:element name="books">
        <xs:complexType>
          <xs:sequence>
       <xs:element name="book" type="b:bookType" 
              maxOccurs="unbounded" />
          </xs:sequence>
          <xs:attribute name="version" type="xs:string" />
        </xs:complexType>
      </xs:element>
    
      <xs:complexType name="bookType">
        <xs:sequence>
          <xs:element name="title" type="xs:string" />
          <xs:element name="author" type="xs:string" />
          <xs:element name="isbn" type="xs:string"  minOccurs="0" />
          <xs:sequence minOccurs="0" maxOccurs="1">
            <xs:element ref="bc:delimiter" /> 
            <xs:element name="edition-number" type="xs:positiveInteger"  minOccurs="0" /> 
       <xs:sequence minOccurs="0" maxOccurs="unbounded"> 
         <xs:element ref="bc:delimiter" /> 
         <xs:any namespace="##targetNamespace ##local" minOccurs="0"maxOccurs="unbounded"/> 
       </xs:sequence>
       <xs:element ref="bc:end" />
          </xs:sequence>
          <xs:any namespace="##other" minOccurs="0" maxOccurs="unbounded"  
              processContents="lax" />
        </xs:sequence>
          <xs:attribute name="publisher" type="xs:string" />
       </xs:complexType>
    
     </xs:schema>
    BOOKS.XML
    <books version="2.0" xmlns="http://www.example.com/books"
                         xmlns:p="http://www.example.com/book/extensions" 
                      xmlns:bc="http://www.example.com/book-core">
      <book publisher="HCI">
        <title>A Child Called It</title>
        <author>Dave Pelzer</author>
        <isbn>1-55874-766-9</isbn>
        <bc:delimiter />
        <edition-number>1<edition-number>
        <bc:end />
        <extensions>
         <p:price>9.95</p:price>
        </extensions>
      </book>
    </books>
    

    Unlike the New Constructs in a New Namespace approach, this approach keeps all the core components of the format in a single namespace and is forward compatible as well as backward compatible. It should be noted that forward compatibility is dependent on not adding any required constructs to the format that were not required in the previous version. Another benefit of this approach is that it obviates the need for having an explicit mustUnderstand construct because one can simply specify that if a consumer encounters any element from the target namespace of the format that is not understood, then it should result in a fatal error.

    The primary drawback of using version extensibility points is that it makes both schemas and XML instances more verbose, and thus potentially more confusing. A simplification of this approach is to use nested elements instead of delimiters. This approach is advocated by David Orchard in his article on Versioning XML Vocabularies.

Acknowledgements

I would like to thank Denise Draper, Martin Gudgin, Doug Purdy, Sowmy Srinivasan, Chris Lovett, and David Orchard for their feedback and comments during the process of writing this article.

Dare Obasanjo is a member of Microsoft's WebData team, which among other things develops the components within the System.Xml and System.Data namespace of the .NET Framework, Microsoft XML Core Services (MSXML), and Microsoft Data Access Components (MDAC).

Feel free to post any questions or comments about this article on the Extreme XML message board on GotDotNet.