Web Services, Opaque Data, and the Attachments Problem

Article
04/03/2007

Matt Powell
Microsoft Corporation

June 2004

Summary: Explore the many possibilities for dealing with opaque data in Web services (including SwA, WS-Attachments and DIME, PASwA, MTOM), and see the problem modeled with a classic childhood toy. (13 printed pages)

Applies to:

   Web services
   Web services protocols and specifications
   LEGO® DUPLO™ building bricks

Introduction
Solving the Opaque Data Problem in the LEGO World
So What Approach Should You Take?
What Do I Use Today?
What about Binary XML?
Conclusion

Introduction

Despite the flexibility and global acceptance of XML, there are times when serializing data into XML does not make sense. Media data and data that includes digital signatures are two common examples. In the case of media data, there are already highly standardized formats where compression plays a major part. For digital signatures, the binary integrity required would not be preserved if it were serialized into a structured XML format. This sort of arbitrary binary data is often called opaque data and presents a challenge in the realm of Web services. A standardized SOAP-based infrastructure has been developed in the Web services world, and sending opaque data within a SOAP envelope is not conducive to working nicely with that infrastructure. Several solutions have been proposed, including SOAP Messages with Attachments (SwA), WS-Attachments using DIME, the Proposed Infoset Addendum to SOAP Messages with Attachments (PASwA) (which has largely been incorporated into the SOAP Message Transmission Optimization Mechanism (MTOM) from the W3C), base-64-encoding the data within the XML, and there has even been investigation into binary XML formats. I will look at the different approaches, discuss the benefits and problems with each, and show that base 64 encoding is a viable solution to the opaque data problem today. But for more efficient transfer of data, MTOM is a solution for the future that will also work well with the Web services infrastructure.

Solving the Opaque Data Problem in the LEGO World

My children are currently in a phase where they love to play with LEGO® DUPLO™ building bricks. These bricks are really amazing things and you can build almost anything out of them. One of the more popular pieces is a large brick with four wheels on the bottom of it. The beauty of the wheeled brick is that no matter what gets built with the LEGO pieces, we can just snap it on top of the wheeled LEGO DUPLO brick and drive the creation from the LEGO city built by one child to the LEGO city built by another child. This enhances the playing possibilities significantly.

Figure 1. The beautifully extensible, composable, and interoperable wheeled LEGO DUPLO brick

But invariably during the course of a multi-hour construction-fest, some other toy comes into play. Usually it is an action figure that wants to join in the ruckus and take its turn on the mobile LEGO brick. The problem is that this toy is not a LEGO brick. When you try to drive it around the room, it slides off the wheeled brick. Often it cannot be balanced on the brick at all. The wheeled brick, which seemed like the epitome of versatility, is now at a loss.

So what do I do? Do I take advantage of the amazing abilities of LEGO bricks and create something using them that looks like the original action figure? Do I abandon the wheeled brick and replace it with the generically concave back of a toy truck big enough to carry LEGO brick creations, action figures, and anything else that fits within certain size restrictions? Or do I get out my knife and start carving round holes in the feet of the action figure so that the holes will fit the posts in the wheeled brick precisely?

This is very similar to the opaque data problem with Web services.

Web services are great for passing around infinitely versatile XML. You can represent everything in XML, and the fact that XML is so standardized means it can ride on the wheeled LEGO brick of the XML world, the SOAP envelope, and countless hours of fun can be had by all.

Enter the action figure.

Opaque data is the action figure of XML Web services. So what are your options for transporting opaque data with a SOAP message?

One approach would be to follow the option of building an action figure out of LEGO bricks. You could replace the standardized formats for multimedia files and the like with a new XML format and cross your fingers and hope it catches on. The problem is that it won't catch on. The formats for GIF, JPEG, and MP3 files are well established in the world, and there just isn't a compelling enough argument for coming up with a new standard. There are other kinds of data where transitioning to an XML format makes a lot of sense, but multimedia files are certainly one example where it does not.

Another approach might be similar to scrapping the wheeled brick and using the unrestrictive bed of a toy truck. The toy truck approach is similar to the SOAP with Attachments and DIME/WS-Attachments approaches to transmitting binary data with SOAP messages. Both of these attachment solutions are flexible in the sense that you can throw anything in a MIME or DIME message, including a SOAP envelope, but something is lost. The wheeled LEGO DUPLO brick was cool because not only could I snap my creation onto it, but I could snap other bricks onto it as well. If I use a toy truck, do I just throw extra bricks in the back of the truck or do I snap them onto my creation? When mobilizing LEGO brick creations with the wheeled brick from one child's LEGO city to another's, I knew that the creation would snap onto its destination building perfectly since it already lived happily on the wheeled DUPLO brick. The toy truck does nothing to insure that its cargo will mesh with the destination LEGO city. On top of it all, the entire LEGO world built by the children was created with roads that fit the size of the wheeled brick and docking garages where the wheeled brick could precisely load and unload its cargo. As soon as the truck comes into play, a rampant path of destruction is created as it blasts down streets too narrow for its wheels. I could build an infrastructure for the truck, but it would be a different infrastructure, and all the cities already had loading docks for wheeled LEGO DUPLO bricks.

Similarly, attachment solutions to the binary data problem lose something in the process. Is metadata that should accompany the SOAP envelope sent in the SOAP envelope or in the attachment mechanism? Is the data attached to the SOAP message going to work in the SOAP world? But the most important problem is that the critical mass required for the success of SOAP and its groundswell of support and the creation of the entire SOAP infrastructure are based off the principles of SOAP envelopes. Supporting both attachment streams and SOAP message streams requires a complete duplication of effort for what has already been achieved with SOAP. Plus, the fact that SOAP is successfully supported on so many platforms and toolsets cannot be considered a given for other binary-friendly protocols. Interoperability is a long, hard battle that SOAP has already won.

So is carving holes to fit LEGO posts the correct solution? This approach is similar to base-64-encoding your data and placing it within your XML. The data is still fundamentally not a LEGO brick, but it has been made to look like a LEGO brick. Base 64 encoding does increase the size of the data by a third, but this is probably significantly better than if you tried to parallel the opaque data's structure in XML. The good news is that the infrastructure still loves this approach. You are dealing with XML, and all the work done for doing things, such as digitally signing XML, still work for base-64-encoded opaque data.

In the end, I would like to be able to have my data fit nicely into the XML world, but I don't want to limit myself to the wheeled LEGO DUPLO brick. In other words, there are advantages to having a toy truck move things around (it is bigger, faster, and is more efficient than a wheeled brick), but the key problem is working within the infrastructure that already exists for the wheeled brick world.

In my case, SOAP messages are my wheeled bricks. I want to maintain all the benefits of the current infrastructure without tearing it apart or trying to replace it. However, I would like some benefits, such as sending opaque data in some way. Therefore, I simply create a transport scenario that looks like SOAP to all the interfaces involved, but can then bring in more efficient means for doing the actual carrying of the data than the base-64-encoding mechanism. This would be the equivalent of throwing the wheeled LEGO DUPLO brick and its contents into the toy truck for the actual transmission so I can take advantage of its size and speed, and then reassembling the wheeled brick and its cargo when it came time to plug into the LEGO infrastructure at the next LEGO city. This is the SOAP Message Transmission and Optimization Mechanism (MTOM) approach that, as of the writing of this article, recently went to Last Call status at the World Wide Web Consortium (W3C).

So What Approach Should You Take?

LEGO DUPLO bricks, toy trucks, and action figures aside, I have basically outlined five options for transmitting opaque data. Those options are:

XML Representation: Replace any non-Xml data with a structured XML representation and include it in your message.
SOAP with Attachments (SwA): Use SOAP with Attachments to send binary data with your SOAP in a MIME Multipart message.
WS-Attachments with DIME: Use DIME and WS-Attachments to send binary data with your SOAP in a DIME message format.
Base 64 Encoding: Use base 64 encoding to include opaque data within your SOAP message
Message Transmission and Optimization Mechanism (MTOM): Use an MTOM approach to take advantage of the SOAP infrastructure, but gain the transport efficiencies provided by a SOAP with Attachments solution.

The reality is that although the MTOM approach offers some intriguing capabilities for the future in the binary data/attachments area, there may be factors where other approaches may make sense to use. I will take a look at each of these approaches, the benefits of each, the problems that each may have, and when you should use each approach.

XML Representation

Ultimately SOAP messages are made of XML. Therefore, any data that can be effectively represented with XML should be. The key term is effectively. For instance, you could represent a bitmap like this:

<picture>
  <row>
    <pixel>
       <color>00FF00</color>
    </pixel>
    <pixel>
       <color>00FF00</color>
          ...

Of course, the problem here is that it bloats the data tremendously and there are already standardized data formats that are well accepted. However, if you are sending something like Microsoft Word documents, you might want to consider saving them in XML format.

This article is generally focused on the situations where structured XML representation is not a viable solution, so I will not dwell on this approach.

Interoperability: Perfect

Composability: Excellent. The family of WS-* specifications that Microsoft has developed with its partners compose seamlessly with the SOAP message approach to sending data.

Efficiency: Debatable. A lot is made of how inefficiently something like an array of integers is represented in XML. Sure, this is true, but the same could be said for HTML and I don't see anybody clamoring for the removal of HTML tags because it is causing the world's networks to come to a screaming halt. I know there are differences between XML and HTML, but if the back end of your Web service is making a network request to a SQL server or is calling another Web service, then the parsing overhead is minimal. XML is not the most efficient mechanism for holding data, but it is so popular because of its flexibility and interoperability.

When to use: Whenever you can.

SOAP with Attachments (SwA)

SOAP with Attachments was the first attempt by Microsoft (along with Hewlett Packard) at solving the opaque data/attachments problem.

Interoperability: Potentially decent in the future with some notable exceptions. SOAP with Attachments is simply a W3C Note; it is not a W3C Recommendation or even a W3C Working Draft. The Web Services Interoperability Organization (WS-I) is working on an attachments profile based on SOAP with Attachments. Presumably, after the profile is final and conforming implementations are available, they should be interoperable. The problem is that when it comes to the attachments profile, WS-I is ignoring the fact that SwA breaks the Web services model. Among other issues, SwA specifically does not work with WS-Security at this time. Because of this fundamental problem with SwA, the W3C is forgetting about SwA and going in the direction of MTOM. Similarly, Microsoft is not planning on supporting SwA on any of its platforms. Can the fundamental problem with SwA be fixed? Yes! That is exactly what MTOM is—a tweaking of SwA to work within the composability of advanced Web services.

Composability: Poor. Without an encompassing SOAP envelope to build upon, SOAP with Attachment messages do not compose well with the WS-* specifications. In particular, a non-SOAP-message-based transport cannot support the recently finalized OASIS WS-Security specification.

Efficiency: Good. Although not as efficient and streamlined as the DIME and WS-Attachments approach, opaque data can be sent in its raw form without base 64 encoding. There are issues with buffering incoming data due to the lack of a length header on the message sections, but it is a viable solution to the problems around sending potentially large, opaque blocks of data. That is why the MIME approach that SwA uses was deemed as an acceptable basis on which to build MTOM.

When to use: If you have interoperable SOAP with Attachments implementations on all the platforms that you might want your Web service to be accessible from, then you might consider using SwA. Since there is no standard specification or even a final WS-I profile yet (as of this writing), you should test thoroughly to make sure implementations are interoperable. Also note that since Microsoft is not planning on implementing SwA at this time, you would only want to use SwA if knew you never wanted to interoperate with Microsoft platforms. If you find you are really only communicating between computers running the same platform, then the reason for using Web services (interoperability) is moot, and you might as well use a proprietary communication mechanism. Finally, you will specifically want to avoid SwA if you need OASIS-compliant WS-Security type signatures or encryption on your opaque data.

WS-Attachments using DIME

DIME and WS-Attachments is a faster and more efficient solution to the attachments approach to opaque data compared to SOAP with Attachments, but has the same composability problems as SwA.

Interoperability: Modest. Although DIME is a simpler protocol than MIME that would bode well for its interoperable possibilities, efforts for creating widely interoperable DIME and WS-Attachment implementations have stopped. As with SwA, there is no recommendation for DIME or WS-Attachments at this time. There are also no plans for standardizing DIME and WS-Attachments. Microsoft tools currently only support a DIME approach to attachments, but will be changing to support MTOM in the future. There is at least one documented scenario where DIME implementations interoperate between platforms.

Composability: Poor. As is the case with SwA, a non-SOAP envelope approach to messaging like DIME and WS-Attachments limits the ability of the higher-level Web service specifications to properly work upon a message.

Efficiency: Very Good. DIME is a simple binary format for transmitting raw data and has support for such efficiencies as chunking and jumping easily between message records.

When to use: Web services are all about interoperability, and if DIME and WS-Attachments does not provide wide interoperability, then there is little reason to use it. If your are not interested in interoperability but believe DIME is attractive for its performance reasons, then you would probably be better off using other proprietary, high-performance solutions such as .NET Framework Enterprise Services. DIME and WS-Attachments also suffer from the same limitations as SOAP with Attachments, in that it is only viable if you do not require higher-level capabilities for your Web services. At this point the only real reason to be using DIME and WS-Attachments would be to interoperate with an existing DIME and WS-Attachment implementation.

Base 64 Encoding

Base 64 encoding has been around for some time and has probably received a bad rap for the opaque data problem. Base 64 encoding has some extremely nice benefits, such as its seamless integration in SOAP envelopes, it is a well-known standard, and it has better size efficiency than structured XML.

Interoperability: Excellent. Base-64-encoded data is understood on every platform that understands XML. Whether the opaque data included is in an interoperable form may be another question that is not already positively answered, like it is with the XML Representation approach. However, if you are sending opaque data and you are doing so with a Web service, then the interoperability issue has probably already been positively addressed.

Composability: Excellent. The fact that the data lives within the SOAP envelope means that all the WS-* specifications that Microsoft has developed with its partners will work seamlessly with a base-64-encoded XML element.

Efficiency: Modest. Although base-64-encoded data is usually more efficient from a message-size perspective than structured XML, as mentioned earlier, it still bloats the data by about 33% even when only single byte characters are used. The character set for base-64-encoded data does not require multi-byte characters, so if you can use UTF-8 encoding you can avoid the extra 100% increase in size. Of course you cannot mix single and multi-byte encodings within a single XML message, so if the rest of your SOAP envelope requires multi-byte encoding, then you have to live with the two-fold increase.

When to use: Base 64 encoding is probably the best way to pass opaque data if transport size efficiency is not your first concern. It will work seamlessly with higher-level WS-* protocols and is smaller than a standard XML encoding.

Message Transmission Optimization Mechanism (MTOM)

MTOM combines the composability of Base 64 encoding with the transport efficiency of SOAP with Attachments. The opaque data is inherently dealt with just like it is with SwA—it is simply streamed as binary data as one of the MIME message parts. However, if at the Infoset level, immediately before message serialization, the opaque data is accessed, it is temporarily treated as base-64-encoded text. For instance, a WS-Security layer creating a digital signature would stream the binary data through a base-64-encoding layer as it calculated the signature. The base-64-encoded data is not actually transferred, nor would it need to be held in memory, or decoded back into the binary. The process for creating an MTOM message is illustrated in Figure 1. On deserialization, the same concepts apply. The raw binary is transferred directly to the higher layer, but any processing layers that access it would do so through a temporary base-64-encoding layer.

Figure 2. The MTOM Processing Model

Interoperability: Potentially great. As of this writing, MTOM is at Last Call in the W3C publishing pipeline. What this means is that it is on track to reach final recommendation, probably in the third quarter of 2004. If you compare this with the WS-I attachments profile that is standardizing SOAP with Attachments, both these standards should finalize in roughly the same timeframe. You can expect public implementations of MTOM to be available shortly thereafter. So the timeframe to interoperability should be at least as good as SOAP with Attachments interoperability.

Composability: Excellent. The end result of an MTOM transfer is a SOAP envelope, so all higher-level Web service protocols work as designed.

Efficiency: Excellent. Although the MIME Multipart data transfer isn't quite as efficient at transferring raw data as DIME, it can still transfer data without additional bloating.

When to Use: You can probably expect to see MTOM implementations created by the various platform and tool providers in the third quarter of 2004. If Base 64 encoding is not a viable solution and you need an answer today, you are probably better off going with a DIME and WS-Attachments solution (as is supported today by WSE), or potentially an SwA implementation if you are running on another platform. Just be aware that you will probably need to rebuild with the latest technology at some point in the future.

What Do I Use Today?

Base-64-encoding data within an XML message is a very viable solution for including opaque data in a SOAP message. It works with today's tools and is at least as efficient as sending structured XML data. Base 64 encoding will also continue to work in the future and will work seamlessly with all the advanced Web services specifications.

Sending base-64-encoded data in a SOAP message is relatively easy with the .NET Framework. Byte arrays are serialized into base-64-encoded elements, so all I have to do is copy my opaque data into a byte array parameter for my ASP.NET WebMethod. The following code returns a base-64-encoded image from a WebMethod called GetPicture.

[WebMethod]
public byte [] GetPicture()
{
    FileInfo inf = new FileInfo(IMAGE_FILE_PATH);
    FileStream fs = new FileStream(IMAGE_FILE_PATH, 
                                   FileMode.Open);
    byte [] ret = new byte [inf.Length];
    fs.Read(ret, 0, (int)inf.Length);
    fs.Close();
    return ret;
}

Consuming this Web service from a managed client is also quite easy. The following code creates a Bitmap object from the binary data and displays it in a Picture Box control.

private void button1_Click(object sender, 
                           System.EventArgs e)
{
    localhost.Service1 proxy = new localhost.Service1();
    byte [] byteImage = proxy.GetPicture();
    this.pictureBox1.Image 
        = new Bitmap(new MemoryStream(byteImage));
}

I was able to transfer a 17MB image file in this way without hitting any buffer limitations.

But what do you do if you cannot afford the bloat of base 64 encoding? Until MTOM implementations are available, your best bet is to use one of the non-bloating attachment mechanisms that are currently available (such as the DIME and WS-Attachments support in WSE). You should realize that you probably will need to port your solution to a non-attachments approach at some point in the future. You also need to realize that if you use SwA or WS-Attachments that you lose the support for higher-level protocols like WS-Security.

What about Binary XML?

Since its inception, XML has suffered under the attacks of its detractors over the amount of bloating that XML adds to the data that it holds. It has long been proposed that a standardized binary representation of XML data be created that would avoid things such as repetitive element tags and unnecessary character encoding. But the problem of optimizing XML representations brings up the question of what exactly you are optimizing for. Do you optimize for size, or do you optimize for processing speed? Consider just the question of using Little Endian or Big Endian byte order? Big Endian byte order is considered the "network representation" but if you optimize for processing speed, would it not make sense to use the representation native to the vast majority of machines that are networked today? In the end, the complexity of the issue could easily mandate that there be several binary representations of XML. Now you are really adding complexity to an issue that negates the interoperable benefits of having a text-based encoding mechanism like XML. Expecting a groundswell of industry-wide support for several non-intuitive data formats is probably not realistic.

Standardizing on compressed SOAP using well-known compression algorithms like GZIP has also been proposed. GZIP will do a good job at compressing a SOAP message (not as good if most of your message is base-64-encoded), but then you incur the significant overhead of compressing and decompressing the data. It is a rather frustrating coincidence that probably the biggest beneficiaries of having small messages—mobile devices—are also one of the biggest sufferers of having high processing overhead.

It is worth noting that in September 2003, the W3C held a workshop on Binary XML. The committee concluded that further investigation was necessary, and that it was not clear whether it was worth trying to establish a standardized format or process for binary XML. A working group has now been created, but its charter is strictly to come up with use cases for a binary format, determine if one or more binary formats can solve the problems defined in all the use cases, and if so, determine if the efficiency would outweigh the potential loss of interoperability. There is obviously a lot of work to be done even before work on an actual specification would be started. And it is very realistic that such a specification may not even be considered.

Conclusion

The opaque data problem with Web services is a problem that has had a number of proposed solutions. These solutions have suffered from either a lack of industry acceptance or a lack of support for the composability of Web service protocols that provide the rich set of functionality that makes Web services so appealing. One solution to the opaque data problem is base 64 encoding data within the SOAP message, but this approach suffers because of its inefficient bloating of data. The SOAP Message Transmission Optimization Mechanism (MTOM) appears to be a great solution that treats the opaque data as part of the SOAP envelope at the Infoset level, but serializes the data for efficient transport over MIME messages. MTOM works in the highly composable world of Web service protocols and allows for the efficient transfer of opaque data.

Microsoft .Net and J2EE Interoperability Toolkit

.NET Web Services: Architecture and Implementation

Understanding Web Services Specifications and the WSE

Web Services Enhancements: Understanding the WSE for .NET Enterprise Applications

Web Services, Opaque Data, and the Attachments Problem

Contents

Introduction

Solving the Opaque Data Problem in the LEGO World

So What Approach Should You Take?

XML Representation

Interoperability: Perfect

SOAP with Attachments (SwA)

WS-Attachments using DIME

Base 64 Encoding

Message Transmission Optimization Mechanism (MTOM)

What Do I Use Today?

What about Binary XML?

Conclusion

Additional resources

Web Services, Opaque Data, and the Attachments Problem

Contents

Introduction

Solving the Opaque Data Problem in the LEGO World

So What Approach Should You Take?

XML Representation

Interoperability: Perfect

SOAP with Attachments (SwA)

WS-Attachments using DIME

Base 64 Encoding

Message Transmission Optimization Mechanism (MTOM)

What Do I Use Today?

What about Binary XML?

Conclusion

Related Books

Additional resources