Mobile Ink Jots 3: Ink on the Web
Shawn A. Van Ness
Leszynski Group
August, 2004
Applies to:
Microsoft® Windows® XP Tablet PC Edition 2005
Summary: The upcoming release of the Tablet PC platform and SDK, codenamed "Lonestar", offers long-awaited support for Ink-enabling Web applications. This month, Shawn tells us what that means, and shows us how it's done.
Contents
Introduction
What is a Web Application?
Hosting Ink-enabled Controls in Internet Explorer
Posting Back to the Web Server
Waste Not, Want Not
Deploying a Rich-Client Tablet PC Application over HTTP
Options for Supporting Non-Tablet Clients
Conclusion
Introduction
Welcome back to Mobile Ink Jots. It's July, and the countdown to the release of Windows XP Service Pack 2 (SP2) is well underway. If you read my previous column or have been keeping up with the latest Tablet PC developments in any way, you no doubt already know that Windows XP SP2 will encompass the "Lonestar" release of the Tablet PC Platform—a release officially dubbed Windows XP Tablet PC Edition 2005.
Last month, we took a tour of Lonestar, covering the new features from a developer's perspective. We covered a broad range of topics, from enhancements to the handwriting recognition experience to the new RealTimeStylus API, but only superficially. Starting this month, we'll cover each of those topics in more detail, one by one.
Of all the new features Lonestar has in store, the one most eagerly anticipated by application developers is the newly added support for Ink on the Web. So, let's begin by taking an in-depth look at what it takes and, indeed, what it means to run Ink applications in various Web-based application scenarios.
What is a Web Application?
And what does it mean to Ink-enable one? These are existential questions, but here in the year 2004 the Web is still a very messy place, technologically speaking. We need to be very careful and very specific when we talk about this kind of stuff. The phrase "Web application" means very different things to different people.
For starters, we're not talking about ASP.NET versus PHP versus JavaServer Pages (JSP) or what have you. Well, not necessarily—any of those server-side Web technologies can be used to provide rich, Ink-enabled content to Tablet PC clients (more on that, later). Rather, the Web technology that we need to concern ourselves with is almost entirely client-tier: how to provide a rich user experience that transcends the built-in capabilities of Web browsers, allowing the collection and rendering of Ink. For example, how do you load (and script) an Ink-enabled control embedded in a Web page?
Of course, these days the Web is no longer married to the browser. It's not uncommon to see full-fledged, rich client applications communicating with Web Services, where no HTML is ever displayed. However, that sort of Web-enabled client application is not a very useful definition for "Web application". Many Tablet PC applications are already designed that way. For the purposes of this discussion, I'll define a Web application as any program or component code which is, itself, deployed over HTTP and thereby made seamlessly available to clients accessing a Web server, without the need to install, configure, or maintain anything.
Java applets and Microsoft ActiveX® controls have formed the basis of rich client-tier Web application technology, for most of the past decade. But in the face of the .NET Framework, ActiveX controls are a largely deprecated technology (and in the space of Tablet PC development, Java is a non-starter). That's fine. The .NET Framework offers a much better alternative to both: we can use C# or Microsoft Visual Basic® .NET, and the .NET Framework's Windows Forms classes to implement rich client-tier functionality in our Web applications. This is true whether we need to add a custom control in an Internet Explorer Web page, or develop a stand-alone executable with its own frame window (or perhaps some combination of the two). The same .NET Framework technologies—Windows Forms, code access security, and now Microsoft.Ink—apply to both solutions.
The .NET Framework has supported the deployment and distribution of code over HTTP since version 1.0. Windows Forms executables can simply be deployed on a Web server and run over the URL—a practice known as no-touch deployment—and Internet Explorer version 5.01 introduced a new syntax for the <object> tag, which allows the loading of managed Windows Forms controls in lieu of ActiveX controls. However, support for running Ink in these "mobile code" scenarios is new to the Lonestar release of the Tablet PC.
In either case, whenever the .NET Framework runtime executes mobile code (any code obtained from a remote source—typically a Web server, but maybe an FTP site or a mapped network drive) the code is constrained by what's known, somewhat whimsically, as a sandbox. Java pioneered the sandbox security model, making it safe for users around the world to interact with Java applets on arbitrary Web sites, without much worry that the applet might reformat our hard drives, steal our personal information, or otherwise misbehave.
The .NET Framework improves on the Java sandbox security model by automatically mapping a set of permissions based on evidence, the evidence effectively being trustworthy information about where the code came from and/or who wrote it. This means that by default controls and programs downloaded from the Internet at large run with a more restricted set of permissions (think of it as a smaller sandbox) than controls and programs downloaded from the local Intranet. At one logical extreme, code run from the computer's local hard drive gets the full set of permissions (either no sandbox at all, or an infinitely large sandbox, depending on your point of view). At the other logical extreme, code run from a site listed in Internet Explorer's Restricted Sites zone gets no permission to run at all.
We'll discuss code access security in greater detail, as we go along. First, let's jump right in and learn how to host custom, Ink-enabled controls in Internet Explorer, which no doubt still hosts the vast majority of thin-client Web applications out there.
Hosting Ink-enabled Controls in Internet Explorer
Jay Allen's article, Host Secure, Lightweight Client-Side Controls in Microsoft Internet Explorer, in the January 2002 MSDN Magazine, provides a very good background on the subject of hosting managed Windows Forms controls in Internet Explorer. The sample in the article uses an overload of the ActiveX-era <object> tag. For the uninitiated, here's a simple example of what the requisite HTML code looks like:
<object classid="MyControls.dll#MyControls.UserControl1"
width="300" height="200" id="uc1">
<param name="BackColor" value="#808080" />
<param name="Text" value="You get the idea..." />
Sorry, your browser doesn't seem to support .NET controls.
</object>
Compared to hosting ActiveX controls, there are really just a few important things to note about this new HTML syntax. As you can see, the classid attribute has been overloaded, to specify the location of the control's assembly and its full managed type name, concatenated together with a '#' character. The assembly's location is specified as a URL, relative to the page's base URL, just like any ordinary hyperlink or link to an external image, style sheet or script file.
The official story is that Internet Explorer version 5.01 or later is required in order to load managed controls when using this new syntax; however, I recommend later versions. Internet Explorer versions 5.5 and 6.0 provide several significant bug fixes in this area, some especially relevant to interaction with components of the Microsoft.Ink namespace.
One important thing to note about this new <object> tag syntax is that it's not possible to directly load a managed control from the global assembly cache. The control's assembly must be accessible through the specified URL. For example, it's not possible to load a Windows Forms PictureBox control directly into a Web page:
<!-- THIS DOESN'T WORK -->
<object
classid="System.Windows.Forms.dll#System.Windows.Forms.PictureBox"
width="300" height="200" id="pb1">
</object>
It is, however, entirely possible to derive your own control from the Windows Forms' PictureBox class and host that control in the Web browser. Your assembly's dependency on System.Windows.Forms.dll will be resolved over the default probing paths of the .NET Framework, which of course includes the global assembly cache.
So, what of Ink support? For the same reason we can't directly load an instance of a Windows Forms PictureBox, we can't directly instantiate an InkPicture or InkEdit control from the Microsoft.Ink assembly residing in our clients' global assembly cache. We must derive our own controls, and that's fine because we really don't want to limit ourselves to InkPicture and InkEdit, anyway. In the general case, we'll almost certainly want to write our own user interface controls, attaching InkCollector or InkOverlay objects and performing recognition exactly how and where we please.
The following sample code demonstrates a simple but complete UserControl-derived Windows Forms class, with an attached InkCollector object.
namespace MyControls
{
public class InkableUserControl : System.Windows.Forms.UserControl
{
private Microsoft.Ink.InkCollector inkCollector;
public InkableUserControl()
{
// This call is required by the Windows.Forms Form Designer.
InitializeComponent();
// Further initialization
this.BackColor = Color.FromKnownColor(KnownColor.Window);
this.inkCollector = new InkCollector(this); // secure!
this.inkCollector.Enabled = true;
}
}
}
What's new here for Lonestar? The first thing you might notice, especially if you don't have version 1.7 of the Tablet PC Platform SDK installed, is the new constructor overload for the InkCollector object, which takes a reference to a managed Windows Forms Control instance instead of a raw Win32 window handle. Subtle, eh? (Without the Lonestar SDK, the above code produces a much less subtle compiler error.) This new constructor overload is required for InkCollector to interoperate with Windows Forms objects in a secure, tightly constrained sandbox suitable for Internet applications.
If you use the old-style constructor (for example, replace new InkCollector(this) with new InkCollector(this.Handle) in the previous code) the control will require permissions to touch any and all windows on the system and to execute unmanaged code. That's a much, much bigger sandbox, and it's one not really suitable for running mobile code at all. Indeed, attempts to call the old-style constructor will throw a SecurityException in both Intranet and Internet zones, by default.
The second new thing, which you can't see from the above code, is that version 1.7 of the Microsoft.Ink assembly now bears the AllowPartiallyTrustedCallersAttribute. This all-important little attribute with the big long name (it's sometimes called APTCA for short) is like a safety switch. The attribute is an opt-in flag to declare that an assembly has undergone some measure of security review and has been deemed safe for use in a partially trusted context. This means that the assembly is safe for your applications to play with in a sandbox. If your assembly is strong-named, you'll need to add this attribute to your own code, too (after giving due consideration to any potential security problems, of course) in order to use it from a Web page.
Note The topic Security and Trust in the Lonestar SDK documentation describes the various API elements of the Tablet PC SDK that require specific permissions. Indeed, the entire section, Using Ink on the Web, is new, and the samples in the SDK now include a Web-enabled edition of the AutoClaims sample and even the beginnings of an Ink-enabled blog engine.
Posting Back to the Web Server
A few new constructors, and a new attribute—is that all there is to it? Not quite. Unless your application's only requirement is for Tablet PC users to be able to scribble transient little notes to themselves as they view your Web site, collecting Ink is only half the story. You'll probably want to actually do something with the Ink, like post it back to a Web service or database.
There are two options here. The first option is scripting: we can call most any public property or method on our managed control, by using client-side <script>. The return values can be used to populate hidden fields in post-backs to the Web server or perform any other DHTML tricks you can imagine.
There is a simpler, less well-documented option. If your control's <object> tag resides inside an HTML <form> block and you wish to post a simple value back to the Web server (for example, returning the Ink as base64 string), the easiest thing to do is expose the Ink as the default value-property. There is no well-defined interface for this. Internet Explorer simply reflects against the control's public API for a property named Value and includes the result in the form's post-back collection of name/value argument pairs. The following pair of HTML and C# code samples demonstrates a little of both of these techniques: a scriptable method to perform handwriting recognition, and a default value-property to return the base64 Ink:
<form action="/inktest/backend.asmx" method="post">
<p>Please sign in the box, below:</p>
<object id="ink1" classid="MyControls.dll#MyControls.InkableUserControl">
<param name="BackColor" value="#E0E090">
</object>
<a href="#" onclick="setRecoText()">Recognized text:</a>
<input type="text" name="text1" size="20" />
<input type="submit" name="submit" value="Submit!" />
</form>
<script language="jscript">
function setRecoText() {
var recotext = document.forms[0].ink1.Recognize();
document.forms[0].text1.value = recotext;
}
</script>
The HTML code is an expansion of the fragment we've already seen. Our control's <object> tag is now contained within a larger <form> element, alongside some other controls, and we've written a few lines of script to bind them together with some rich behavior: clicking the hyperlink invokes the Recognize() method on our control, and populates the form's text box with the results.
namespace MyControls
{
public class InkableUserControl : System.Windows.Forms.UserControl
{
private Microsoft.Ink.InkCollector inkCollector;
public InkableUserControl()
{
this.BackColor = Color.FromKnownColor(KnownColor.Window);
this.Size = new System.Drawing.Size(300, 200);
this.inkCollector = new InkCollector(this);
this.inkCollector.Enabled = true;
}
// We invoke this method explicitly, through script,
// to recognize the user's writing.
public string Recognize()
{
return this.inkCollector.Ink.Strokes.ToString();
}
// This property is invoked by Reflection,
// when Internet Explorer posts our UserControl
// as part of a <form> block.
public string Value
{
get
{ return GetBase64Ink(); }
}
// Serialize the Ink as Base64 ISF, max compression.
private string GetBase64Ink()
{
byte[] inkbits = this.inkCollector.Ink.Save(
PersistenceFormat.Base64InkSerializedFormat,CompressionMode.Maximum);
return Convert.ToBase64String(inkbits);
}
}
}
This C# code is also similar to the Windows Forms control code we've already seen. The only new features are the script-callable Recognize() method and the Value property, which is implicitly invoked by Internet Explorer during the form's post.
Are we there yet? Almost. There's one more task, one we really mustn't neglect: resource management.
Waste Not, Want Not
Version 1.5 of the Tablet PC SDK added IDisposable implementations to many of the core Ink classes, notably InkCollector and InkOverlay. While many APIs in the .NET Framework class library allow us to ignore IDisposable without much problem (their finalizer methods being designed to perform the exact same cleanup) the classes in Microsoft.Ink do not do this for us. There is a perfectly good explanation for this, and if I knew what it was, I'd share it with you. Suffice to say, to fully deconstruct an instance of InkCollector or InkOverlay, two things must happen: we must call IDisposable.Dispose, and wait for the garbage collector to run.
Keep this advice in mind when developing any Tablet PC application. It applies to Windows Forms applications as well as Web applications. But in the latter case, where and how can we call Dispose()? We'll need to write a few more lines of script in our HTML, to explicitly destroy the Ink control when the page is unloaded:
<body onunload="onUnload()">
...
<script language="jscript">
function onUnload() {
// Must explicitly dispose Ink resources!
document.forms[0].ink1.Dispose();
}
</script>
And of course, back in our C# code, we must make sure our control's Dispose() override forwards to the InkCollector appropriately:
protected override void Dispose(bool @explicit)
{
if (@explicit)
{
// Fwd to IDisposable members.
this.inkCollector.Dispose();
this.inkCollector = null;
}
base.Dispose(@explicit);
}
And that's all there is to it.
Deploying a Rich-Client Tablet PC Application over HTTP
Internet Explorer has long been the mainstay of thin-client Web applications, but anyone who's used a Web interface to check their e-mail when away from home or office knows that browser-based interfaces have their limitations. They just don't come anywhere close to the usability and power of a rich-client application like Microsoft Outlook®.
And let's face it; developing browser-based applications is hard. Debugging client-side script can be very frustrating, and a large portion of many projects' development budgets is ill-spent on fighting CSS quirks or designing unique and dastardly ways to prevent the user from clicking the Back button. That's not adding value to anyone's Web experience. Certainly, both our bosses and our users would prefer that time was spent fixing real bugs and adding real features to our applications.
At long last, the .NET Framework offers us a better alternative in a rich, fully functional Windows Forms applications deployed over HTTP. As I mentioned earlier, this technology is known as no-touch deployment. Chris Sells' seminal article about no-touch deployment in the July 2002 issue of MSDN Magazine, Security and Versioning Models in the Windows Forms Engine Help You Create and Deploy Smart Clients, has been followed by a number of other good articles, including Launching No-Touch Deployment Applications with Command Line Arguments and Deploying and Updating Smart Client Applications. In addition, no-touch deployment is scheduled to receive improvements in upcoming releases of Windows and the .NET Framework, under the new moniker ClickOnce—gaining better, more flexible deployment options, including support for off-line operation.
There's not much to say about no-touch deployment. Getting started is easy: simply develop a Windows Forms executable and publish it on your Web server. When users follow a hyperlink to the .exe file, Internet Explorer downloads and launches the application, running it in the same secure sandbox—mentioned earlier—for controls embedded in the Web page. The .NET Framework runtime even displays a balloon tip on the application's frame window, informing users that the program is running with restricted functionality.
The devil, as always, is in the details. Configuring your Web server to hand out .config files and other dependencies can sometimes be tricky, and the intersection of the .NET Framework's versioning semantics and HTTP's If-Modified-Since header semantics can be downright bewildering at times. Chris Sell's article does a pretty good job of walking you through all these traps. Until ClickOnce arrives, I'd say that Chris' article is mandatory reading before you attempt any no-touch deployment in the real world.
In any case, we must take care to remember that the default code access security policy for applications run from the Internet zone is pretty strict: it's not too difficult to trip over a code access security hurdle when attempting to design a rich, full-featured Windows Forms application. Here's a partial list of some of the limitations you can expect to run into for .NET Framework version 1.1 (version 1.0 doesn't allow execution from the Internet Zone, at all):
- No unmanaged code or native window handling.
- No reflection, no serialization.
- No disk I/O (except for IsolatedStorage, and OpenFileDialog).
- No network I/O (except the ability to connect back to the originating server).
- No clipboard interoperability with other applications.
- Limited ADO.NET access.
- No XSL transforms.
By default, the sandbox for applications served from within the Intranet zone is only slightly larger:
- Unrestricted use of file-open and file-save dialogs.
- Unrestricted access to user interface features (but no native window handling).
- Added abilities to make DNS queries, learn the current username, post to the EventLog, and read from the local directory—all of questionable usefulness.
Make no mistake, these same default code access security policies apply to controls hosted within Internet Explorer Web pages. It's just that they can sometimes seem unreasonably constraining when developing a full-featured rich application, compared to developing a simple control to collect Ink on a Web page.
On the other hand, code access security is often a greater barrier for controls hosted in a Web page. It's not readily possible for the page's script code to sink events from managed controls, unless a high level of trust has been granted to the originating Web site. And Windows XP Service Pack 2 may disable the ability for script to sink events from managed code, altogether. (Sinking managed events with unmanaged script is tantamount to managed code calling into COM code, which implies the ability to execute unmanaged code—never mind the fact that the unmanaged script has likely already been executing in the browser all along!)
So, if you're careful to consider the code access security context in which your applications will run, and tolerant of the limitations code access security imposes, you'll have much more success providing a quality experience for Tablet PC users with a no-touch deployed smart client, than with a Web page. Put simply, HTML is not a good platform for new application development. Even the finest Web-based e-mail interfaces demonstrate that fact all too well.
Options for Supporting Non-Tablet Clients
Regardless of whether you're maintaining a legacy, browser-based Web application or developing a new, state-of-the-art smart client, you might find yourself needing to support both Tablet PC and down-level (non-Tablet PC) client platforms.
For browser-based applications, this typically means omitting the <object> tag from the HTML stream (perhaps providing some alternative UI elements, or a different page layout altogether). Smart-client applications have numerous options for degrading their functionality gracefully in the absence of Microsoft.Ink. The most obvious, though perhaps inelegant, is to build and deploy different binaries for Tablet PC and down-level clients, and redirect clients appropriately.
In either case, the same trick is required: the Web server needs to know whether or not the client is running Windows XP Tablet PC Edition 2005, in order to serve the most appropriate content, be it HTML or executable, managed code.
As I mentioned last month, Internet Explorer on Windows XP Tablet PC Edition 2005 will squeeze a few extra bits into the User-Agent string, allowing Web servers to do just that:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
.NET CLR 1.0.3705; .NET CLR 1.1.4322; Tablet PC 1.7)
It was (and still is) nigh impossible for Web servers to distinguish previous versions of Tablet PC clients from ordinary Windows XP clients and then respond with different content. Technically, this is not a problem, because hosting Ink in mobile-code applications was not officially supported at all, before Lonestar.
The ASP.NET code to detect the Tablet PC signature in the User-Agent HTTP header is trivially simple. But what you do, in response, is up to you. (Do you emit different page content? Redirect to a different sub-site? Generate and submit a purchase order for a new Tablet PC? You get the idea.)
if (Request.UserAgent.IndexOf("Tablet PC 1.7") > -1)
{
// The client browser is a Lonestar-upgraded Tablet PC.
ServeInkEnabledContent();
}
else
{
// The browser is either not a Tablet PC, or a pre-Lonestar Tablet PC
// (neither of which support hosting Ink).
ServeDownLevelContent();
}
What if you don't feel like writing custom back-end code, or you don't like the idea of maintaining and testing two different "views" of your Web application, for different client platforms? I hear you. There is a more elegant solution, although it's somewhat hackish: it is possible to wrap the instantiation and usage of any Microsoft.Ink types within a try/catch block, gracefully degrading our control's functionality on down-level platforms, in the absence of the Ink runtime library. Here's an example of how one might gracefully degrade the C# InkableUserControl we saw above, to act as a multi-line TextBox control on down-level systems:
public class InkableUserControl : System.Windows.Forms.UserControl
{
private object inkCollector; // weakly typed!
public InkableUserControl()
{
this.BackColor = Color.FromKnownColor(KnownColor.Window);
this.Size = new System.Drawing.Size(300, 200);
// Guard against absence of Tablet PC runtime with messy downcasting
// and exception-handling around JIT compilation.
try
{
// Note: Because JIT compilation happens on a method-by-method
// basis, we must factor out any calls to Ink into a subroutine,
// in order to catch the exception thrown from Fusion, here.
AttemptInitializeInk();
}
catch (System.IO.FileNotFoundException) // missing Microsoft.Ink.dll!
{
// We couldn't find Microsoft.Ink (not the version we wanted,
// anyway) so let's degrade down to multiline TextBox.
TextBox downlevelTextBox = new TextBox();
downlevelTextBox.Multiline = true;
downlevelTextBox.Dock = DockStyle.Fill;
downlevelTextBox.Text = "What? No Tablet PC?";
this.Controls.Add(downlevelTextBox);
}
}
private void AttemptInitializeInk()
{
inkCollector = new InkCollector(this.Handle);
((InkCollector)inkCollector).Enabled = true;
}
protected override void Dispose(bool @explicit)
{
if (@explicit)
{
// Fwd to IDisposable members.
((IDisposable)this.inkCollector).Dispose();
this.inkCollector = null;
}
base.Dispose(@explicit);
}
...
}
Note the need for messy downcasting (the InkCollector member is replaced by a weakly-typed System.Object reference) in addition to the crude exception handling around JIT compilation, thus avoiding a hard dependence on Microsoft.Ink. This is a very ugly, very error-prone coding style. I really don't recommend it. I hesitate even to demonstrate it publicly, because it's also fantastically inefficient. Each potential call into a Microsoft.Ink method will cause the .NET Framework assembly loader to make several requests to the Web server, probing for Microsoft.Ink.dll in the usual places, before eventually giving up and throwing an exception, which can be an expensive operation in itself. But I also realize that it can, sometimes, be very convenient to build and deploy one set of binaries in support of multiple client platforms. The good news is that we have options.
Conclusion
The Tablet PC is a powerful, innovative platform, and Microsoft has given us a plethora of tools and components for building rich applications to leverage its unique features. However, it's only natural to wonder how the capabilities of this new platform can be made to work well with last great innovation in software technology: thin-client Web applications.
At long last, the Lonestar release of the Tablet PC platform adds support for Ink in thin-client, mobile code application scenarios, whatever that means for you—HTML-based Web applications hosted in Internet Explorer or full-featured Windows Forms applications deployed over HTTP. No-touch deployment is the best bet for new development, but Ink functionality can be retrofitted into legacy browser-based applications fairly easily.
Once Lonestar ships, as part of Windows XP Service Pack 2, it's a good bet that hundreds (perhaps thousands) of Web-based applications will be greatly improved by taking advantage of the pen-based UI and tiny form factor of the Tablet PC. Exploring ways to make that happen has been the subject of this month's column. Only one burning question remains: who will the first to Ink-enable their blog?