Creating Interactive Hand-Written Web Pages on Tablet PC

  • Article

 

Gavin Gear
Microsoft Corporation

December 2005

Applies to:

   Microsoft Windows XP Tablet PC Edition
   Windows XP Tablet PC Edition Development Kit 1.7
   Web Development

Summary: This paper explains how to create and transform ink content from your application into interactive HTML Web pages by using components from the Windows XP Tablet PC Edition Development Kit 1.7. (12 printed pages)

Click here to download the sample code for this article.

Contents

Introduction
Overview of the InkWebPublisher Sample Application
Example Web Page Created on the Tablet PC
The InkWebPublisher Application
Collecting the Ink
Adding Images
Associating Strokes with a URL (InkHyperlinks)
Generating the HTML and Images
Background and Inactive Ink Layers
Anti-Aliasing Issues
High Quality Rendering of Ink onto Images
Loading a Session from HTML
Parsing the HTML
Other Possibilities

Introduction

Most Web pages are based on text, pictures, and graphics. The Tablet PC provides another method to convey information by incorporating written content. Adding handwriting to your Web pages enables you to have a more personal look and feel. Incorporating pen-enabled actions means that you users can enter data while standing, rather than needing a place to rest a keyboard. Users can also annotate over images, providing greater ability to easily point out specific details in those images. By creating interactive ink content, you make your Web site more interesting, more useful, and provide extended feedback to the user.

Overview of the InkWebPublisher Sample Application

The InkWebPublisher sample application enables users to:

  • Draw ink.
  • Insert images.
  • Create InkHyperlinks (collections of strokes that represent a hyperlink).
  • Save ink to HTML
  • Load ink from HTML.

This application creates interactive hyperlinks (InkHyperlinks) from ink strokes by using rollover images and JavaScript rollover code. This article demonstrates how to transform ink content into interactive Web pages by using a Windows Forms application written in C# and that uses the Tablet PC SDK 1.7.

Example Web Page Created on the Tablet PC

The following screenshots show the InkWebPublisher application and content exported to a Web browser, in this case Internet Explorer. Details are incorporated in the following sections.

Figure 1. Sample Web page with ink in InkWebPublisher

Figure 2. Sample Web page with ink in the browser

The InkWebPublisher Application

The following steps describe how to create a Tablet PC application that produces ink-based Web pages:

Collecting the Ink

The InkWebPublisher application uses a Panel object for its client area, and attaches an InkOverlay to the panel in order to collect ink, as shown in the following code:

private System.Windows.Forms.Panel panelInkSurface;
this.panelInkSurface = new Panel();
this.panelInkSurface.Size = new Size(1600, 1200);
…this.inkOverlay_ = new InkOverlay();
this.inkOverlay_.AttachedControl = this.panelInkSurface;
this.inkOverlay_.AttachMode = InkOverlayAttachMode.InFront;

Use the InkOverlayAttachMode.InFront value to ensure that the ink always renders on top of the other client content.

Adding Images

The InkWebPublisher application uses PictureBox objects to display and store images and their accompanying metadata. The PictureBox objects are attached to the Panel, and render underneath any collected ink.

The InkWebPublisher application enables users to select strokes and associate those strokes with a URL. When the user saves the HTML page, the applications saves ink as two images: one for static rendering (inactive) and one with a transform for a rollover effect.

Internally, the InkWebPublisher application keeps track of these InkHyperlinks with objects of a simple class, as shown in the following code:

   public class InkHyperlink
   {
      public InkHyperlink(Strokes strokes, string href)
      {
         LinkStrokes = strokes;
         Href = href;
      }

      public Strokes LinkStrokes;
      public string Href;
   }

The Strokes collection helps the images persist for each InkHyperlink object. The application uses the href string for the actual hyperlink URL. When persisted, this becomes an <a> element, where href=<contents of InkHyperlink.Href>. The JavaScript rollover GIF images are displayed for static and onMouseOver effects.

Generating the HTML and Images

Web pages created with the InkWebPublisher are comprised of the following layers (ordered from back to front):

  • Background
  • Inactive ink GIF image (ink that is not associated with links)
  • Images that may contain ink annotations (ink rendered onto the image)
  • Hyperlinks that use JavaScript rollover images—two for each hyperlink
  • JavaScript code for rollovers

The Ordering of Layers

As an example, consider the following Web page in Figure 3.

Figure 3. Web Page Created in InkWebPublisher

Web pages created by InkWebPublisher contain the following layers, as shown in Figure 4.

Layer summary:

0—The HTML page background.

1—The "Inactive ink" layer. This is the GIF image that represents the ink that is not included in any InkHyperlinks.

2—Annotated images. This layer consists of images placed in the page that may include ink annotations.

3—Rollover images. These are the images that make up the interactive InkHyperlinks.

Figure 4. The layers of the Web page

The Web page is represented by the following HTML:

<html><head><title>Ink Web Page</title></head>
<body bgcolor="FFFFFF">

<IMG STYLE="position:absolute; BORDER:0px; TOP:22px; LEFT:14px; WIDTH:272; HEIGHT:231" SRC="Windows_Page_images/632482947617218208.gif"></IMG>

<IMG STYLE="position:absolute; BORDER:0px; TOP:26px; LEFT:75px; WIDTH:200px; HEIGHT:200px;" SRC="./Windows_Page_images/ms-windows_logo.jpg"></IMG>

<a href="https://www.microsoft.com/windows" onMouseOver="mouseover('image_632482947619721808', 'rollover_632482947619721808')" onMouseOut="mouseover('image_632482947619721808', 'leave_632482947619721808')">
<IMG NAME="image_632482947619721808" STYLE="position:absolute; BORDER:0px; TOP:211px; LEFT:178px; WIDTH:133; HEIGHT:48" SRC="Windows_Page_images/632482947619721808.gif"></IMG>
</a>

<script language="javascript">
function mouseover(imageName, imageObject) {
document.images[imageName].src = eval(imageObject + ".src");
return true;
}

rollover_632482947619721808 = new Image(133,48);
rollover_632482947619721808.src = "Windows_Page_images/632482947619721808_rollover.gif";
leave_632482947619721808 = new Image(133,48);
leave_632482947619721808.src = "Windows_Page_images/632482947619721808.gif";

</script>
</body>
</html>

Background and Inactive Ink Layers

Examining them from back to front, the first layer for this page is the background layer. On top of the background layer is the inactive ink GIF image, the layer shown in Figure 5.

Figure 5. Inactive ink GIF image

You generate the inactive ink GIF image from a Strokes collection that includes all Stroke objects not associated with a hyperlink. Because the Ink class defines the Save method, and a Strokes collection does not define a Save() method, first create a temporary Ink object and then call Save(). You do this whenever the InkWebPublisher saves ink as a GIF image. The following code sample shows this entire process:

// Save the gif image
Ink tempInk = new Ink();
foreach(Stroke s in strokes)
{
   Stroke stroke = CloneStroke(ref tempInk, s);
   stroke.DrawingAttributes = s.DrawingAttributes;
}

string ticksString = DateTime.Now.Ticks.ToString();
string gifFileName = imagesPath_ + "\\" + ticksString + ".gif";
SaveInkAsGif(tempInk,  gifFileName);
...

// CloneStroke
// Helper method to create a copy of a stroke, 
// including X, Y, Pressure (if avail) and DrawingAttributes
public static Stroke CloneStroke(Ink ink, Stroke stroke)
{
       TabletPropertyDescriptionCollection properties = new TabletPropertyDescriptionCollection();
       
       properties.Add(
              new TabletPropertyDescription(
                     PacketProperty.X, 
                     stroke.GetPacketDescriptionPropertyMetrics(PacketProperty.X)));
       properties.Add(
              new TabletPropertyDescription(
                     PacketProperty.Y, 
                     stroke.GetPacketDescriptionPropertyMetrics(PacketProperty.Y)));

       try
       {
              TabletPropertyDescription pressure = new TabletPropertyDescription(
                     PacketProperty.NormalPressure, 
                     stroke.GetPacketDescriptionPropertyMetrics(PacketProperty.NormalPressure));

              properties.Add(
                     new TabletPropertyDescription(
                     PacketProperty.NormalPressure, 
                     stroke.GetPacketDescriptionPropertyMetrics(PacketProperty.NormalPressure)));
       }
       catch
       {
              // If ink was input with the mouse, we don't have pressure
              // so do nothing here...
       }
       
       Stroke returnStroke = ink.CreateStroke(stroke.GetPacketData(), properties);
       returnStroke.DrawingAttributes = stroke.DrawingAttributes.Clone();
       return returnStroke;
}...

private void SaveInkAsGif(Ink ink, string filename)
{
   byte [] inkData = ink.Save(PersistenceFormat.Gif, CompressionMode.NoCompression);
   FileStream imageStream = File.OpenWrite(filename);
   imageStream.Write(inkData, 0, inkData.Length);
   imageStream.Close();
}

Anti-Aliasing Issues

Saving ink as GIF images may cause issues when the GIF image is transparent, no ink exists, and ink is rendered over a dark background. This can occur when ink is used to annotate a dark portion of an image or when a persisted ink GIF image is placed over a dark background. This happens because anti-aliasing (the blending of edges) is performed when the ink is persisted as a GIF image, and the Renderer defaults to a white background surrounding the ink. The effect occurs where the transition from opaque ink to the transparent portion of the image around that ink blend together. You can see how smooth the ink looks when the ink is blended over a white background, as in Figure 6.

Figure 6. Annotated Image.

Because the Renderer defaults to a white background around the ink, the persisted ink GIF image over a darker background image renders a very different look, as shown in Figure 7.

Figure 7. Example of Aliasing

High Quality Rendering of Ink onto Images

Calling Renderer.Draw with a graphics object created by Control.CreateGraphics does not result in a high quality anti-aliased rendering. In order to perform high quality anti-aliased rendering, you must call into native code and have the Renderer use a Device Independent Bitmap. The result is shown in Figure 8. For more information about using a Device Independent Bitmap for smoother rendering, see Printing Ink in the MSDN Library.

Figure 8. Example image with ink rendered onto image

When you save a Web page to HTML, a copy of the original image is saved with a different extension. When you load a Web page back into the InkWebPublisher application, the original image is loaded rather than the rendered image. This prevents re-rendering losses, and enables you to move the ink without leaving old ink renderings on top of the image.

Normal:

Rollover:   

Figure 9. Hyperlink rollover images

Figure 9 shows images created from InkHyperlink objects, each of which contains a Strokes collection. For each InkHyperlink, the Stroke objects are cloned to a second Strokes collection, and a transformation is applied that creates a rollover effect. In Figure 9, a white-on-black effect is used. You then save the two Strokes collections as GIF images by using the method described earlier. The DoOutlineEffect method creates the rollover effect.

private void DoOutlineEffect(Ink ink, Strokes sourceStrokes)
{
   // Now apply the transforms for the rollover image
   foreach(Stroke s in sourceStrokes)
   {
      // Create the fat black outline 4 times as thick as the normal strokes
      Stroke stroke = CloneStroke(ink, s);
      stroke.DrawingAttributes.Width = stroke.DrawingAttributes.Width * 4.0f;
      stroke.DrawingAttributes.Color = Color.Black;
   }
   // Create a copy of the sourceStrokes as is, make these white for high contrast
   foreach(Stroke s in sourceStrokes)
   {
      Stroke stroke = CloneStroke(ink, s);
      stroke.DrawingAttributes.Color = Color.White;
      stroke.DrawingAttributes.AntiAliased = true;
   }
}

Loading a Session from HTML

You use the XmlDocument class to save and load HTML in the InkWebPublisher application. This requires that the HTML follows XML guidelines.

Parsing the HTML

The following actions occur when you load an HTML document back into the InkWebPublisher application:

  • InkWebPublisher parses the background color and page title, and the application applies values to the session.
  • For each GIF image, the application calls Ink.Load() on a temporary Ink object. The Stroke objects in that Ink object are then cloned and added to the application’s main Ink object. If the GIF image corresponds to a rollover image, an InkHyperlink is added in the application.
  • For all other images, the application attempts to load the original (pre-rendered) image, if it exists. The pre-rendered images have a unique extension. If the pre-rendered image does not exist, the application loads the rendered image.

Other Possibilities

This discussion merely begins to look at the features that you could incorporate into an ink-enabled Web control. Some other possibilities include:

  • Using handwriting recognition to add alt text for images that make up the Web page. This would assist in search engine support, and would provide something to display if the images could not be loaded.
  • Using handwriting recognition to set the page title, description, metatags, and other components of the Web page.
  • Scaling images and handwritten content. Often people’s handwriting is larger than it needs to be in order to be legible. If you scale the written content, it may enable more content to be shown per Web page.
  • Programming in-place ink to text conversion by using the Tablet PC SDK recognition APIs.
  • Combining handwriting and typed text in the application.