span.sup { vertical-align:text-top; }
Write On!
Create Web Apps You Can Draw On with Silverlight 2
Julia Lerman
Code download available from theMSDN Code Gallery
This article discusses:
|
This article uses the following technologies: Silverlight 2, Expression Blend, Visual Studio 2008 |
This article is based on prerelease versions of Visual Studio 2008 SP1, Silverlight 2, and Expression Blend. All information herein is subject to change.
Contents
Introducing the InkPresenter
Tablet PC Not Required
InkPresenter 101
Adding Events and a Background
A More Stylish InkPresenter
Getting That Silverlight Look
Add a Background Image or Video to the InkPresenter
Stroke Design Basics
Handwriting Recognition
The InkAnalyzer Class
Analyzing on the Server Side
Hooking Up the Service to the Silverlight Client
Persisting Annotations Using Services
Store and Retrieve Annotations
Putting It All Together
Silverlight is one of the most exciting new Web technologies to come out of Microsoft in the past few years. It's such a big deal that the annual MIX conference now seems to center around Silverlight™ and its many capabilities. Silverlight 1.0 was initially released in 2007 and each new release of Silverlight 2 prior to its final release has been filled with impressive new features. One of the very cool features of Silverlight that hasn't gotten the attention it deserves is the InkPresenter control. The InkPresenter control enables Internet users to draw directly onto Silverlight applications from their browser.
Because Silverlight can be used on a variety of OSs and in a variety of browsers, so too can the InkPresenter, which removes another set of limitations around browser, OS and hardware. The sample application I'll present here uses the Microsoft® .NET Framework programming features of Silverlight 2 combined with the InkPresenter to let users annotate a predefined collection of images, perform handwriting recognition, save the annotations and recognized text to a server-side database, retrieve annotations for a selected image, and filter images based on their associated text. The database and handwriting recognition functionality are provided by a Windows® Communication Foundation (WCF) service. Figure 1 shows the completed application.
Figure 1 The Completed Application
It should be noted that everything described in this article can also be accomplished in Silverlight 1.0. In fact, prior to the availability of Silverlight 2, I wrote a similar application in Silverlight 1.0 with all of the same features. However, without the .NET-targeted code at your disposal on the client-side, you would need to leverage .NET Web services for more of the features.
Introducing the InkPresenter
The InkPresenter control that makes this application possible is a container for a collection of strokes. Each stroke is made up of a collection of StylusPoints. Note that there is another Stroke property in Silverlight that is a member of the Shape class, so be careful not to confuse the two.
If you think in terms of pen and paper, each time you touch the pen to the paper, you begin a stroke, and you then proceed to move the pen around whether you are drawing a circle or writing a word. Lifting the pen from the paper signals the end of that stroke. Placing the pen down again begins a new stroke.
Strokes have DrawingAttributes that define characteristics such as the stroke's color, width and a few other attributes. The individual points in a stroke also have properties: X and Y coordinates for location and a PressureFactor. PressureFactor is interpreted for computers with a digitizer and allows you to programmatically impact the stroke based on how hard the user is pressing the stylus to the digitizer. Figure 2 illustrates the class hierarchy.
Figure 2 The InkPresenter Class
As with other visual elements in Silverlight, the InkPresenter object and its children can be represented as XAML. Figure 3 shows three small strokes in an InkPresenter and Figure 4 shows the XAML representation of one stroke in the StrokeCollection. Even though the strokes are small, the digitizer collects quite a lot of data. If you were to do the same test using a mouse, much less data would be collected, as the Stylus points would only be doubles and points would be collected at less frequent intervals.
Figure 3 Some Strokes
Figure 4 First Stroke in an InkPresenter Control
<StrokeCollection>
<StrokeCollection xmlns="https://schemas.microsoft.com/client/2007">
<Stroke>
<Stroke.DrawingAttributes>
<DrawingAttributes Color="#FF000000"
OutlineColor="#00000000" Width="3" Height="3" />
</Stroke.DrawingAttributes>
<Stroke.StylusPoints>
<StylusPoint X="81.4583358764648" Y="96.5833282470703" />
<StylusPoint X="81.4583358764648" Y="96.5833282470703" />
<StylusPoint X="81.0833358764648" Y="96.4166717529297" />
<StylusPoint X="81.0833358764648" Y="96.4166717529297" />
<StylusPoint X="81.0833358764648" Y="96.4166717529297" />
<StylusPoint X="81.0833358764648" Y="96.4166717529297" />
<StylusPoint X="81.0833358764648" Y="96.4166717529297" />
<StylusPoint X="80.4583358764648" Y="96.8333282470703" />
<StylusPoint X="80.4583358764648" Y="96.8333282470703" />
<StylusPoint X="80" Y="97.2916717529297" />
<StylusPoint X="80" Y="97.2916717529297" />
<StylusPoint X="79.625" Y="97.75" />
<StylusPoint X="79.625" Y="97.75" />
<StylusPoint X="79.625" Y="97.75" />
<StylusPoint X="79.625" Y="97.75" />
<StylusPoint X="79.625" Y="96.5416717529297" />
<StylusPoint X="79.8333358764648" Y="95.7083358764648" />
<StylusPoint X="80.25" Y="94.7916641235352" />
<StylusPoint X="80.7916641235352" Y="93.5416641235352" />
<StylusPoint X="81.5" Y="92.125" />
<StylusPoint X="82.4166641235352" Y="90.4583358764648" />
<StylusPoint X="83.4583358764648" Y="88.5833358764648" />
<StylusPoint X="84.75" Y="86.5416641235352" />
<StylusPoint X="86.1666641235352" Y="84.3333358764648" />
<StylusPoint X="87.7083358764648" Y="82.1666641235352" />
<StylusPoint X="89.25" Y="79.9166641235352" />
<StylusPoint X="90.75" Y="77.9583358764648" />
<StylusPoint X="92" Y="76.0833358764648" />
<StylusPoint X="93.1666641235352" Y="74.8333358764648" />
<StylusPoint X="94" Y="73.625" />
<StylusPoint X="94.7083358764648" Y="73.1666641235352" />
<StylusPoint X="95.125" Y="73.1666641235352" />
<StylusPoint X="95.125" Y="73.1666641235352" />
<StylusPoint X="95.125" Y="73.1666641235352" />
<StylusPoint X="94.7083358764648" Y="73.5" />
</Stroke.StylusPoints>
</Stroke>
...
</StrokeCollection>
Working with the InkPresenter is actually a matter of creating and interacting with its strokes. However, the InkPresenter does none of this by default. It provides Events and Methods; it allows you to add and remove StrokeCollections and it allows you to access the Strokes in order to interact with them. But it is up to you to programmatically keep track of the mouse or stylus activity within the boundaries of an InkPresenter control and build the strokes.
Tablet PC Not Required
Before Windows Presentation Foundation (WPF) and Silverlight, developers were dependent on the Tablet PC SDK to create custom programs that took advantage of the Tablet PC drawing features. The SDK is set of COM APIs with .NET wrappers that enable development in .NET, Visual Basic® 6.0, and C++. The Windows XP Tablet PC Edition was the required OS.
Beginning with version 1.7 of the Tablet PC SDK, the critical control, InkOverlay, no longer required Full Trust on a system in order to run. It became possible to create ink-enabled Windows Forms controls and embed them into Web pages, as you would with any ActiveX® control. And although ActiveX controls were limited to use in Internet Explorer® only, this provided a first step at enabling developers to bring drawing and other inking capabilities to the Web.
In Windows Vista®, Tablet PC functionality is built into the OS as a first-class citizen and the development APIs for Tablet features have been incorporated into the WPF InkCanvas object. This means that any computer with the .NET Framework 3.0 installed can support Tablet functionality even if it isn't a Tablet PC. But it's important to recognize that using a stylus on a Tablet's digitizer results in much higher resolution (by orders of magnitude) than what a user will achieve using a mouse.
The ink capabilities in Silverlight are a subset of what's available in WPF. One major difference, however, is that while the WPF InkCanvas has a property called InkPresenter that displays the ink on the InkCanvas, in Silverlight there is no InkCanvas and you must work with the InkPresenter directly.
Most importantly, because Silverlight is not limited to Windows or Internet Explorer, more environments can use ink-enabled Web sites.
InkPresenter 101
Silverlight Tools for Visual Studio® 2008 embed a Silverlight XAML designer into Visual Studio 2008, but the design surface itself is read-only, so you may find it more convenient to work in Expression Blend™ 2.5 for some of the InkPresenter design work. The two applications are very well integrated, so this is quite simple to do and once you become familiar with Expression Blend, it's also a lot of fun.
I prefer to start my project in Visual Studio because the default project templates are very helpful. Then when I want to do some design work in XAML, I open the project in Expression Blend. You can do this easily from the Visual Studio IDE by right-clicking a XAML file in the Solution Explorer and selecting the option to open the file in Expression Blend. If you don't have Expression Blend 2.5, you can edit the XAML by hand right in Visual Studio 2008 and instantly see the results of your changes.
When creating a Silverlight project, you have the option to use Visual Basic or C#. Even though my Visual Basic skills are better, for this project, I choose to work in C# because I had a stockpile of JavaScript code from work I did in Silverlight 1.0, which readily ported to C#.
After creating the project in Visual Studio, the first task is to open the project in Expression Blend so that you can start designing. Once you have the XAML file open in the Expression Blend designer, you can add the InkPresenter control to the design surface. To find the InkPresenter control, you'll need to open the Expression Blend Asset Library by clicking the >> icon at the bottom of the toolbox. Then you'll have to click the Show All checkbox to see the InkPresenter as well as number of other less commonly used controls. Alternatively, you can use the search box to search for controls.
Drag the InkPresenter onto the XAML design surface; then, using its properties window, give the control a name. I've chosen "inkP" which will be used frequently in the code later in this article.
When the InkPresenter is selected on the design surface, you can see its boundaries, but when it is not selected it seems to disappear. The InkPresenter control is a container that is able to render ink and while it does have a background property, it does not have Fill, Stroke (for borders) or many other properties you will find in other controls. Therefore you'll want another control to provide the visual boundaries.
For a simple example, add a Rectangle to the canvas with the same placement and dimensions as the InkPresenter. By default, a Rectangle will have a black border (the Stroke property) with a StrokeThickness of 1. The Rectangle, which I've named "inkBorder," and InkPresenter should be siblings in the Canvas. The z-Order of the InkPresenter must be higher (on top of) the Rectangle—otherwise the Rectangle will hide the InkPresenter.
Because it is difficult to find the InkPresenter control on the design surface, when you need to select it, the easiest way is to select it in the Objects and Timeline panel. This action will then highlight the control on the design surface. If you were to test the solution so far (by pressing F5), you would see the Rectangle border, but if you tried to draw inside of it with a mouse or a stylus, nothing would happen yet.
Adding Events and a Background
As I mentioned above, the InkPresenter is merely a container for Stroke Collections. On its own, it does not have the ability to create strokes. You will have to do that programmatically by responding to events on the InkPresenter. The key events for capturing strokes are MouseLeftButtonDown, MouseMove, and MouseLeftButtonUp. When the InkPresenter receives a MouseLeftButtonDown event, you will need to create a new stroke in memory and add it to the InkPresenter's StrokeCollection. As the mouse moves around inside the InkPresenter creating MouseMove events, you will need to add StylusPoints to that Stroke. When the user causes the MouseLeftButtonUp event to fire, either by lifting the stylus from the digitizer or by releasing the mouse, you will then need to complete the stroke.
Expression Blend 2.5 makes it easy to wire up events to controls using the Properties window. Select the inkP control, then click the event icon in the top of the Properties window to see the control's events. Next, double-click the text box for each of the three aforementioned event handlers. For each event handler, a corresponding method will be added to the codebehind of the XAML control in Visual Studio. The integration between Visual Studio and Expression Blend works in both directions. Visual Studio will require you to confirm each change as it is made; so watch for the blinking Visual Studio icon on the taskbar.
Once the three handlers have been created, switch to Visual Studio to add the code in Figure 5 that will allow the InkPresenter to collect and display stroke data as the user interacts with the control. This code performs the tasks laid out previously: creating a new stroke and adding stylus points. The stylus points are accessed through the MouseEventArgs of both the MouseLeftButtonDown and MouseMove events.
Figure 5 Collect and Display Stroke Data
System.Windows.Ink.Stroke newstroke;
void inkP_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
inkP.CaptureMouse();
newStroke = new System.Windows.Ink.Stroke();
newStroke.StylusPoints.Add(e.StylusDevice.GetStylusPoints(inkP));
inkP.Strokes.Add(newStroke);
}
void inkP_MouseMove(object sender, MouseEventArgs e)
{
if (newStroke != null)
{
newStroke.StylusPoints.Add(e.StylusDevice.GetStylusPoints(inkP));
}
}
void inkP_MouseLeftButtonUp (object sender, MouseEventArgs e)
{
newStroke = null;
inkP.ReleaseMoustCapture();
}
There is one more piece to this puzzle. The InkPresenter must have a Background property in order to receive the mouse events. The background is similar to the Fill property of other controls but without the additional customization that you can perform on Fill. Depending on your design scenario, you could get creative with the background, but for now, simply set the Background property to "Transparent."
You are able to set this background in Expression Blend using the Properties window for the InkPresenter control by choosing a solid color brush for the background and then setting its Alpha value to 0. As an alternative to this, you can simply type Background="Transparent" directly into the XAML.
The following is the XAML for the two controls after the events have been wired up and the Background property has been designated:
<Rectangle Margin="20,30,35,24"
x:Name="inkBorder" Stroke="#FF000000"/>
<InkPresenter Margin="20,30,35,24"
x:Name="inkP"
MouseLeftButtonDown=
"inkP_MouseLeftButtonDown"
MouseLeftButtonUp=
"inkP_ MouseLeftButtonUp"
MouseMove="inkP_MouseMove"
Background="Transparent"
Opacity="1"/>
Now when you run the project from Expression Blend or from Visual Studio, you can see the strokes on the InkPresenter as they are drawn (see Figure 6).
Figure 6 Strokes on the InkPresenter
A More Stylish InkPresenter
As you have seen, the InkPresenter is a container, but more like a canvas than other visual elements, such as Rectangle. You will need to combine the InkPresenter with other elements in order to make it visually interesting; otherwise you would be doing Silverlight an injustice. Therefore, open up the existing XAML in Expression Blend so you can spruce things up a bit more by using the Rectangle that you added earlier to create the border for the InkPresenter.
First let's round out the corners of the border Rectangle. Select the Rectangle and change its RadiusX and RadiusY properties to 25. The Rectangle now has nice rounded edges; however, the InkPresenter's corners stick out of the visual border and will accept ink. The solution is to change, or clip, the InkPresenter's boundary to match the visual border. You can accomplish this using the Silverlight clipping feature to reshape the InkPresenter.
Expression Blend makes it easy to clip an element to match the shape of another element. Before doing this, however, create a copy of the inkBorder Rectangle for later use. In the Objects and Timeline window, right-click the new Rectangle. From its context menu, select Path and then Make Clipping Path. Expression Blend will then pop up a window telling you to choose which object will be clipped by the path—in other words, which object will adopt the shape of the Rectangle. Select the InkPresenter. Two things happen as a result of this: the InkPresenter now has the shape of the Rectangle and the new Rectangle is gone. When the Rectangle became the clipping path for the InkPresenter, it ceased to be an object. Now you see why you copied the Rectangle.
The resulting XAML looks like Figure 7. Run the project to test the new edge of the InkPresenter. It will look like Figure 8. You can use any shape as a clipping path in Silverlight and thus you can create any shape you like for an InkPresenter. Figure 9 shows an example of a random shape being used to clip an InkPresenter.
Figure 7 XAML for the Clipped Object
<Rectangle x:Name="inkBorder" Width="346" Height="234"
Stroke="#FF000000" Canvas.Top="25" Canvas.Left="25"
RadiusX="25" RadiusY="25"/>
<InkPresenter x:Name="inkP"
Width="607" Height="408" Canvas.Left="25" Canvas.Top="34"
MouseLeftButtonDown="inkP_MouseLeftButtonDown"
MouseLeftButtonUp="inkP_MouseLeftButtonUp"
MouseMove="inkP_MouseMove"
Background="Transparent"
Clip="M0.5,25.5 C0.5,11.692881 11.692881,0.5 25.5,0.5 L581.5, 0.5 C595.30712,0.5 606.5,11.692881 606.5,25.5 L606.5, 382.5 C606.5, 396.30712 595.30712,407.5 581.5,407.5 L25.5,407.5 C11.692881, 407.5 0.5,396.30712 0.5,382.5 z" >
</InkPresenter>
Figure 8 Clipping the Edges of the InkPresenter
Figure 9 This Randomly Drawn Shape Was Used to Clip an InkPresenter
Getting That Silverlight Look
Silverlight allows you to create interesting visual layers using transparency. Your InkPresenter can have the look of a semi-transparent surface as well. To achieve this effect, you should start by adding a background image to the canvas. I've used the background from the Silverlight.net Web site. You can easily set the background by dragging the image onto the canvas and setting its Stretch property to Fill. It's important to make sure that the z-Order of the Image is the first control listed in the design surface. Otherwise it will be positioned on top of the Rectangle and the InkPresenter, hiding both.
Next, modify the Rectangle to give it a black background. One way to do this is to select the Fill Brush in the Properties window and set its R, G, and B values to 0. Change the Opacity of the Rectangle to 10 percent to make this semi-transparent.
While it is possible to set the Background color on the InkPresenter and give it transparency, this transparency will impact the ink as well. I prefer to leave the Background of the InkPresenter completely transparent and use some other control to provide the effect. You may find it interesting to experiment with changing the Alpha value of the control's Background color and compare it to the effect of changing the control's Opacity. You can also impact the ink's transparency directly using the Alpha property of the DrawingAttribute's Color and OutlineColor properties. This has the exact same effect as the DrawingAttribute.Transparency property that exists in the WPF InkCanvas and in the Tablet PC SDK. Figure 10 shows the InkPresenter combined with a semi-transparent Rectangle to provide a nice visual effect for your drawing background.
Figure 10 Creating a Semi-Transparent Background
Add a Background Image or Video to the InkPresenter
The semi-transparent background is a very appealing solution for some scenarios, but it is also possible to use images and even videos as the background. To create such a background, the actual Background property of the InkPresenter does not change. Images or videos are added as child elements of the InkPresenter object. If the child element has no height, width, left, or top properties, it will inherit the properties of the parent InkPresenter.
Give it a try—add an image element to your XAML design surface using Expression Blend and then drag that image into the InkPresenter. Alternatively, you can add the XAML directly:
<InkPresenter x:Name="inkP"
Width="607" Height="426" Canvas.Left="25" Canvas.Top="34"
MouseLeftButtonDown="inkP_MouseLeftButtonDown"
MouseLeftButtonUp="inkP_MouseLeftButtonUp"
MouseMove="inkP_MouseMove"
Background="Transparent">
<Image Source="Assets/Leaves.jpg" Stretch="Fill" />
</InkPresenter>
Now it's possible to draw directly on the image. Alternatively, you could lower the Opacity value of the image to make it semi-transparent as well, as you can see in Figure 11.
Figure 11 Creating Semi-Transparency Using the Opacity Property
Adding a video is just as easy. Rather than an Image, you will use a MediaElement. Expression Blend treats videos differently than images. While you can drag and drop the video from the Silverlight project onto the XAML design surface, the file won't be found when you run the project. Instead, you will need to put the video inside the host Web project. Then the source attribute for the MediaElement needs to refer to the URL of the file. The Media Element will not automatically fill the InkPresenter as the Image did. You will have to manually adjust the size or add Stretch="Fill" directly into the XAML. Here's an example of a video that plays in the background of an InkPresenter:
<InkPresenter x:Name="inkP" . . . >
<MediaElement Height="246" x:Name="Butterfly_wmv" Width="345"
Source="https://localhost:52476/MSDNMagAnnotationClient_Web/Assets/Butterfly.wmv"
Stretch="Fill"/>
</InkPresenter>
You can learn more about using and interacting with MediaElements in the Silverlight documentation.
Stroke Design Basics
By default, the stroke's default drawing attributes create a black stroke with a height and width of 3. The value represents Device Independent Pixels (DIPs) and cannot be set to a value that is smaller than 2.
In the code, you can create methods and event handlers to impact the various stroke attributes such as penWidth and penColor. For example, a variable called currentColor can have its value changed by a control's click event and then currentColor can be used in the inkP_MouseLeftButtonDown event when a new stroke is created.
To try this out, add a variable declaration to the class, setting the default value to black, like so:
System.Windows.Media.Color currentColor = Colors.Black;
In the following example, I created a single method that can be used as the MouseLeftButtonDown event for any number of colored Rectangles. The method will determine the color of the Rectangle, then use the color as the value of the currentColor variable:
private void ChangeColor(object sender, MouseButtonEventArgs e)
{
Rectangle rec = (Rectangle)sender;
SolidColorBrush scb = (SolidColorBrush)rec.Fill;
currColor = scb.Color;
}
In the inkP_MouseLeftButtonDown method, add code to set the DrawingAttributes to the currentColor variable:
newStroke.DrawingAttributes.Color = currentColor;
Finally, you'll need a way to trigger the change. Add two Rectangle elements setting the Fill of one to black and the Fill of the other to red. Each Rectangle will need a MouseLeftButtonDown event to call the ChangeColor method. This XAML creates two Rectangle objects that render as circles:
<Rectangle MouseLeftButtonDown="ChangeColor"
Width="24" Height="22" Fill="#FF000000"
Stroke="#FF000000" RadiusX="25" RadiusY="25"
Canvas.Left="-10" Canvas.Top="282"/>
<Rectangle MouseLeftButtonDown="RedInk"
Width="24" Height="22"
Fill="#FFCE0C0C" Stroke="#FF000000"
RadiusX="25" RadiusY="25"
Canvas.Left="25" Canvas.Top="282"/>
Alternatively, you could write code that drills into a StrokeCollection in order to change the color of the existing strokes.
One of the DefaultDrawingAttributes for the Stroke object is OutlineColor. If you plan to draw on a multicolor background, such as an image, it is helpful to have a consistent outline color for the ink being drawn. You can add code to the inkP_MouseLeftButtonDown event to set the OutlineColor of the newStroke:
newStroke.DrawingAttributes.OutlineColor = Colors.White;
Figure 12 illustrates the concept.
Figure 12 Ink with a Border
Handwriting Recognition
One of the very cool features of Ink in applications is handwriting recognition. While this has been part of the Tablet PC SDK since the beginning and is a feature of WPF, it is not part of Silverlight. But you can still use handwriting recognition in your Silverlight applications by sending the stroke data to an ASP.NET Web Service or a WCF Service which will perform the recognition and return the results.
Microsoft Research used handwriting samples from more than one million people to create the algorithms for handwriting recognition. It may surprise you to know that the recognition works better with cursive than with block letters. There are also handwriting recognition engines for many languages including a variety of Asian languages.
The engine will analyze the collection of strokes. It can determine whether the strokes represent a word, a sentence, a paragraph, or a drawing. The recognition engine can identify individual words within a set of strokes that has multiple words and then recognize the word as a unit. Then, using its samples and algorithms, it responds with a selection of possibilities for what that word might be. If you send a set of words (such as a sentence), then it will return a set of words as well as a series of alternates. In the past, recognition was limited to Tablet PCs. Now it can be performed on any computer/browser combination that supports Silverlight!
The InkAnalyzer Class
System.Windows.Ink.InkAnalyzer performs the handwriting recognition. It's surprisingly simple to use this class. You merely pass a collection of strokes to an InkAnalyzer object, call its Analyze method, and determine if the Analyze succeeded using a Boolean property called Successful. If Successful is true, the GetRecognizedString method will return the best guess while the GetAlternates method returns an array of alternates strings.
Although the InkAnalyzer class is not part of the Silverlight API, you can still provide handwriting recognition to your Silverlight apps using Web or WCF services. The Web server can host the WPF APIs and provide the functionality for performing the recognition. This, however, will require a few conversions.
First, you will need to reference the APIs in Figure 13 in any component that will perform Ink Recognition. The first two APIs are readily available through the Add References interface in Visual Studio. The last two are installed with the Tablet PC SDK and can be found in Program Files\Reference Assemblies\Microsoft\Tablet PC\v1.7. You will want to watch out for ambiguous references between the IACore and IAWinFX namespaces when you are coding. Finally, you need to additionally reference the IALoader.dll that is installed with the Tablet PC SDK. You can find this file in C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin if you're running Windows Vista.
Figure 13 WPF APIs
| API | Function |
|---|---|
| PresentationCore.dll | Contains the System.Windows.Ink APIs. |
| WindowsBase.dll | Contains functionality used by Collections. |
| IAWinFX.dll | Adds the InkAnalyzer class and functionality to System.Windows.Ink. |
| IACore.dll | Provides the InkAnalyzerBase class and functionality via System.Windows.Ink.AnalysisCore. |
In order to transmit the Strokes from the InkPresenter across the wire to a service, the strokes need to be serialized. However these objects are not serializable. Therefore, you will need to create a string-based XAML representation of the StrokeCollection. The string can then be serialized and sent to the service.
In Silverlight 1.0, you needed to use JavaScript to create this string representation. In Silverlight 2, you have the benefit of being able to use LINQ to XML. (If you are using Visual Basic, you get the added benefit here of XML literals.)
The code shown in Figure 14 drills into the StrokeCollection object, reading the DrawingAttributes, the StylusPoints and their details, and uses LINQ to XML to create its XAML representation. This code will be used for a number of purposes, including the handwriting recognition, even though the recognition functions will ignore the DrawingAttributes. If you were creating this method solely for the purpose of recognition, you could leave out the code that collects the DrawingAttributes.
Figure 14 Creating XAML Representation from Strokes
public XElement StrokestoXAML(StrokeCollection mystrokes)
{
//this method uses LINQ to XML
//be sure to add the namespace to each element in order to load back
//into a new StrokeCollection later with the XAMLReader
string xmlnsString = "https://schemas.microsoft.com/client/2007";
XNamespace xmlns = xmlnsString;
XElement XMLStrokes = new XElement(xmlns + "StrokeCollection",
new XAttribute("xmlns", xmlnsString));
//create stroke, then add to collection element
XElement mystroke;
foreach (Stroke s in mystrokes)
{
mystroke = new XElement(xmlns + "Stroke",
new XElement(xmlns + "Stroke.DrawingAttributes",
new XElement(xmlns + "DrawingAttributes",
new XAttribute("Color", s.DrawingAttributes.Color),
new XAttribute("OutlineColor",
s.DrawingAttributes.OutlineColor),
new XAttribute("Width", s.DrawingAttributes.Width),
new XAttribute("Height", s.DrawingAttributes.Height))));
//create points separately then add to mystroke XElement
XElement myPoints = new XElement(xmlns + "Stroke.StylusPoints");
foreach (StylusPoint sp in s.StylusPoints)
{
XElement mypoint = new XElement(xmlns + "StylusPoint",
new XAttribute("X", sp.X.ToString()),
new XAttribute("Y", sp.Y.ToString()));
//add the new point to the points collection of the stroke
myPoints.Add(mypoint);
}
//add the new points collection to the stroke
mystroke.Add(myPoints);
//add the stroke to the collection
XMLStrokes.Add(mystroke);
}
return XMLStrokes;
}
Notice the use of the Namespace to build the XAML. The Silverlight 2 XMLReader requires this namespace to be used in the root of a XAML element to Load the XAML into an object at a later time.
Analyzing on the Server Side
Now the string can be passed as a parameter to a service operation, which will return the results as a string. The method used by the operation must recreate a StrokeCollection from the XAML string. Then the StrokeCollection can be sent to the InkAnalyzer. On the server, you no longer have access to the Silverlight APIs. Instead, you'll be using the WPF APIs.
While WPF also has a XAMLReader.Load method, the WPF StrokeCollection is slightly different from the Silverlight StrokeCollection, so the schema will not be recognized and XAMLReader.Load will fail. Instead, you can use LINQ to XML to drill easily into the XAML string and read the data from individual Stroke elements, then build up new WPF Stroke objects.
The differences between the WPF and Silverlight StrokeCollection are minimal. For example, WPF Stroke.DrawingAtrributes do not have an Outline Color and the StylusPoints in a WPF stroke are based on a device-independent coordinate system rather than the pixel values used in Silverlight. Even with these differences, the Stroke, StrokeCollection, and StylusPoint classes are in the same namespaces in Silverlight as they are in WPF.
The CreateWPFStrokeCollectionfromXAML method in Figure 15 uses LINQ to XML to create a collection of Stroke elements, then iterates through those elements creating a new Stroke object for each one. The method uses LINQ again to create a collection of the StylusPoints and then creates the StylusPointsCollection for each stroke. Note that this is not a complete recreation of the StrokeCollection as you do not need DrawingAttributes, such as color, in order to perform the recognition. These two methods rely on the PresentationCore and WindowsBase APIs discussed previously. Once the StrokeCollection has been created, it can be passed to a method that will perform the analysis.
Figure 15 The CreateWPFStrokeCollectionfromXAML Method
private StrokeCollection CreateWPFStrokeCollectionfromXAML
(string XAMLStrokes)
{
//because namespace was used to create this
// (for Silverlight to reuse the XAML),
//you need to insert the namesace into the Descendent's parameter
var xmlElem = XElement.Parse(XAMLStrokes);
XNamespace xmlns = xmlElem.GetDefaultNamespace();
StrokeCollection objStrokes = new StrokeCollection();
//Query the XAML to extract the Strokes
var strokes = from s in xmlElem.Descendants(xmlns+ "Stroke") select s;
foreach (XElement strokeNodeElement in strokes)
{
//query the stroke to extract its StylusPoints
var points = from p
in strokeNodeElement.Descendants(xmlns + "StylusPoint") select p;
//create Stylus points collection from point element values
StylusPointCollection pointData =
new System.Windows.Input.StylusPointCollection();
foreach (XElement pointElement in points)
{
double Xpoint = Convert.ToDouble(pointElement.Attribute("X").Value);
double Ypoint = Convert.ToDouble(pointElement.Attribute("Y").Value);
pointData.Add(new StylusPoint(Xpoint, Ypoint));
}
//create a new Stroke from the StylusPointCollection
System.Windows.Ink.Stroke newstroke = new
System.Windows.Ink.Stroke(pointData);
//add the new stroke to the StrokeCollection
objStrokes.Add(newstroke);
}
return objStrokes;
}
The method in Figure 16 can be used in a WCF or ASMX Web service to accept a StrokeCollection from an InkPresenter and return a string representing the best guess. This functionality relies on the IAWinFX and IACore assemblies.
Figure 16 Passing String to Recognizer
public string RecognizeStrokes(string XAMLStrokes)
{
try
{
//custom method to create WPF StrokeCollection from the string-based XAML
var strokeColl = CreateWPFStrokeCollectionfromXAML(XAMLStrokes);
var IA = new System.Windows.Ink.InkAnalyzer();
IA.AddStrokes(strokeColl);
var status = IA.Analyze();
if (status.Successful)
return IA.GetRecognizedString();
else
return "Not Recognized";
}
catch (Exception ex)
{
//trap and display errors at design time, not in production code
return "error:" + ex.Message;
}
}
Hooking Up the Service to the Silverlight Client
With the RecognizeStrokes method wrapped inside an ASMX or WCF service, you can easily call the Web service method from within the Silverlight application. I have used a WCF service to provide the handwriting recognition in my solution. If you need any help with WCF, there is a simple QuickStart on the Silverlight.net Web site that shows how to create a WCF service that can be accessed by Silverlight.
While it is common in Windows-based applications to have recognition performed on the fly as the ink is being drawn, in a distributed application it is more sensible to have the user explicitly request recognition. Therefore you will need a control on the design surface to start the process. Create a control such as a button in XAML. You'll need an event handler for the button's Click event that will trigger the GetXAMLfromStrokes method; then send the resulting XAML to a Web service for recognition. In my Web service, the operation is called Recognize. You will also need a TextBlock control to display the returned string. I've called it RecoText.
When adding WCF Service references to a Silverlight app, the proxy only implements asynchronous calls for the service operations. Therefore, you'll need a method for handling RecognizeCompleted (see Figure 17). If the annotation is written on multiple lines, it will be recognized with hard returns, so I've used a ScrollViewer control in place of a TextBox for RecoText, as you see in Figure 18.
Figure 17 Recognize Operations
private void RecognizeButtonHandler(object sender, RoutedEventArgs e)
{
StrokeCollection sc = inkP.Strokes;
XElement sXML = StrokestoXAML(sc,true);
string ss = sXML.ToString();
GetRecoString(ss);
//the next two lines are to test the validity of XAML created
//(this would make a great unit test for this application)
//StrokeCollection sc2 = new StrokeCollection();
//sc2 = (StrokeCollection)System.Windows.Markup.XamlReader.Load(ss);
}
private void GetRecoString(string inkStrokes)
{
Binding binding = new BasicHttpBinding();
EndpointAddress endpoint = new
EndpointAddress("https://myserver/SilverlightInkService.svc");
var svc = new ServiceReference1.SilverlightInkServiceClient(binding,
endpoint);
svc.RecognizeCompleted += new
EventHandler<ServiceReference1.RecognizeCompletedEventArgs>
(svc_RecognizeCompleted);
svc.RecognizeAsync(inkStrokes);
}
private void svc_RecognizeCompleted(object sender,
ServiceReference1.RecognizeCompletedEventArgs e)
{
RecoText.Text = e.Result.ToString();
}
Figure 18 Using a ScrollViewer Control for Recognized Text
For a better testing experience, you may want to add one more button on the XAML page that will clear the ink and the recognized text so that you can experiment with a variety of annotations. As its name indicates, InkPresenter.Strokes.Clear will remove the strokes from the InkPresenter.
Persisting Annotations Using Services
Another important use of Web services in this solution is for persisting annotations. Whether the StrokeCollection is handwritten text, a drawing, or some other markup, in most cases, you will want to retain it in a database or some other type of data store. It is also possible to save the annotation data on the user's computer using IsolatedFileStorage which is available in Silverlight. However, in this article I will focus on server-side persistence.
In the case of persistence, most often you will save the entire StrokeCollection, which can be added to another InkPresenter at any time. And as I've mentioned, because the StrokeCollection object must be serialized, the simplest method is to create the XAML string representation using the GetXAMLfromStrokes method.
With the XAML string in hand, the rest of the job is no different than storing any other text in a database. Keep in mind that the XAML could get quite large in the case of drawings, so you will have to consider that possibility when defining the database field that will store the XAML as well as client and service settings to accommodate the transfer of large amounts of data.
With WCF, you may also want to consider using JavaScript Object Notation (JSON) serialization, which is a more compressed format than XML serialized data. While it is possible to take advantage of the XML data type in SQL Server® 2005 and later, unless you plan to leverage the benefits of the XML data type (which can be queried and indexed, has schema support, and allows for data modification) the SQL Server types such as nvarchar or even nvarchar(MAX) for anticipated large drawings should do the trick.
In the sample app, users will have a variety of images to select from. The annotations for each image will be stored in a database with the image's unique file name. When an image is selected, its annotation can be retrieved from the database and displayed.
Store and Retrieve Annotations
In the case of the recognition, only a single string was being passed to and from the service. However, when storing and retrieving the data, you will be passing in the XAML and possibly other metadata relating to the annotation such as date created, the user, or references to an image or video to which the annotation belongs. WCF manages these different components of a message with DataContracts. Using the example of an annotation that belongs to an image file I'll create an operation that can store the path of an image along with the XAML and then retrieve the XAML for a specified file path.
This WCF service uses a combination of interfaces and attributes to define the operation and the data contracts. You can see in Figure 19 that three separate service operations are defined: StoreImageXAML, RetrieveImageXAML, and Recognize. The StoreImageXAML takes a parameter of type ImageXAMLComposite, which is defined by the ImageXAMLComposite class as a DataContract. The other two operations are much less complex as they only accept and return a single string. The service class implements these three methods which call helper methods for the recognition and database interaction. The methods, which work with the database, use LINQ to SQL (see Figure 20).
Figure 19 Defining OperationContracts
namespace SilverlightInkWCFService
{
[ServiceContract]
public interface ISLInk
{
[OperationContract]
void StoreImageXAML(ImageXAMLComposite value);
[OperationContract]
string RetrieveImageXAML(string imageName);
[OperationContract]
string Recognize(string XAMLString);
}
[DataContract]
public class ImageXAMLComposite
{
string imagePath;
string xamlString;
[DataMember]
public string XAMLString
{
get { return xamlString; }
set { xamlString = value; }
}
[DataMember]
public string ImagePath
{
get { return ImagePath; }
set { ImagePath = value; }
}
}
}
Figure 20 Calling Recognition Helper Methods
public void StoreImageXAML(ImageXAMLComposite value)
{
//ExistingRows, InsertRow and updateRow use LINQ to the SQL
//SubmitChanges
if (ExistingRows(value.ImagePath) == true)
insertRow(value.ImagePath, value.XAMLString);
else
updateRow(value.ImagePath, value.XAMLString);
}
public ImageXAMLComposite RetrieveImageXAML(string imagePath)
{
//getXAML method performs a LINQ to SQL query
return getXAML(imagePath);
}
public string Recognize(string XAMLString)
{
return RecognizeStrokes(XAMLString);
}
Putting It All Together
Here I've covered all of the critical pieces for creating and interacting with an InkPresenter, performing handwriting recognition, and storing and retrieving the XAML representation of the annotations through services. If you do not have a Tablet PC or are not running Windows Vista, you will still be able to use the application, even with a mouse, though the resolution won't be as fine.
Another interesting possibility is to use videos. While you can use the Silverlight animation and triggers to play back strokes, coordinating the timing with a video will be challenging, but fun.
You can download Silverlight, the SDK, Silverlight Tools Beta2 for Visual Studio 2008 and Expression Blend 2.5 June 2008 Preview from go.microsoft.com/fwlink/?LinkId=122132.
Special thanks to Stefan Wick from Microsoft, who offers help in the Notebook, Tablet PC, and UMPC Development MSDN® forum. He provided important guidance for this article.
Julia Lerman, a .NET consultant, has been building software for more than 20 years. She is well-known in the .NET community as a conference speaker, author, Microsoft.NET MVP, and leader of the Vermont .NET User Group. Her upcoming book is titled Programming Entity Framework. Julia blogs at thedatafarm.com/blog.