Application Verifier for Windows CE and Windows Mobile 5.0 (Windows CE 5.0)

Douglas Boling
President, Boling Consulting Inc.

July 2005

Applies to:
   Windows CE .NET version 4.2
   Windows CE version 5.0
   Windows Mobile version 5.0
   Visual Studio 2005

Summary: This article discusses how to use the Application Verifier program to increase the stability of your embedded Windows CE and Windows Mobile applications. (17 printed pages)

Contents

Introduction
Using Application Verifier
Finding Real-World Leaks
Separating the Wheat from the Chaff
Using Application Verifier on Embedded Systems
Learning How Application Verifier Works
Conclusion

Introduction

Memory leaks can be fatal to embedded and mobile applications. It's not that embedded operating systems are fragile; it's simply the way they are designed. Desktop computer systems run very fast Pentium III and IV class processors with hundreds of megabytes of random access memory (RAM), but embedded Windows CE and Windows Mobile systems run significantly slower embedded CPUs (which require much lower power) with perhaps 2 to 6 megabytes of RAM that is devoted to running 10 to 20 applications.

Because so little RAM is available, mobile and embedded developers have to constantly think about how to reduce the memory footprint of their applications. With little RAM to spare, wasting the RAM by forgetting to close a handle or destroy a Graphics Device Interface (GDI) object can affect the performance of the system. Unfortunately, neither the Win32 application programming interface (API) or the Microsoft Foundation Classes (MFC) makes it easy to avoid memory leaks. Although managed programming would solve many — but not all — of the resource problems, many developers resort to writing native code for size, performance, or real-time reasons.

To fight memory and other types of leaks, Microsoft developed the Application Verifier program. Application Verifier analyzes applications at run time to monitor for leaks of all sorts, including memory leaks, handle leaks, and leaks of GDI objects. You may think that memory leaks come from for those lazy developers who simply don't know how to program. However, an application can leak resources in many subtle ways, and the best developers can be caught unaware. Fortunately, Application Verifier checks for as many leak situations as possible. The following list shows the types of leak conditions that Application Verifier monitors:

• Heap creation
• Fibers
• Heap allocations (from various sources)
• Device context handles
• File handles
• Brush objects
• File mapping objects
• Image lists
• Find handles
• Pen objects
• Registry handles
• Regions
• Thread handles
• Cursor objects
• Duplicate handles
• Icon objects
• Power notifications requests
• Palette objects
• Driver power relationships
• Font objects
• Driver power requirements
• Bitmap objects
• Device notification requests
• Message queue handles

In addition to tracking handles and GDI objects, Application Verifier will also test the integrity of an application's heap. Checks include flagging use after free, double frees, and buffer overruns.

Using Application Verifier

Application Verifier is a tool that typically runs on a development desktop computer that is attached to a Windows CE–based system: either a Windows Mobile–based device, such as a Pocket PC or Smartphone, or an embedded device running Windows CE. Application Verifier monitors all API calls that may allocate or free handles or memory. When the application closes, Application Verifier checks to see if all of the handles that were allocated during the life of the application were freed or released by the application before termination. Any handles or memory that the application did not free are recorded in a file — along with a stack trace for the thread that made the allocation at the point when the handle was created or memory was allocated.

Developers do not need to do any special programming to use Application Verifier. In fact, Application Verifier can analyze applications without source code; although, if you have .map files for the application, localizing the leaks is significantly easier.

Application Verifier is part of the Windows CE Test Kit (CETK) and is delivered with every version of Platform Builder, the tool used to port Windows CE to new hardware platforms. Fortunately, developers don't have to buy Platform Builder to get access to Application Verifier — a trial version of Platform Builder is available from Microsoft. Although the evaluation version has a four-month time bomb for creating new images, it includes a fully functioning version of the CETK.

Application Verifier is closely tied to the version of Windows CE it is written for. Because of this dependency, you must use the correct version of Platform Builder for the device platform you are targeting. For example, if you want to use Application Verifier on Windows Mobile 2003 devices, you should use the version of Application Verifier that comes with Platform Builder version 4.2. For Windows Mobile version 5.0 devices, you need the version of Application Verifier designed for those devices. Windows Mobile 5.0–based systems are based on Windows CE version 5.01, an updated release of Windows CE version 5.0. Although there isn't a version of Platform Builder for Windows CE 5.01, Microsoft has released a stand-alone version of Application Verifier that supports Windows Mobile 5.0–based devices.

Simple Walkthrough

The install program for the CETK does not provide a link to Application Verifier on the Start menu, so you must go to the program directory and start it. The default directory for Application Verifier for Platform Builder 5.0 is as follows: C:\Program Files\Windows CE Platform Builder\5.00\cepb\wcetk\ddtk\desktop. The name of the remote version of Application Verifier is appverifce.exe.

When started, Application Verifier displays the window shown in Figure 1 on your desktop computer.

Figure 1. The Application Verifier main window. Click the thumbnail for a larger image.

Although Application Verifier uses the older Platform Manager infrastructure for communication between the development desktop computer and the device, it does not use the standard menu layout of other remote tools that Windows CE and Windows Mobile developers are familiar with. If you click the Connect button, the dialog box in Figure 2 displays.

Figure 2. Application Verifier's connect dialog box

If you click the Connect button in the Device Connection dialog box, a device selection dialog box that is familiar to all developers who have used Microsoft's remote tools opens. Figure 3 shows the Select a Windows CE Device dialog box.

Figure 3. The Platform Manager connect dialog box

In Figure 3, note that the dialog box does not list any Windows Mobile 5.0 devices. The newer tools, such as Visual Studio 2005, do not use the Platform Manager interface for communication; therefore, the dialog box does not list the software development kits (SDKs) for the Windows Mobile devices. However, if your device is connected by means of ActiveSync, and your computer has the Pocket PC 2003 SDK installed, you can select the Pocket PC device, and Application Verifier should connect. The connection process may take a few seconds, and Application Verifier may freeze momentarily.

After you are connected, and you are back to the main window, your next task is to select the application, or applications, to monitor. If you click the Add button, it prompts Application Verifier to provide a list of all applications and dynamic link libraries (DLLs) on the device. Figure 4 shows the Add Application dialog box.

Figure 4. The list of applications on the device

After you select an application, you start it by using the Run button on the Application Verifier dialog box. You can also start the application directly from the device or by using Visual Studio. (The way you start the application isn't critical to Application Verifier.) Application Verifier monitors the application until it closes.

After you close the application, you can upload the log containing the leak analysis by clicking the Get Logs button on the main window. You should wait a few seconds after the application closes before you click the Get Logs button because the device-side monitoring code takes some time to write the data to the log file after the application closes.

Although the log is a text file, the best way to view the data is in Application Verifier's exported log viewer window, which you open by clicking the View Exported Log button on the main window. The exported log window displays the data in a tree view format, delineating the suspected error conditions from informational data such as application started and stopped notifications.

If you would like to view log files independently of Application Verifier, you can use the log viewer program, avlogview.exe. The file is in the same directory as the Application Verifier binary.

To demonstrate the capabilities of Application Verifier, you can use a simple application that leaks a handle. The test application, named Leaker3, was compiled and downloaded to the device by using Visual Studio 2005. Before compiling, the project was configured to generate both .map and listing (.cod) files for the test application. The .map file was then manually copied to the \Windows directory on the device. For historical reasons, the location of the .map files must be either in the \Windows directory or in the \Release directory. After running the Leaker3 application, the log file is imported. Figure 5 shows the error log for Leaker3.

Figure 5. The exported log from the test application Leaker3. Click the thumbnail for a larger image.

After the informational first two lines of the log file, the third line (with the red bullet) lists the first potential leak, a system handle. Expanding the third bullet shows the stack trace for that leak. Figure 6 shows the expanded view of the leak.

Figure 6. The handle leak tree expanded in the Exported Log view. Click the thumbnail for a larger image.

The stack trace shows that the handle in question is a thread handle. Further down the stack trace are the calls from the shim_hleak DLL and then two lines from the Leaker3 application. Because Application Verifier has access to the .map file for Leaker3, the routines with Leaker3 are also listed. It is the shim DLLs that provide the data for the Application Verifier tool. How they work will be discussed later in this article in the section titled Learning How Application Verifier Works.

The stack trace indicates that the offending call is at offset 30 hex from the start of wmain. The complete example code for Leaker3 is as follows.

// Leaker3.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <windows.h>
#include <commctrl.h>

DWORD WINAPI ServerThread (PVOID pArg);

int _tmain(int argc, _TCHAR* argv[])
{
    if (CreateThread (NULL, 0, ServerThread, 0, 0, 0) == 0)
        printf ("Error creating thread\r\n");
    Sleep (50);
    return 0;
}

DWORD WINAPI ServerThread (PVOID pArg) 
{
    printf ("Thread created\r\n");
    return 0;
}

The sharp-eyed reader will easily spot the problem, but perhaps more investigation is needed. An excerpt of the listing file for the program is as follows.

00000         |wmain| PROC

; 13   : {

  00000         |$LN6@wmain|
  00000    e1a0c00d     mov         r12, sp
  00004    e92d0003     stmdb       sp!, {r0, r1}
  00008    e92d5000     stmdb       sp!, {r12, lr}
  0000c    e24dd018     sub         sp, sp, #0x18
  00010         |$M31284|

; 14   :     if (CreateThread (NULL, 0, ServerThread, 0, 0, 0) == 0)

  00010    e3a03000     mov         r3, #0
  00014    e58d3004     str         r3, [sp, #4]
  00018    e3a03000     mov         r3, #0
  0001c    e58d3000     str         r3, [sp]
  00020    e3a03000     mov         r3, #0
  00024    e59f2040     ldr         r2, [pc, #0x40]
  00028    e3a01000     mov         r1, #0
  0002c    e3a00000     mov         r0, #0
00030eb000000bl          CreateThread
  00034    e58d0014     str         r0, [sp, #0x14]
  00038    e59d3014     ldr         r3, [sp, #0x14]
  0003c    e3530000     cmp         r3, #0
  00040    1a000001     bne         |$LN1@wmain|

The call at offset 30 hex is the call to CreateThread. If you take a quick look at the Help topic for CreateThread, it shows that the function returns a thread handle as its return code. Because the previous code checks only for the failure of the function and doesn't save the handle that is returned and close it, the program leaks the thread handle.

After you finish testing a particular application, be sure to uncheck the test settings for the application and remove the application from the list of applications. Failure to do so can leave data in the registry that will cause the client-side code of Application Verifier to continue to monitor an application each time it starts.

Finding Real-World Leaks

In the demonstration of the test application, Leaker3, you may have noticed that cleanup by the operating system was not considered. In truth, the application does not actually leak the handle mentioned. When the application closes, the operating system will clean up (or at least reduce the usage count of) any thread, and other kernel handles, that the application creates. However, there are cases where such a leak could be a problem. The following code example creates a thread, and leaks a handle, whenever a client connects to the TCP/IP socket.

// Wait for remote requests
while (fContinue) {

    LogIt (hWnd, TEXT("waiting..."));
    nSize = sizeof (t_btaddr);

    // Block on accept
    r_sock = accept (s_sock, (struct sockaddr *)&t_btaddr, &nSize);
    if (r_sock == INVALID_SOCKET) {
        LogIt (hWnd, TEXT(" accept failed %d"), GetLastError());
      break;
    }
    LogIt (hWnd, TEXT("sock accept..."));
    CreateThread (NULL, 0, ReceiveThread, (PVOID)r_sock, 0, NULL);
}
closesocket (s_sock);

Clearly, the previous code will leak a thread handle each time a client connects to the device. It is this type of slow, drip-by-drip leak that can cause problems in an embedded or mobile application.

This example illustrates another point regarding Application Verifier. When testing an application with Application Verifier, you should test the application as completely as possible to locate all problems. In the previous code, the handle leak would not have been found if the test had not included a client connected to the application. (For those desktop computer developers who are wondering about the use of CreateThread, it is permissible to call CreateThread directly on Windows CE. In fact, Windows CE SDKs do not define a beginthread macro.)

GDI objects present more opportunities in which an application can unknowingly leak memory. Device Context handles are a classic area of confusion. You can use either of two techniques to get a handle to a device context. If the device context already exists, an application can call GetDC. When the application doesn't need the handle anymore, it should call ReleaseDC. If the application needs a new device context, it calls CreateDC or CreateCompatibleDC. When the application is finished with the device context, it should call DeleteDC. The problem occurs when these two sets of functions, get and release versus create and destroy, are confused.

The following code example from a simple drawing application illustrates another classic GDI leak.

case WM_MOUSEMOVE:
    if (fDown) {
        HDC hdc = GetDC(hWnd);
        // Create a red pen 
        HPEN pen = CreatePen (PS_SOLID, 1, RGB (255, 0, 0));
        SelectObject (hdc, pen);

        // Draw a line from the old point to the new point
        MoveToEx (hdc, LOWORD (lParam), HIWORD (lParam), NULL);
        LineTo (hdc, ptCurr.x, ptCurr.y);

        // Delete the pen and release the DC
        DeleteObject (pen);
        ReleaseDC (hWnd, hdc);

        // Update saved point
        ptCurr.x = LOWORD (lParam);
        ptCurr.y = HIWORD (lParam);
    }
    break;

The previous code leaks a GDI object every time the user moves the stylus across the application's window. Running Application Verifier on the application creates the log file shown in Figure 7.

Figure 7. Application Verifier capturing a GDI leak. Click the thumbnail for a larger image.

According to Application Verifier, the previous code example leaked a pen object 202 times during a short test of the application. However, a quick glance at the code shows that the pen is deleted, so is Application Verifier wrong in this case? The answer is no. A closer look at the code reveals that a call to DeleteObject is made to delete the pen, but the delete call occurs while the pen is still selected in the device context. Because a selected object can't be deleted, the DeleteObject call fails. This failure isn't detected by the application because the return code is not checked (it rarely is in this situation). The solution to this leak is to select the original pen back into the device context before deleting the one you've created. To do this, you must store the pen that the initial call to SelectObject returns and then prior to the call to DeleteObject, call SelectObject passing the stored pen.

Separating the Wheat from the Chaff

One of the problems with tools such as Application Verifier is that they can return too much information. The issue is that Application Verifier cannot determine if some actions by the application being tested will or will not result in a leak. To ensure completeness, Application Verifier reports the questionable actions as leaks and leaves the determination to the developer. Most of these questionable items occur when the application developer, or even the operating system extensions, do not free up objects because they will be cleaned when the application closes. Assuming that these objects will be cleaned is typically a valid assumption, but these notifications can clutter the Application Verifier log, and, frankly, they should be cleaned. The simplest of applications, the traditional Hello, World is shown in the following example code.

// Hello.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <windows.h>
#include <commctrl.h>


int _tmain(int argc, _TCHAR* argv[])
{
    return 0;
}

Analyzing the code for the Hello, World application results in the following error log from Application Verifier, as shown in Figure 8.

Figure 8. A false error report. Click the thumbnail for a larger image.

Clearly, if this application leaks, there isn't hope for any application. So, assuming that this potential leak is not a leak, you should remember this signature and ignore it in more complex applications. To illustrate another potential leak, you can add a line to the Hello, World application to create a message box, which results in a number of additional flagged items. The example code is as follows.

// Hello.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <windows.h>
#include <commctrl.h>


int _tmain(int argc, _TCHAR* argv[])
{
    MessageBox (NULL, TEXT("Hello World"), TEXT("Title"), MB_OK);
    return 0;
}

The preceding code produces the error log shown in Figure 9.

Figure 9. False errors that are reported on any thread that creates a window. Click the thumbnail for a larger image.

When you examine the possible leaks shown in Figure 9, you will see that they are caused by the input manager in the Windows Mobile device. In fact, these items appear in any Windows Mobile 5.0 application that creates a window. Here again, if these are real errors, there isn't much you can do about them. The best you can do is to recognize these false positive indications and ignore them while looking for real leaks.

Using Application Verifier on Embedded Systems

A huge number of systems exist that are Windows CE–based but are not consumer devices. Developers of these devices often have to adapt tools and techniques developed for Windows Mobile systems, so they can be used on their embedded systems.

Fortunately for embedded systems engineers, Application Verifier was designed for embedded systems and Windows Mobile systems. In addition to connecting to remote devices by using ActiveSync, Application Verifier can connect remotely by using any of the other transport protocols supported by Platform Manager. Therefore, if ActiveSync is not available, Application Verifier can connect by means of Platform Manager's Kernel Independent Transport Layer (KITL) or directly over TCP/IP with a manual client start.

Running Application Verifier on the Device

On systems that have large screens, 800 x 600 pixels or larger, and that have either a mouse or touch screen interface, Application Verifier can be run directly on the device. The user interface for the device-side version is nearly identical to the Application Verifier interface on the desktop computer. To run Application Verifier directly on the device, you must copy the following Application Verifier files to the device.

appverif.exe
htracker.dll
shimexp.exe
shim_heap.dll
shim_hleak.dll
shim_usergdi.dll
vlog.dll
symhlp.dll

The files are in the following directory, where <cpu> is the CPU type for your device: C:\Program Files\Windows CE Platform Builder\5.00\cepb\wcetk\ddtk\<cpu>.

After you copy the files to your device, you can start Application Verifier by running the executable file appverif.exe. The only advantage to running Application Verifier on the device is that ActiveSync or some other connection to the computer is not needed.

Running Application Verifier Through Platform Builder

When using Platform Builder, you can use Application Verifier as a Shell extension. Shell is the debugger shell application that runs on systems and is best known to embedded developers by its familiar wake-up prompt, as follows.

Welcome to the Windows CE shell. Type ? for help
Windows CE>

You add Application Verifier to the shell command set by using the loadext command. First, copy the Application Verifier binary files previously listed to the Release directory of the project or directly onto the device. After the files are copied, go to the Shell window in Platform Builder and type the following commands.

loadext shim_heap.dll
loadext shim_hleak.dll
loadext shim_usergdi.dll

After you install the DLLs as extensions, these three DLLs provide command extensions to the shell without Application Verifier present. Each DLL tracks handles, GDI objects, and other memory leaks from the command line. In addition, you can change the settings for each of the extensions from the shell command prompt.

If you would prefer to use Application Verifier in its more standard way to compile the data in a log after the application closes, it can update the registry to load the shims on a per-application basis from the Shell prompt by using the following command: Appverif –m <application name> -s <shim name>.

In the preceding command, application name is the name of your application and shim name is one of the three DLLs, shim_heap.dll, shim_hleak.dll, or shim_usergdi.dll.

Application Verifier saves the log data to a file in the Release directory with the format AppVerifier_appname_ProcID.log, where appname is the application being logged, and ProcID is the process ID of the process when it ran.

Learning How Application Verifier Works

Often, when developers use tools such as Application Verifier, they wonder how the tool does its work. Application Verifier works by using an extension to the Windows CE kernel called the shim engine inside Windows CE. The shim engine enables specially written DLLs, referred to as shims, to be inserted between an application or other third-party DLLs and the operating system. The shim DLLs can then intercept calls to the operating system.

The shim engine works in conjunction with the kernel's loader. The loader is responsible for loading and resolving the dynamic linking of application and DLLs. When an .exe file or a .dll file is loaded, the loader notifies the shim engine of the load. The shim engine then looks in the registry to see if this .exe file or .dll file should be shimmed.

The shim information in the registry is listed under the following key: [HKEY_LOCAL_MACHINE]\shimengine.

Under this key, subkeys list each of the applications to be shimmed. Under each of these application subkeys are values that list the function to shim and the shim DLL to redirect that function to. Functions are listed both by name and ordinal because applications link to the function either way. Figure 10 shows the registry of a device that has shimmed an application named GDILeak.

Figure 10. Registry entries used by the shim engine. Click the thumbnail for a larger image.

In addition to the module-specific listing as shown in Figure 10, a shim can be globally inserted by using the {all} subkey. The format for the values in the {all} key is the same as the module specific keys.

If the registry indicates that a .dll or .exe file should be shimmed, the shim engine directs the imports of the module being shimmed to the shim DLL. When the module calls what it thinks is the API, the call actually goes to the shim DLL. The shim, in turn, does its work and typically passes the call on to the operating system.

Application Verifier uses the shim engine by providing a number of its own shims. When Application Verifier connects to the remote device, it downloads the shim DLLs to the device. When the user selects an .exe file or .dll file by using the Add button, Application Verifier modifies the registry of the device to list this new application and adds the values to redirect calls that allocate and free handles, GDI objects, and memory blocks to its shims. The shims track the allocations and matching frees. Any allocation that does not have a matching free is reported as a possible leak.

Conclusion

Application Verifier is a great tool for finding various types of memory and resource leaks. The ability to track not just memory leaks but GDI object and handle leaks makes it a particularly useful tool for embedded and mobile developers. The ability of the tool to run remotely on the device, or from within Platform Builder, also makes it possible for the embedded developer to use Application Verifier on deeply embedded devices that have unusual user interfaces or no interface at all.

Although quite useful, Application Verifier has some eccentricities. The false positives force you to understand each of the possible error indications and understand why each is or isn't a problem. The use of shims affects the performance of the device, and unless you are careful, the shim registry data can remain on the device after you have closed Application Verifier. Despite these issues, you will benefit by using Application Verifier to test your embedded and mobile applications. The effort it takes to test your application is more than compensated by not having to deal with leaky code on the device.