Debugging Distributed Web Applications

 

Anand Rajagopalan
Development Specialist
Premier Support for Internet Business Team
Microsoft Corporation

August 2000

Summary: Provides insight into the different tools used to debug Distributed Web Applications. Geared toward debugging the middle-tier Component Object Model (COM) components in a Web-based development environment using Internet Information Services (IIS) Active Server Pages (ASP). (25 printed pages)

Contents

Preface
Introduction
Setting Up a Debugging Environment
The Debug Process
Using the Debuggers
Debugging Common Problems
Conclusion
References

Preface

The Focus of This Article

This article provides insights into the different tools used to debug Distributed Web Applications. It is geared toward debugging the middle-tier Component Object Model (COM) components in a Web-based development environment using Microsoft® Internet Information Services (IIS) Active Server Pages (ASP). After reading this article, the user should be able to:

• Understand the IIS architecture.
• Understand the terms in-process, out-of-process, and pooled applications, and describe the advantages and disadvantages of each.
• Understand how to use Exception Monitor to monitor IIS applications and COM components.
• Understand how to use User Dump to dump a process's memory to disk and examine it.
• Understand how to use WinDBG to debug a live process.
• Understand how to use the Performance Monitor to troubleshoot a 100 percent CPU problem.
• Understand the use of Console-Based Debugger (CDB) and Microsoft® Windows NT® Symbolic/Systems Debugger (NTSD).

Target Audience

Developers, support professionals, and Web managers involved in development and maintenance of Web sites would gain a better understanding of the IIS architecture and the debugging process by reading this article.

Introduction

Internet Information Server Architecture

To understand the process of debugging a component it is essential that you first have a good grasp of the IIS architecture. The figure below will give you insight into the relationships between IIS, ASP, Internet Server Application Programming Interface (ISAPI), and components.

Figure 1. Internet Information Server architecture

Internet Services

The inetinfo.exe process contains the many Internet services (HTTP, FTP, SMTP, NNTP) provided by IIS. Inetinfo.exe receives requests from the TCP/IP subsystem and directs the requests to the respective services, which in turn interpret the requests, execute them, and return the results to the client. Because all of these services run within the same process (inetinfo.exe), they are able to share cached data such as file handles, account information, and log file data.

ISAPI

There are two types of ISAPI DLLs: ISAPI filters and ISAPI extensions.

An ISAPI filter is a DLL that runs in the inetinfo.exe process to filter data traveling to and from the server. The filter registers for notification of events. When the selected events occur, the control is transferred to the filter and you can monitor and change the data (on its way from the server to the client or vice versa). ISAPI filters can be used to provide enhanced logging of HTTP requests (for example, to track who is logging on to your server), custom encryption, custom compression, or additional authentication methods. If ISAPI filters are implemented, all the requests and responses have to go through them, which can have a great impact on the performance of the site.

An ISAPI server extension is a DLL that can be loaded and called by an HTTP server. Internet server extensions, also known as Internet server applications (ISAs), enhance the capabilities of an ISAPI-compliant server. An ISA is invoked from a browser application and provides similar functionality to a Common Gateway Interface (CGI) application, but has considerable performance gains over a CGI.

Active Server Pages

ASP is implemented as an ISAPI extension DLL. IIS and ASP have separate thread pools. IIS uses an internal thread pool called the Asynchronous Thread Queue (ATQ), also called pool threads. IIS uses ATQ for accepting the socket connection and sending static content to clients. Once the request is determined to be an ASP page, IIS sends the page request to the ASP/MTS thread pool for processing.

All of the ISAPI management and ASP code is moved into a component called the Web Application Manager (WAM). All of the information enters and exits the application service layer through the WAM.

Asynchronous Thread Queue

The ATQ provides a pool of reusable threads to process each client request. The scheduling is performed using the Completion Port Logic of the operating system.

Web Application Manager (WAM)

WAM is a COM component layer above Web services which houses the components for ISAPI DLLs and provides services like support for multiple ISAPI DLLs, crash recovery, and process isolation. WAMs are registered as MTS/COM+ packages/applications.

IIS In-Process Applications

If an IIS application is configured to run as in-process (low application protection in IIS 5.0), it runs in Web Server's (inetinfo) process space. This gives a great performance advantage because costly cross-process marshalling is avoided. The drawback is that a single malfunctioning application could bring down the whole Web Server. This would seriously decrease the stability and availability of the Web Server. All applications are run in-process in IIS 4.0 and pooled out-of-process (medium application protection) in IIS 5.0 by default.

IIS Out-of-Process Applications

Running an application out-of-process prevents a single application error from bringing down the inetinfo process. This kind of server protection is invaluable for managing complex production Web sites and hosting multiple applications. If an application malfunctions, its process is automatically terminated without affecting the inetinfo process.

Another advantage of running an application out-of-process is that you can unload the application without having to restart IIS.

An application running out-of-process gets its own ASP Worker threads. This is a great advantage because every time a processor is added to the box, the out-of-process application would get another 10 ASP Worker threads. This is the default case; changing the ProcessorThreadMax and ASPProcessorThreadMax values would affect this number. We will discuss these counters in detail a little later in this article.

The main disadvantage of out-of-process applications is that they each have their own process space, so there is overhead that affects performance.

Also, if a number of applications are configured to run out-of-process in a single server box, it can become quite resource intensive and affect the performance of the server.

IIS Pooled Out-of-Process Applications (IIS 5.0 Only)

A new option in IIS 5.0 occurs in between out-of-process and in-process. All IIS applications configured as pooled out-of-process will run under one server package/application created under MTS/COM+.

This option shares the advantage of the out-of-process option because pooled apps have their own process space, and a failure would not affect the availability of the Web server. In addition, since all the pooled applications share the same process space, it is not as resource intensive as the out-of-process option.

The disadvantage is that since all the pooled applications share the same process space, a failure in one application would bring down all the applications that share the pool.

In IIS 5.0, applications run as pooled out-of-process by default.

Figure 2. Process isolation: performance vs. robustness

Why Debugging Is Important

Debugging is a method used by software developers to find and fix problems in their code. Even after trying hard to find the bugs and fix them during unit and integration testing, some bugs remain in production versions. Even products coming out of a high-quality software life cycle that involves thorough unit and integration testing are not immune.

Debugging can be the most frustrating part of the Web application developer's job. Unlike desktop applications, which are self-contained, Web applications can be spread over several systems and frequently combine different programming languages and technologies. Although some application errors can be avoided with careful planning, others result from unexpected side effects of complex component interactions. Tracking down these problems often requires the use of a variety of debugging and development tools.

The most important thing to keep in mind while debugging is that you are debugging the data, not the code. Asking the right questions to get the right data is very important for a successful debugging process. Data can be either global or local. The most common mistake people make in debugging is to assume that the symptoms they are seeing are the actual cause for a problem.

Before diving into the process of debugging itself, it is a good idea to become familiar with certain terms that will be used frequently in this document pertaining to debugging.

• Attaching. Attaching means monitoring a live process in order to debug it.
• Checked/Debug Build. Checked/Debug Build refers to the build that has debugging information included in the process.
• Free Build. Free Build omits all debugging information so that the process is lightweight and more efficient.
• Dumping. Dumping is the process during which whenever a process throws an exception its image in memory can be written into a file on the hard disk. A dump is a raw snapshot of the process address space. It is roughly the same size as the amount of memory that the processor is using. A debugger can be used to analyze the dump files to determine what the process was doing at the time it was dumped.

Setting Up a Debugging Environment

Debug Symbols (Symbolic Debug Information)

Symbols provide a way to resolve global variables and function names in the loaded executable file. Symbols are nothing but function\API names and global variables mapped to offsets from the base DLL's loading address. During the compilation process of a component, the linker produces symbols. Having symbolic information inside the component itself (internal symbols) makes it easy to work with because there is no chance of having mismatched symbols. Typically, symbols are stripped out of components in production, reducing the size of the component considerably, decreasing the memory consumed by the component when loaded, and making it more efficient.

A DBG file is created by splitting symbols from a process image into a separate file using SPLITSYM or REBASE. The DBG file generally does not contain Code View-Style symbols; hence it cannot be used for source-level debugging.

The current versions of Microsoft Visual C++® and Microsoft Visual Basic® are capable of creating a symbol format called Program Database (PDB). Debuggers can directly use PDB files.

Apart from installing the symbols for the custom components, having symbols for Microsoft products will come in handy.

Symbol Locations

The following sites point to the symbol locations

Windows NT 4 Service Pack 6 symbols

https://www.microsoft.com/ntserver/nts/downloads/recommended/sp6/debug/default.asp

Windows NT 4 Service Pack 5 Symbols

https://www.microsoft.com/ntserver/nts/downloads/recommended/sp5/debug/default.asp

Windows NT 4 Service Pack 3 Symbols:

ftp://ftp.microsoft.com/bussys/winnt/winnt-public/fixes/usa/nt40/ussp3/i386/nt4sym3i.exe

https://support.microsoft.com/download/support/mslfiles/Nt4sym3i.exe

Windows NT Option Pack Symbols:

ftp://ftp.microsoft.com/bussys/IIS/iis-public/iis40/symbols/x86/symbols.cab

ftp://ftp.microsoft.com/bussys/IIS/iis-public/iis40/symbols/x86/install.inf

ftp://ftp.microsoft.com/bussys/IIS/iis-public/iis40/symbols/x86/install.exe

Windows NT Option QFE Update Symbols:

ftp://ftp.microsoft.com/bussys/IIS/iis-public/fixes/usa/IISUPD/SYMBOLS/x86/IISUpdis.exe

Site Server 3.0 Service Pack 1 Symbols:

ftp://ftp.microsoft.com/bussys/sitesrv/sitesrv-public/fixes/usa/siteserver3/sp1/x86/symbols/ss3sp1db.exe

Site Server 3.0 Service Pack 2 Symbols:

ftp://ftp.microsoft.com/bussys/sitesrv/sitesrv-public/fixes/usa/siteserver3/sp2/x86/Symbols/ss3sp2debug.exe

Microsoft Windows 2000® Server Symbols:

https://www.microsoft.com/WINDOWS2000/downloads/otherdownloads/symbols/default.asp

These are also on the Windows 2000 Customer Support and Diagnostic Tools CD.

Overview of Commonly Used Debuggers

CDB

CDB is a console-based debugger that runs within the command window in which it was started. It is very efficient because of its very small memory footprint. It can be used to debug Win32 applications on Windows NT.

NTSD

NTSD is similar to CDB, except that it launches a new command window and runs in the new window instead of the one in which it was started.

WinDBG

WinDBG is a popular GUI lightweight debugger. It can be used to debug Win32 applications on Windows NT, Windows 95, and Windows 98. It is currently included on the Windows 2000 Customer Diagnostic and Support Tools CD. It can perform source-level debugging, can read dump files, and has excellent support for symbols. Because it is a GUI debugger it is a bit slower than CDB and NTSD.

IIS Exception Monitor 6.1

IIS Exception Monitor is a lightweight and extremely easy to use debugger. Using this debugger you can isolate down to the DLL that caused the problem. It provides a snapshot text file consisting of the process image in memory. It uses CDB to gather the information. Sometimes the information it gathers might not be enough. The IIS Exception Monitor requires Windows Script Host to run. You can download the tool from Troubleshooting with the IIS Exception Monitor.

Exception Monitor 7.x

The newest version of IIS Exception Monitor is a part of the DBGPLUS suite, which contains all the debugging tools discussed here. The word "IIS" has been removed from its name because it is no longer specific to debugging IIS problems; it can also be used to debug any process. It uses WinDBG to gather information about the process it is attached to. Although its functionality is very similar to its predecessor, the user interface has been thoroughly revamped and new features have been added. It is included with the Windows 2000 Customer Diagnostic and Support Tools CD. You can also download the tool from Windows 2000 Customer Support Diagnostics.

Note The examples in this article are based on IIS Exception Monitor 6.1. However, with minor changes they can also be applied to Exception Monitor 7.x.

Developer Studio

Developer Studio is an integrated developer environment, which includes a very powerful debugger. It is mostly used during the development phase. It is great for source-level debugging. It leaves a very large memory footprint. Installing Developer Studio on a computer modifies a lot of the system files; therefore, it is not commonly used in a production environment. Developers can write their own add-ins to customize the debugger according to their requirements.

The Debug Process

Verifying Symbols

The first step in the debug process is to verify that the symbols are installed properly. The CHECKSYM tool can be used for this. For example, to verify that symbols installed for all the DLLs running in a process are valid, you can use the following command:

checksym -p processname.exe -r -v -y c:\symbols

Wildcards (*.*) can be used to specify all files in a directory. The -v switch instructs the tool to verify the symbols for this component and the -y switch specifies the path to the symbol files.

For more information about the switches and options that can be used with CHECKSYM, type the following command:

checksym -??

Choosing a Debugger

Debuggers vary in their capabilities, so it's important to have a good grasp of the task before deciding to use a specific debugging tool. Debugging scenarios can be broadly classified under three headings: local debugging, remote debugging, and debugging the user dump.

Local Debugging

When you are able to reproduce a problem locally, it's a good idea to use local debugging; because it provides full control of the environment, you may be able to solve the problem more quickly. The choice of the debugger and its setup are also relatively easy in this scenario. The disadvantage of local debugging is that it is sometimes difficult to accurately duplicate or reproduce ("repro" in support professionals' language) a problem locally.

Remote Debugging

When the debugger and the process being debugged do not reside on the same machine, it is called remote debugging. Remote debugging can be done either by having a debugger attached to a remote process through an agent or running the debugger on the remote machine and sharing the command line of the debugger on the local machine. In this case the agent runs on the remote system, and the debugger runs on the local system; the agent attaches to the process, reads the process memory, and passes it over to the local debugger. The advantage of remote debugging with an agent is that it is simple and fast to set up. The disadvantage is that it is prone to network problems that sometimes cause the agent on the remote machine to crash. Since the symbols are installed only on the local machine, they may not match those of the remote machine. Consequently, remote debugging can be quite slow.

Remote debugging with a shared command line is accomplished by using the remote.exe tool, which is distributed with the Windows 2000 Customer Diagnostic and Support Tools CD. The remote.exe tool can run both as a server and a client. Named pipes are used to connect the server and the client. WSRemote.exe is a socket version/variation of the remote.exe tool. Remote.exe allows the sharing of the STDIN (standard input) and STDOUT (standard output). On the remote machine you can run the server instance of the remote with the following command:

remote /s application pipename

On the local machine you can run the remote.exe as a client by using the following command:

remote /c machinename pipename

This setup is very reliable and tolerant of network failures.

Debugging the User Dump

User dump creates a snapshot of the process memory to the hard disk when an exception occurs. Windows NT has a tool called DrWatson that can create a user dump when an exception occurs in the system. There is another debugging tool called User Mode Process Dump. You can configure the user dump tool to monitor a particular process for exceptions. The user dump thus created can be later viewed in a debugger like WinDbg and analyzed at leisure, instead of debugging a live process.

Using the Debuggers

Using NTSD

The NTSD debuggers require the user to specify a process to connect to when starting the debugger. Using TLIST or PVIEWER, you can get the process ID for an existing process and type NTSD -p pid to debug that process. The NTSD command line uses the following syntax

NTSD [options] imagefile

where imagefile is the name of the image to be debugged and options is one of the following:

Table 1. NTSD imagefile options

Option	Description
-2	Opens a new window for debugging character mode applications.
-d	Redirects output to the debugging terminal.
-g	Causes execution past the first breakpoint automatically.
-G	Causes NTSD to exit immediately when the child terminates.
-o	Enables debugging of multiple processes. The default is for one process directly spawned by the debugger.
-p	Specifies debugging of the process identified by process ID.
-v	Produces verbose output.

For example, let's assume that the process ID (PID) for the inetinfo.exe process is 104. Type the following to attach the NTSD debugger to the inetinfo process (IIS).

NTSD –p 104

NTSD may also be used to start a new process for debugging. For example, NTSD notepad.exe will start a new notepad.exe process and attach to it.

Once attached to a process, you can use various commands to view the stack, set breakpoints, dump memory, and so forth.

Table 2. Commonly used commands

Command	Meaning
~	Show a list of all the threads
KB	Show the stack trace for the current thread
~*KB	Show the stack trace for all the threads
R	Show the register output for the current frame
U	Un-assemble the code and show the procedure name and offset
D[type][<range>]	Dump memory
BP[#] <address>	Set break point
BC[<bp>]	Clear break point(s)
BD[<bp>]	Disable break point(s)
BE[<bp>]	Enable break point(s)
BL[<bp>]	List break point(s)

Using WinDBG

1. Start WinDBG.

2. On the Debug menu, click Attach to a Process, select the process you want to debug from the list, and then click Select.

   WinDBG lists the DLLs related to the inetinfo process in the command window and also shows if symbols were loaded for a particular DLL. At this point the inetinfo.exe application (IIS) that is being debugged is stopped. To start it again, type g in the command window.

3. In the command window, type ~*kb.

   This lists all of the stacks for all of the threads in the process.

4. To change the symbols path, on the View menu, click Options. In the Windows Debugger Options dialog box, click the Symbols tab, specify the path to the symbols file in the Debug Symbol Search Path text box, and then click OK.

5. In the command window, type kb.

6. This lists the stack for the current thread.

   Another way to view the stack of the current thread is to view the Call Stack window by clicking Call Stack on the View menu.

7. In the command window, type lm.

   This lists all of the modules (DLLs) in the process with their names and their base loading addresses.

8. To quit, type q in the command window.

Using IIS Exception Monitor

In the following example, IIS Exception Monitor is used to attach to the inetinfo (IIS) process and debug it.

1. Click Start, point to Programs, point to IIS Exception Monitor, and then click IIS Exception Monitor.

2. In the Welcome dialog box, click Next.

3. In the Check Symbols dialog box, if you choose yes, the wizard will look for files used by IIS and determine if there is a matching file in the \%SYSTEMROOT%\Symbols\dll directory. Alternately you can select Use Alternate Symbol Path for Exception Monitoring to specify a different symbol path.

4. Select Yes. Verify the IIS Symbols that are installed, and then click Next.

5. In the Verify Symbols dialog box, examine which DLLs have mismatched symbols on your system. You can also select the Determine which symbol packages can be installed from Microsoft's Internet site check box to download symbols from the Microsoft Internet site. This will take you to the download page. For our example, let's assume that you have the symbols installed correctly. Click Next.

   Note The Verify Symbols dialog box uses CHECKSYM.EXE to obtain its symbolic information. CHECKSYM.EXE is located in the C:\ixcptmon\bin directory.

6. You can find three options in the Process Options dialog box. The In-Process option refers to the IIS applications running in the inetinfo process (the option "low" in IIS 5.0). The Out-Of-Process option refers to the IIS applications that are configured to run in their own/pooled process space (the options "High" and "Medium" in IIS 5.0) and components running under MTS/COM+ as a server package. The Other Process option refers to essentially any other process running on the system. If you register a component under MTS as a "Server application/package", it runs under the dllhost.exe process (mtx.exe in Windows NT 4.0). There will be several instances of dllhost.exe running for each of the applications. If you have to debug a particular component of interest, first you have to find the instance of the dllhost.exe under which it is running and then attach to that instance. A useful feature of the Out-Of-Process option is that it lists all the components registered with MTS/COM+ by their application/package name, thereby saving you the trouble of finding the correct instance of dllhost.exe and then attaching to it.

   For our example, select the In-Process option, and then click Next.

7. This brings us to the Session Options dialog box. If you choose Automatic, a log file is created whenever an exception occurs. If you choose Manual, you are prompted for a Port, Username, and Password so that a support engineer can interactively troubleshoot your problem. This option should be used primarily if you are currently working on an issue with technical support. If Manual is selected, then once an exception occurs the Web server will be suspended and will not be able to accept any Web requests until it is either restarted or the monitoring session has been canceled. Select the Enable Recursive Mode check box if you want the IIS Exception Monitor to continue to monitor even after it has written a log and restarted the process. Select the Notify Admin check box if you want to receive a popup message when the IIS Exception Monitor encounters an exception or if you have CDO and the SMTP service installed. Specify a fully qualified e-mail address to receive an e-mail message with the .dbl file as an attachment when an exception is detected. By selecting the Schedule this session check box, you can schedule an Exception Monitor session to take place at a specified time or at system startup. This requires the Task Scheduler to be installed; if Task Scheduler is not installed, this option will be unavailable.

   For our example, select the Automatic option and then click Next.

8. In the Start Monitoring dialog box, click Run This Monitoring Session, and then click Next.

9. The Session Status dialog box lists all the processes that were monitored previously or are being monitored currently. Click Finish.

   You will see a command prompt window that has "IIS_<Process ID>" on the title bar. This monitors the inetinfo.exe process. Whenever an exception occurs, the IIS Exception monitor creates the log file (.dbl), restarts the IIS Service, and closes the command prompt window.

   The log file can be found in the \IXCPTMON\Logs directory.

Using the Log File to Identify the Problem

The log reader program Readlogs.exe is installed when the IIS Exception Monitor is installed. You can use Readlogs to open the just-generated .dbl log file.

Figure 3. Readlog dialog box

When Readlogs opens the log file, it displays the stack of the thread that caused the exception. If the symbols are installed properly, Readlogs resolves the function names and displays them under the Function column as shown in the preceding figure. The function name is separated from the DLL name by the "!" symbol. For example, in the preceding figure the function OutputDebugStringA resides inside the KERNEL32 DLL.

You can always view the faulting thread stack by clicking the Fault Stack button in the analyzer. To scroll through the stack list you can use the << and >> buttons. You can also choose to execute any of the listed commands in the Choose a Command drop-down list. If you do not want to see Microsoft files, you have an option to hide them. This helps to quickly identify the involvement of any third-party DLLs involved in the process. Apart from listing the stack, the log file also contains information about the list of loaded DLLs or modules, any locks and their owning thread IDs, and errors that were detected.

By clicking the DLLs button, you can list all the DLLs that were loaded in the process.

You can use the Errors button to list all the errors including the warnings that occurred when the process was running. You can also customize the IIS Exception Monitor to trap certain errors.

By clicking the Locks button, you can display a list of locks and their owning thread IDs. Whenever a thread wants to access a resource exclusively, the thread will request that a lock be placed on the resource. When two or more threads lock a particular resource and wait for the other lock to be relinquished, a phenomenon called "deadlock" occurs. You can examine the list of locks and their owning threads to identify potential sources for a deadlock problem.

You can list the last 20 lines in the log just before the process broke down by clicking the P.T.C. (Prior to Crash) button.

You can also paste a copy of all the information to the clipboard in a report format by clicking the Report button.

Clicking the Sys Info button displays some useful information about the system that you can use for debugging.

You can customize the read logs to suit your needs by using the Config button.

Using the User Mode Process Dump Tool

The User Mode Process Dump tool attaches to a particular process and dumps the process memory to the hard disk whenever an exception occurs. First you have to set up the User Mode Process Dump service by running setup.exe in C:\Program Files\Debuggers\bin\userdump\.The User Mode Process Dump setup program is installed as a part of the DBGPLUS suite installation.

1. In the Welcome dialog box, click Next.

2. In the Confirmation dialog box, click Finish.

3. Setup installs a control panel icon named Process Dump that can be used to configure the User Dump service.

4. Double-click the Process Dump icon to bring up the User Mode Process Dump Configuration dialog box.

5. Click New and enter the name of the application that you want the user dump tool to monitor.

   You can customize the tool to trap certain exceptions only by creating rules. Click Rules.

If an exception occurs in the process that the tool is monitoring, it will dump the process memory to the disk. Unlike the IIS Exception Monitor, the User Mode Process Dump tool does not restart the IIS Service after dumping the process.

The user dump thus created can be opened in the WinDBG debugger and analyzed to identify the cause for the crash:

1. Start WinDBG. On the File menu, click Open Crash Dump and select the Crash Dump file.
2. In the command window, type g (go) to start debugging the process.
3. To view the faulting stack, type kv.

Which Process to Debug?

Identifying which process to attach to and debug can be difficult. As discussed earlier in the "Internet Information Server Architecture," there are different scenarios under which components and IIS applications are used. The following table shows which process to attach to based on component type and IIS configuration.

Table 3. Component types and IIS configurations

Component type	In-process IIS application	Out-of-process IIS application	Pooled out-of-process IIS application (IIS 5.0 only)
Component in a library package/application or Component not under MTS/COM+	Attach to inetinfo process	Attach to the MTX/DLLHOST instance of the package/application under which the IIS application resides
Component in a server package/application	Attach to the MTX/DLLHOST instance of the package/application under which the component resides

Debugging gets tricky when you are trying to debug a component or an IIS application that is under MTS/COM+ and installed as a server package/application. There might be several server packages/applications running in a single system, so each of them will have their own MTX.exe or DLLHost.exe. Which one is yours? The first task at hand is to find the PID (process identifier) of the process that has loaded the component that you wish to debug. You can use IIS Exception Monitor to get this information, choosing the Out-Of-Process option in the Process Options page.

This will take you to the Select Out-Of-Process Application page, where you can see processes listed by their application/package names, making it easy to identify which PID to attach to.

Debugging Common Problems

The most common problems that occur in Distributed Web Applications are:

• ASP 0115—access violation
• ASP server too busy error
• 100 percent CPU problem

In this section, we will analyze the possible causes for these problems and discuss how to debug and identify the line of code that is causing each problem.

ASP 0115—Access Violation

Access violations are primarily due to heap and stack corruption.

Access violations occur if you do something like allocating a block of memory and trying to read/write beyond that block.

For example:

char* pszTemp =  (char*) malloc(strlen("BOMB THIS"));
strcpy(pszTemp,"BOMB THIS");

In this example the user allocates 9 bytes of memory but fails to include the NULL terminator ("\0"). When trying to copy the string to the memory location, an error occurs because the function is trying to access more than the allocated segment of memory.

Another scenario would be trying to access an invalid location in memory.

For example:

FunctionA()
{
   m_pAV = NULL;
   FunctionB(m_pAV);   
}
FunctionB(CAccessViolate* pAV)
{
   pAV->BombThis();
}

In this example FunctionB uses the pAV pointer passed to it without checking its validity, causing an access violation.

Most access violations occur when an ASP page is trying to access a component, and the component is trying to do something wrong. For simplicity, let's assume that you have a component installed as a library package. When you try to access an ASP page that is using this component, the ASP page may return the following

error 'ASP 0115'
Unexpected error
/<Web Server Name>/<ASP file name>.asp
A trappable error occurred in an external object. The script cannot continue running.

The error message states that there was an error in a particular ASP page and it was caused by an exception that occurred in an external object. This means that you are trying to access some component in your ASP page that is failing. You can try to isolate the problem further by removing all references to any components in the ASP page and then adding them back one at a time until you find the culprit.

If the isolation process is tedious or you do not get a clear error message stating the ASP page name, you can use WinDBG, User Mode Process Dump, or IIS Exception Monitor to attach to inetinfo and start debugging it.

Finding the Line of Code That Caused the Access Violation

Let's assume that you have used the User Mode Process Dump tool to attach to the inetinfo process and have created a dump file. Let's examine the dump file in WinDBG and try to figure out what happened.

Open the crash dump in WinDBG. On the View menu, click Call Stack.

Figure 4. Windows Debugger dialog box showing the stack

Even though the topmost function in the stack in the preceding figure is a function inside the KERNEL32 DLL, it doesn't necessarily mean that the misbehaving code is inside that function. Most of the time this is not the case. The next step is to identify if any components other than the standard Windows system components are listed, and determine what they were doing at the time of crash. If you read the stack from the bottom up in the preceding figure, you can see that lines 8 and 7 indicate that ASP used the Microsoft VBScript engine to launch the component. Lines 6 and 5 indicate that the VBScript engine was using the IDispatch interface methods to invoke a method inside the component (CRASHATLAPP). Since IDispatch interface was used, it is natural for the OLEAUT32.dll to come into the picture. Lines 4 and 3 indicate this. Line 2 in the stack shows that the BlowOff method was invoked, which in turn invoked the OutputDebugString API inside the KERNEL32 dll. It would be a safe assumption that the standard Windows components KERNEL32 dll, the OLEAUT32.dll, and the VBScript engine are innocent (from the perspective of problem isolation). So we are left to deal with the only non-Windows component listed in the stack, the CRASHATLAPP.

The BlowOff method inside the CRASHATLAPP has made a call to the OutputDebugString API. We can examine the values of the parameters that BlowOff method passed to the OutputDebugString by typing kb in the command window.

The BlowOff method had passed NULL as the parameter to the OutputDebugString method inside the KERNEL32 DLL. It might be good idea to see what is being done inside the BlowOff method that caused it to pass a NULL value. If you have the source code for the CRASHATLAPP component handy, you can double click on line 2 in the stack.

Figure 5. Windows Debugger dialog box showing the stack

This brings up the source code for the BlowOff method.

Figure 6. Windows Debugger dialog box showing BlowOff method source code

The source code clearly illustrates that a NULL value was passed as the parameter to OutputDebugString.

Although you won't come across such mistakes in production code, the example was provided with an intention to give you an idea of the steps involved in debugging and identifying the cause of an access violation error.

ASP Server Too Busy Error

An ASP Server Too Busy error is generally due to the ASP request queue filling up. Significant amounts of queuing may occur when an IIS Web server experiences a very high load or is waiting on a blocking thread. This normally occurs when a component on which the IIS Web server is waiting is invoked at a rate greater than the component can normally satisfy. All the incoming requests are placed in the ASP Request queue and processed in the order in which they were received. If the blocking lasts for a very short period of time, the queuing mechanism will eventually get rid of the problem and start servicing the requests in a timely fashion. But if the blocking problem lasts for a longer period of time, the number of requests that are queued increases and eventually peaks at the ASPRequestQueueMax metadata setting, which is by default 3000. If the queue size is under 3000, the user with the 2999^th request will see an hourglass as the request is in the queue and waits for the all the other requests to be filled. Even at this point, if the queue clears up quickly the request will be serviced. On the other hand, if the queue size reaches the ASPRequestQueueMax value and a new request comes in, the IIS Server returns the ASP Server Too Busy error.

ASPProcessorThreadMax and ASPRequestQueueMax Settings

The ASPProcessorThreadMax and ASPRequestQueueMax metadata settings affect the performance of the site greatly. The ASPProcessorThreadMax parameter specifies the maximum number of threads that a processor can serve. The ASPRequestQueueMax parameter specifies the maximum size of the request queue. These two settings can be changed by the following commands:

adsutil.vbs set w3svc/AspProcessorThreadMax NewValue

adsutil.vbs set w3svc/AspRequestQueueMax NewValue

For well-written scripts, the ASPProcessorThreadMax value should be low. The primary goal of tuning the ASPProcessorThreadMax is to get processor utilization above 50 percent during peak load.

The physical configuration of each site varies widely, so it is difficult to predict an optimum value for these parameters. You should use Performance Monitor to monitor critical counters to arrive at a decision regarding these values. The statistics using the Performance Monitor should include:

Processor time : %Processor time

ASP: Requests per second, Requests rejected, Requests queued

You can monitor these values in a production site during peak load. Typically in a high-volume site you can observe that the number of requests queued goes up and down. If the processor utilization is low and the total queue length never goes up, it means that you have a site with a capacity more than you actually need. If the total queue length is going up and down and processors are running below 50 percent, it is an indication that some of the requests are blocking and you can benefit from increasing the number of threads. If after increasing the threads you observe that the number of requests queued is worse than before and processor utilization is also getting low, then you have serious blocking problems. A typical blocking problem might be caused by an SQL query made to the database that is taking very long to execute. Increasing the number of threads allows more requests to come in for the page that executes the query; more pressure is put on the database, which makes the query response time even worse. To arrive at a good number for ASPRequestQueueMax, select a maximum acceptable response time, calculate the size of a queue that would clear within that time, and set the value of ASPRequestQueueMax just below that size.

A Debugging Example

Having dealt with the ASPProcessorThreadMax and ASPRequestQueueMax parameters in detail, let's try to simulate the ASP Server Too Busy error and try to debug which component's thread is blocking and why. Assume that you have an ASP page that is making a call to a component that is blocking.

To simulate the problem, let's set the ASPRequestQueueMax parameter to 20 by using the following command:

cscript c:\inetpub\AdminScripts\adsutil.vbs SET w3svc/ASPRequestQueueMax 20

To start the Performance Monitor:

1. Click Start, point to Programs, point to Administrative Tools, and then click Performance Monitor. Right-click the chart and then click Add Counters.
2. In the Add Counters dialog box, select Active Server Pages from the Performance Object list box.
3. From the Select counters from list, select Requests Executing and Requests Queued. Click Add, and then click Close.

Assume that you make a call to a component in your ASP code that is blocking the request.

After adding these counters, watch the values of the counter. You can observe that when the requests queued counter reaches the ASPRequestQueueMax value (20) the server returns the ASP Server Too Busy error.

Figure 7. Performance Monitor showing the Requests Queued counter value

To debug the inetinfo process to determine which thread is blocking and why, start the WinDBG and attach to the inetinfo process. The most common cause for a thread to block is a deadlock. You should try to identify the critical section that has a lock count greater than 0 and its owning thread ID. This thread is the culprit. Once you have the thread ID you can debug the thread stack to find out which specific method is blocking. To accomplish this you can type the following command in the command window:

!locks

This should list the threads that have a lock count greater than zero. The output will be something like this:

CritSec ?g_cs@@3U_RTL_CRITICAL_SECTION@@A+0 at 100355A0
LockCount          24
RecursionCount     1
OwningThread       6e8
EntryCount         18
ContentionCount    18
*** Locked

Look for the critical section that has lock count greater than zero and note the owning thread ID.

Sometimes the !locks command does not provide the required information, usually because symbols for the ntdll.dll are not lined up correctly. In that case, you can use the following crude but effective method.

In the command window, type ~*kb to display the stacks of all the threads. Look for the thread stack that contains the NTDLL!RtlEnterCriticalSection. Once you have identified the thread that has the entry, note the value of param1 in the line that contains NTDLL!RtlEnterCriticalSection. This value is the critical section object passed to the EnterCriticalSection API. Using that value, type the following command in the command window:

!Critsec 100355A0

Using either of the above methods, you should be able to identify the thread ID that has a lock count greater than 0. Convert the thread ID from hexadecimal to decimal.

Let's examine the thread and identify the method that is blocking the thread. On the Debug menu, click Threads.

In the Threads dialog box, click the thread ID you identified in your previous step, and click Select.

On the View menu, click Call Stack. This displays the stack of the thread. Start debugging the stack as we did in the previous (Access Violation) example.

100% CPU Problem

The 100% CPU problem occurs when one thread consumes all the processor time. One possible cause is a program loop that monopolizes CPU resources. Performance Monitor is the perfect tool for troubleshooting 100% CPU utilization problems. First we need to create a Performance Monitor log.

Preparing to Log

To start the Performance Monitor:

1. Click Start, point to Programs, point to Administrative Tools, and then click Performance Monitor.

2. Right-click Counter Logs, and then click New Log Settings. Give the log a name of your choice and click OK.

3. On the General tab, click Add to add counters to the log.

   Add the following counters:

   • ASP: All counters
   • Memory: All counters
   • Process: All counters and all instances
   • Processor: All counters and all instances
   • Thread: All counters and all instances
   • Web service: All counters and all instances

4. After adding all the counters, change the Sampling Interval to every 1 second, and then click OK.

Now you are all set to start logging the data. Right-click the log name and then click Start. Make sure that the log captures all the data during the time when processor utilization shoots up to 100 percent. Start WinDBG and attach it to the inetinfo process. Once the processor utilization value comes back to normal, you can stop logging by right-clicking the log name and clicking Stop.

Examining the Log

Now that we have created the log, we can start examining it closely to identify the process that was using up most of the CPU time, the thread inside that process that was eating the CPU time, and why it was doing so.

The first step is to identify which process was using up most of the CPU time. To open a log file, click the System Monitor folder under the console root. In the detail pane, right-click the chart, and then click Properties. In System Monitor Properties, click the Source tab. Select Log file and browse to select the log file you just created.

Right-click the chart, and then click Add Counters. In the Add Counters dialog box, select Process from the Performance object list. Select Select counters from list, and select %Processor Time from the list box. Select All instances. Click Add, and then click Close.

This gives us a list of all the processes and their CPU utilization. Now you need to find out which process caused the 100% CPU problem.

Make sure that the Highlight tool button (with a bulb-like icon) is clicked. Now you can scroll through the counters and delete all except the one that is nearest to the 100% mark. Make sure that you select only the counters that are relevant to you; for instance, if you are debugging the 100% CPU utilization, you should probably be looking at inetinfo or other out-of-process COM applications/packages (MTX.EXE/DLLHOST.EXE) or IIS applications.

Once you have isolated the problem at the process level, you can concentrate on the threads that were active inside that process during the 100% CPU problem and try to isolate the thread that caused the problem. For that purpose, let's add more counters to display the CPU utilization of all the threads inside that process. For this example let's add counters for all the threads under the inetinfo process. Right-click the chart and click Add Counters. In the Add Counters dialog box, select Thread from the Performance object list. Select Select counters from list, and select %Processor Time from the list box. Select Select instances from list, and select all the instances that begin with inetinfo/*. Click Add, and then click Close.

This will display the CPU utilization for all the threads inside the inetinfo process in the graph.

Now you can scroll through the counters and remove all the counters except the one that is nearest to the 100% mark. This will bring us down to the exact thread that was causing the problem.

Figure 8. Performance Monitor showing CPU utilization counters

We are left with a single thread with the instance ID 34 that was utilizing the maximum CPU time. Now we have to find the thread ID of this thread and look it up in WinDBG and debug it.

To find the thread ID, right-click the chart and click Add Counters. In the Add Counters dialog box select Thread from the Performance object list. Select Select counters from list, and select ID Thread from the list box. Select Select instances from list, and select the instance inetinfo/34. Click Add, and then click Close.

This gives us the thread ID of the thread that is causing the problem. Select the ID Thread counter. Note the Last, Maximum, Minimum, and Average values of this counter. Typically all four values are equal. Note the value (764 in this example) and switch to the WinDBG application.

Figure 9. Performance dialog box showing ID Thread counter

On the WinDBG View menu, click Threads. This displays a list of threads that were running inside the inetinfo process when the 100% CPU problem was encountered. Look for the thread ID number you had noted from the Performance Monitor. If you find it in the list, click it and then click Select. Click OK.

On the View menu, click Call Stack. This displays the stack of the thread that is causing the problem. The stack isolates the problem method. Once we find the method, we can go back to the source code and see which line of code caused the problem.

Conclusion

You can use the IIS Exception Monitor, WinDBG, the User Mode Process Dump tool, and Performance Monitor to effectively solve the most common problems in Distributed Web Applications. These tools do not necessarily have all the answers and solutions to a problem, but they can point you in the direction that will eventually lead to a solution.

References

HOWTO: Troubleshoot a High-CPU-Utilization "Hang" in Internet Information Services

INFO: Troubleshooting Exceptions in Internet Server Products

Additional Reading

Web Sites

Problem Isolation in Windows DNA Solutions

Knowledge Base Articles

Q229814, Configuring IIS to Handle Heavy Usage

Q150934, How to Create a Performance Monitor Log for NT Troubleshooting

Acknowledgements

I would like to thank Aaron Barth, who is the Tech Lead in the Systems Integration Engineering group. I think of Aaron as a debugging guru; he has taught me debugging and was also a tremendous help in writing this paper.