New information has been added to this article since publication.
Refer to the Editor's Update below.
Windows Server 2003
Discover Improved System Info, New Kernel, Debugging, Security, and UI APIs
Matt Pietrek
Code download available at:WindowsServer2003.exe(1,024 KB)
This article assumes you're familiar with Windows, .NET, and C++
Level of Difficulty123
SUMMARY
There's a lot to say about Windows Server 2003. First of all, it's the first operating system with built-in .NET Framework support, and it's the first 64-bit OS from Microsoft. But wait, there's more! There are lots of new features and APIs in this version as well. For instance, Windows Server 2003 features Hot Add Memory and a number of other arcane new tidbits. There are new APIs for handling threads, directories, and files, and new features like the low fragmentation heap for managing memory and system information. There's vectored exception handling and new UI APIs as well.
OS internals expert Matt Pietrek takes a look at the additions he finds most interesting and useful so you'll have a good place to start when you dive into Windows Server 2003.
Contents
New KERNEL APIs
Processes, Threads, and Fibers, Oh My!
Vectored Exception Handling
Directories and Files
Memory and System Information
Debug APIs
Side-by-side Execution
User Interface Additions
Newly Documented Interfaces
New DEBUGHLP Additions
Windows Error Reporting
ADVAPI32
OLE Gets Old
Clustering
Real Time Client API
Conclusion
I really am too young to be saying this, but I remember the good old days when developers eagerly awaited the release of a new version of Windows®. What new goodies might it contain, and what cool things could you do with them? With the advent of the Microsoft® .NET Framework, it's become fashionable in some circles to pretend that Windows is static and not worthy of discussion. Some people seem to think it doesn't matter which version of Windows you're running underneath.
As you might guess, I'm not in that group. While I'm as big a fan of Microsoft .NET as the next person, I still eagerly scan header files and compare the exports of system DLLs from each new version of Windows. I want to know what the fine folks on the Microsoft Windows team have been up to.
The latest and greatest operating system from Microsoft is Windows Server™ 2003, which is in its release candidate phase as I write this. In this article, I'll examine what new goodies are available in Windows Server 2003 for application-level programmers. However, before diving into the new APIs, there's some background information that helps put all this in context.
Let's first establish what "new" means. Windows Server 2003 is intended to replace the Windows 2000 Server family (Server, Advanced Server, and Datacenter Server.) As such, many of the OS features that originally appeared in Windows XP technically fall into the "new" category for the purposes of this article.
Actually, the majority of the new APIs (relative to Windows 2000) first appeared in Windows XP, rather than in Windows Server 2003. There's a good reason for that. Many moons ago, the successor to Windows 2000 was codenamed Whistler, and its betas included both workstation and server versions. In 2001, Microsoft decided that the server version needed to bake a bit longer, and so released the consumer and workstation versions as Windows XP. The intention was that the server versions would be released shortly thereafter. As you now know, that delay turned into a year and a half. Some of this time can be accounted for by the Microsoft Trustworthy Computing Initiative, wherein developers at Microsoft stopped new development in order to look for security issues in their code.
The key point is that up to a certain date, both the workstation and server versions of Whistler were essentially feature complete and used the same code base. The time between the release of Windows XP and Windows Server 2003 was primarily spent making it as robust as possible. Thus, it's not at all surprising that many of the new Windows Server 2003 features and APIs are also available in Windows XP.
The version numbers used in the Platform SDK header files give evidence of how little the public interface to the OS (that is, the APIs) has changed since Windows XP. To enable Windows XP (and later) APIs, you define _WIN32_WINNT to 0x0501 (that is, Windows XP is internally thought of as Windows 5.01.) For Windows Server 2003, the required #define value for _WIN32_WINNT has only been bumped to 0x0502. You'll see _WIN32_WINNT #defines used in the sample programs later.
Windows Server 2003 may hold the record for the most number of name changes during its incubation. It started out codenamed Whistler and then became "Windows 2002 Server." Subsequently, the .NET initiative permeated all things Microsoft, and it was rechristened "Windows .NET Server." After another delay, the name changed to "Windows .NET Server 2003." Finally, wiser heads prevailed, and the name became "Windows Server 2003." It wouldn't surprise me if I missed a name change in there somewhere.
[Editor's Update - 2/15/2004: Windows Server 2003 only includes the .NET Framework 1.1, although version 1.0 can be installed on it manually.] Since .NET has been covered so extensively in MSDN® Magazine and elsewhere, I won't count the .NET Framework as a new API for the purposes of this article.
Besides being the first system with built-in support for the .NET Framework, Windows Server 2003 also holds the distinction of being the first Microsoft 64-bit server OS. Windows XP Professional has a 64-bit version, but the demand for 64-bit Itanium-based workstations hasn't been that great so far. With a 64-bit server OS finally available, companies with huge databases will probably start the transition to 64-bit Windows. A 64-bit version of SQL Server™ is slated to ship alongside the 64-bit versions of Windows Server 2003.
Windows Server 2003 will ship in six configurations initially. For the Intel x86 CPU, the low end is the Web Edition server for basic Web Services. Next up the ladder is the Standard Edition, intended as a departmental server. Up a notch from that is the Enterprise Edition, targeted at medium to large enterprises. Finally, for massive database systems operating on up to 32 processors there's the Datacenter Edition. The remaining two configurations are the 64-bit Itanium versions of both Enterprise Edition and Datacenter Edition.
As an interesting aside, the 64-bit versions of Windows Server 2003 won't include the .NET Framework. Apparently, the 64-bit version of .NET isn't yet ready to ship, so it will be included in a later version.
There are lots of new features in Windows Server 2003 that I won't go into any detail on, but are still worth mentioning, just for their coolness. For example, Internet Information Services (IIS) is now at version 6.0. It has had a major architectural change and big performance gains are attained by doing more work with a kernel-mode listener. If desired, you can run it in the three IIS 5.0 protection levels as well.
Other exotic new features in Windows Server 2003 include, but are not limited to, Volume Shadow Copy, Hot Add Memory, and Non-Uniform Memory Access (NUMA) support.
The Volume Shadow Copy service provides a way to do complete backups, even on files that are open at the time of the backup. While backups aren't my favorite topic, I think it's pretty cool how this functionality was implemented transparently using drivers. Volume Shadow Copy, as well as many other additions, are covered in the article "Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS," by Mark Russinovich and David Solomon in the December 2001 issue.
Hot Add Memory is one of those totally exotic features that make you wonder what those wacky hardware guys will dream up next. It's the ability to add RAM to a system while it's running. The OS automatically detects and starts using the RAM when you plug it in. For this to work, however, you need to be running a system designed to support it. In the Microsoft documentation, you'll find the Surgeon General's warning to not just blindly add RAM to any old system while it's turned on—bad things will happen.
NUMA is a high-end technology used in enterprise-class multiprocessor systems. The idea is that memory and processors are grouped together into cells. A processor accessing memory within its cell can access that memory faster than memory in another cell. The NUMA support in Windows causes the scheduler to attempt to keep related processes running in the same cell.
Before jumping into the blood and guts of new APIs in Windows Server 2003, there's one obligatory disclaimer I need to add. What follows is subjective and not comprehensive. I went through hundreds of header files and beta documentation pages and culled what I thought was most interesting. I had to make a lot of tough choices as to what to include and what to leave out. Generally, for me to select a feature to include here, it had to be a user-mode API that's not completely esoteric or tied to one particular program. There's lots of new functionality at the device driver level, but that's outside the scope of this article.
New KERNEL APIs
I'll begin the tour with kernel-level functionality. You might think that this means KERNEL32 (I'll get to that soon enough). True system programming fanatics know that NTDLL is the real guts of the user-mode kernel. I've compared the exports from the Windows Server 2003 version of NTDLL.DLL to the Windows 2000 version. As you might expect, lots of APIs were added and a few have disappeared. Big deal, they're all undocumented, right? Not so fast!
As I was comparing all those .H files between different versions of the Platform SDK, I came across a very interesting file in the October 2002 version. It's named WINTERNL.H. This is definitely a file to hold on to. It could disappear in future Platform SDKs. At the top of WINTERNL.H is a big three-paragraph warning about how all the data structures and APIs are subject to change and intended for use only by Windows core components. (Not like we haven't seen this before, eh?)
Anyhow, so what goodies are in WINTERNL.H? Unfortunately, not as many as you'd like. But there are tantalizing hints of well-known, yet still undocumented data structures. For example, you'll see both the Process Environment Block (PEB) and Thread Environment Block (TEB) defined. However, most of the fields are listed as reserved. Likewise, the structures and prototypes necessary to call NtQueryInformationProcess and NtQueryInformationThread are there. However, there are a variety of books and Web sites that provide many more details about these APIs and structures than WINTERNL.H does. Admittedly, nothing in this file is new in Windows Server 2003. However, the SDK that accompanies Windows Server 2003 is where this file first appeared.
Why am I making a big fuss about this? It's significant because Microsoft is finally admitting that there are lots of interesting things that aren't documented. A few other APIs in WINTERNL.H are worth noting: NtCreateFile, NtOpenFile, NtClose, and NtWaitForSingleObject. These APIs are the guts of the user mode implementation of common KERNEL32 APIs. Likewise, RtlUnwind is a key API used in structured exception handling (SEH), as described in my January 1997 article in Microsoft Systems Journal "A Crash Course on the Depths of Win32® Structured Exception Handling". It's highly unlikely that RtlUnwind will change. If it did, a very large percentage of existing applications would fall flat on their faces.
Processes, Threads, and Fibers, Oh My!
Moving on to KERNEL32 land, my first group of APIs, shown in Figure 1, relate to processes and threads. With one exception, they all retrieve some piece of information about a process or thread. GetThreadId takes a thread handle and returns the associated thread ID. It's a similar story for GetProcessId. It's hard to believe these functions weren't in the Win32 API 10 years ago! IsWow64Process tells you if the calling process is a 32-bit process running under 64-bit Windows.
Figure 1 Process and Thread APIs
| GetProcessHandleCount |
| IsProcessInJob |
| IsWow64Process |
| CreateJobSet |
| GetProcessIdOfThread |
| GetThreadId |
| GetProcessId |
| GetThreadIOPendingFlag |
| GetCurrentProcessorNumber |
| SetProcessWorkingSetSizeEx |
| GetProcessWorkingSetSizeEx |
| RestoreLastError |
GetProcessHandleCount returns the number of handles that the specified process has open. It's the same value seen in the Performance Monitor data or in the Task Manager. RestoreLastError is an enigma. It's code is identical to SetLastError. It's unclear to me why it was made into a separate API.
To demonstrate some of these new APIs, check out the ProcessesAndThreads program in Figure 2. The code is self-explanatory, so I won't delve into it here. To build it (and the other sample programs), you will need at least the October 2002 Platform SDK, with its Include and Lib directories at the head of the compiler and linker's directory search order. If you're running Windows XP, you can build it to use just the Windows XP subset of the APIs. Simply uncomment the #define W2K3SERVER line near the top and rebuild.
Figure 2 ProcessesandThreads.cpp
//------------------------------------------------------------------- // Matt Pietrek // MSDN Magazine, 2003 // Program: ProcessesAndThreads // Purpose: A demonstration of the Windows XP/Windows Server 2003 Process/ // Thread APIs //------------------------------------------------------------------- #include "stdafx.h" // comment out to get just the XP APIs #define W2K3SERVER 1 int main(int argc, char* argv[]) { DWORD dwThreadID = GetCurrentThreadId(); DWORD dwProcessID = GetCurrentProcessId(); printf( "ProcessId: %X ThreadId: %X\n", dwProcessID, dwThreadID ); HANDLE hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, dwProcessID ); if ( !hProcess ) { printf( "unable to open process\n" ); return 0; } // Show OpenThread() HANDLE hThread = OpenThread( THREAD_ALL_ACCESS, FALSE, dwThreadID ); printf( "hProcess: %IX hThread: %IX\n", hProcess, hThread ); // Show GetProcessHandleCount() DWORD dwHandleCount; GetProcessHandleCount( hProcess, &dwHandleCount ); printf( "Handle Count: %u\n", dwHandleCount ); // Show IsProcessInJob() BOOL bIsInJob; IsProcessInJob( hProcess, NULL, &bIsInJob ); printf( "IsProcessInJob: %s\n", bIsInJob ? "true" : "false" ); // Show GetProcessId() printf( "Process ID from hProcess: %X\n", GetProcessId( hProcess ) ); // Show IsWow64Process BOOL bIsWow64Process; IsWow64Process( hProcess, &bIsWow64Process ); printf( "IsWow64Process: %s\n", bIsWow64Process ? "true" : "false" ); BOOL bIOIsPending; GetThreadIOPendingFlag( hThread, &bIOIsPending ); printf( "IoIsPending: %s\n", bIOIsPending ? "true" : "false" ); #ifdef W2K3SERVER // Show GetProcessWorkingSetSizeEx() DWORD wsFlags; SIZE_T wsSizeMin, wsSizeMax; GetProcessWorkingSetSizeEx( hProcess, &wsSizeMin, &wsSizeMax, &wsFlags ); printf( "Working Set: min:%IX max:%IX flags:%X\n", wsSizeMin, wsSizeMax, wsFlags ); // Show GetProcessIdOfThread() DWORD dwProcessIdFromThread = GetProcessIdOfThread( hThread ); printf("Process ID obtained from thread ID: %X\n", dwProcessIdFromThread); // Show GetThreadId(); printf("Thread ID obtained from thread handle: %X\n",GetThreadId(hThread)); #endif return 0; }
In addition to threading support, Windows Server 2003 adds some new Fiber APIs, shown in Figure 3. The big news here is the addition of fiber local storage (FLS). The APIs are used identically to their thread local storage (TLS) counterparts. A slot is allocated via the FlsAlloc function. To set or retrieve values, use the FlsGetValue and FlsSetValue function. When you are done with the slot, you call FlsFree.
Figure 3 Fiber APIs
| FlsAlloc |
| FlsGetValue |
| FlsSetValue |
| FlsFree |
| ConvertFiberToThread |
| ConvertThreadToFiberEx |
| CreateFiberEx |
Just for grins, I looked into the implementation of the FLS functions. At offset 0xFB4 in the Thread Environment Block is a pointer to a data structure. Eight bytes into this structure is an array of 128 slots. These slots are conceptually the same as the TLS slots. As the thread switches from fiber to fiber, the pointer at offset 0xFB4 is updated accordingly.
The ConvertFiberToThread API undoes the effect of ConvertThreadToFiber. Once called, no more fiber functions can be called on the thread. The remaining two APIs listed in Figure 3 are just "Ex"-tended versions of existing APIs. CreateFiberEx is just like CreateFiber, but with the ability to specify the stack reserve size. ConvertThreadToFiberEx is interesting, though. In the original fiber implementation, the floating point, MMX, and SSE registers weren't saved and restored across fiber switches to improve performance. The new API lets you specify that these registers need to be saved and restored as well.
Vectored Exception Handling
Perhaps the most exciting addition to KERNEL32 is vectored exception handling (VEH), which provides additional flexibility in how to process exceptions. I covered vectored exception handling in depth in my September 2001 Under The Hood column in MSDN Magazine, so I'll just give a brief synopsis here along with an illustration of the process in Figure 4.
Figure 4** Vectored Exception Handling **
Traditional structured exception handling (SEH) with its __try/__except mechanisms is inherently thread specific. Exceptions can only be handled by the thread that set up a handler. (The compiler and OS handle all the messy details of this and expose the relatively simple __try/__except syntax to you.) More importantly, with SEH you might set up a handler, only to have the exception grabbed first by another handler that doesn't know how to deal with the exception properly.
Vectored exception handling works more like a traditional notification callback scheme. To handle exceptions, call the AddVectoredExceptionHandler API, passing it the address of your exception callback function. When an exception occurs, the callback function receives a pointer to an EXCEPTION_POINTERS structure. This is the same structure that SEH callbacks can receive via the GetExceptionInformation API. From fields in the EXCEPTION_POINTERS structure, you can learn the exception code (for instance, 0xC0000005) and the register values (via the included CONTEXT structure).
The VEH callback chooses to either handle the exception or chain it onto the next handler in the list. It determines what happens by returning the appropriate value from the callback. Each process has a linked list of VEH callbacks. As part of processing an exception, the OS walks the VEH list and calls the handlers. To remove a handle from the list, use the RemoveVectoredExceptionHandler API.
How does vectored exception handling coexist with SEH? Good question! Immediately before walking the SEH chain, the system walks the vectored exception handler list. Put another way, VEH handlers have priority over SEH handlers. To see vectored exception handling in action, check out the VEHDemo program in Figure 5. VEHDemo installs a couple of vectored exception handlers and uses a structured exception handler to show how VEH and SEH work together. The resulting output from running the VEHDemo program is shown in Figure 6.
Figure 6 VEHDemo Output
In VectoredExceptionHandler, EIP =00411BBA In VectoredExceptionHandler, EIP =00411BBA In VectoredExceptionHandler, EIP =00411BBA In SecondVectoredExceptionHandler - handling it In VectoredExceptionHandler, EIP =00411BBB In VectoredExceptionHandler, EIP =00411BBB In VectoredExceptionHandler, EIP =00411BBB In SecondVectoredExceptionHandler - NOT handling it Caught the exception in Function1
Figure 5 VEHDemo.cpp
//------------------------------------------------------------------- // Matt Pietrek // MSDN Magazine, 2003 // Program: VEHDemo // Purpose: A demonstration of Vectored Exception Handling //------------------------------------------------------------------- #define WIN32_LEAN_AND_MEAN #define _WIN32_WINNT 0x0501 #include <stdio.h> #include <tchar.h> #include <windows.h> LONG WINAPI VectoredExceptionHandler(PEXCEPTION_POINTERS ExceptionInfo) { // Stupid handler that does nothing except print out the faulting EIP printf( "In VectoredExceptionHandler, EIP =%p\n", (PVOID)(DWORD_PTR)ExceptionInfo->ContextRecord->Eip ); // Don't handle the exception here. Let normal processing continue return EXCEPTION_CONTINUE_SEARCH; } LONG WINAPI SecondVectoredExceptionHandler(PEXCEPTION_POINTERS ExceptionInfo) { // If the faulting EIP points to a HLT instruction, just skip past // it, to show us "Handling" an exception if ( *(PBYTE)ExceptionInfo->ContextRecord->Eip == 0xF4 ) { printf( "In SecondVectoredExceptionHandler - handling it\n" ); ExceptionInfo->ContextRecord->Eip++; return EXCEPTION_CONTINUE_EXECUTION; } else // For everything else, we won't handle the exception here { printf("In SecondVectoredExceptionHandler - NOT handling it\n"); return EXCEPTION_CONTINUE_SEARCH; } } int _tmain(int argc, _TCHAR* argv[]) { // Add a vectored exception handler. In fact, add the same // handler 3 times, to prove that they handlers are stored // in a linked list. This handler always returns "keep looking" AddVectoredExceptionHandler( 1, VectoredExceptionHandler ); AddVectoredExceptionHandler( 1, VectoredExceptionHandler ); AddVectoredExceptionHandler( 1, VectoredExceptionHandler ); // And a second vectored handler that knows how to handle a HLT // caused exception, but nothing else AddVectoredExceptionHandler( 0, SecondVectoredExceptionHandler ); __try { __asm HLT // Cause a privileged instruction fault *(int *)0 = 0; // Access Violate by dereferencing NULL ptr } __except( EXCEPTION_EXECUTE_HANDLER ) { // This handler will catch the *(int *)0 = 0 fault printf( "Caught the exception in Function1\n" ); } return 0; }
Directories and Files
A few new APIs have been added in the File and Directory category, shown in Figure 7. SetDllDirectory adds an arbitrary set of directories that will be searched when the system looks for a DLL. The system searches the specified path(s) after the application load directory but before anywhere else, such as the system directory. The documentation for SetDllDirectory describes the exact search order and makes for interesting reading. GetDllDirectory returns whatever value has been previously set by calling SetDllDirectory. GetSystemWow64Directory is used for finding the 32-bit system directory when on a 64-bit system.
Figure 7 Directory and File APIs
| GetDllDirectory / SetDllDirectory |
| NeedCurrentDirectoryForExePath |
| GetSystemWow64Directory |
| SetFileShortName |
| CheckNameLegalDOS8Dot3 |
| SetFileValidData |
| GetVolumePathNamesForVolumeName |
| FindFirstStream / FindNextStream |
| ReOpenFile |
A little-known fact about the NTFS file system is that it supports multiple streams in a file. This is tricky to explain in a limited space, but the gist is that multiple files can be collectively referred to by a single file name. Most files only have a single default stream associated with them, and that's what most Win32 APIs report about. To create a stream other than the default stream, simply append a colon (:), followed by the stream name. For instance, you could use Notepad to create a file/stream called abc.txt:MyStream. In the Windows Explorer window, you'll see a zero-byte abc.txt file. However, the abc.txt:MyStream exists. The normal Win32 APIs simply don't report any information about it.
In Windows Server 2003, this situation is remedied somewhat. The FindFirstStream and FindNextStream APIs enumerate through all the streams in a file. To demonstrate their use, I wrote the FindFirstStream program shown in Figure 8. To use it, simply pass it a file name. If there are any streams beyond the default unnamed stream, it lists them all. Here is the output for a file, a.txt, with three streams—abc, def, and ghi:
a.txt:abc:$DATA a.txt:def:$DATA a.txt:ghi:$DATA
Figure 8 FindFirstStream.cpp
//------------------------------------------------------------------- // Matt Pietrek // MSDN Magazine, 2003 // Program: FindFirstStream // Purpose: A demonstration of the Windows XP/Windows Server 2003 File // Stream APIs //------------------------------------------------------------------- #include "stdafx.h" int main(int argc, char * argv[]) { if ( argc != 2 ) return 0; WIN32_FIND_STREAM_DATA streamData; wchar_t wszFileName[512]; mbstowcs( wszFileName, argv[1], 0xFFFFFFFF ); HANDLE h = FindFirstStreamW( wszFileName, FindStreamInfoStandard, &streamData, 0 ); if ( h == (HANDLE)ERROR_INVALID_HANDLE ) { // printf( "unable to being stream enum for file %s\n", argv[1] ); return 0; } while ( 1 ) { if ( !FindNextStreamW( h, &streamData ) ) break; printf( "%ls%ls\n", wszFileName, streamData.cStreamName ); } // Do we need to close the stream handle??? return 0; }
The ReOpenFile API is used for taking an existing file handle and getting another handle that has a different set of access rights. A typical use is where you have code that has only a file handle and doesn't know the associated file name. If your code needs access rights or a sharing mode different than the existing handle has, ReOpenFile provides a way to attempt to obtain those rights. Of course, ReOpenFile ensures that the newly requested access rights and sharing mode are legal. It also lets you guard against pipe impersonation attacks.
CheckNameLegalDOS8Dot3 has one of those fluky API names. This API is useful for checking if a file name can be used on a file allocation table (FAT) file system volume.
You may recall that with the advent of long file names (beyond 8.3), the OS needed a way to refer to the file using standard 8.3 conventions. The system has an algorithm for mapping between the long and short versions of the name. These files are easily picked out because the short version ends with a tilde (~), followed by a number (for instance, "foobar~1.txt"). The new SetFileShortName API allows you to override the system's default short file name. To use it however, the target file needs to be on an NTFS volume.
Memory and System Information
In the memory allocation arena, Windows Server 2003 and Window XP have a feature called the low-fragmentation heap. This heap algorithm avoids fragmentation by allocating all blocks from 128 predetermined different block-size ranges, called buckets. When an application needs to allocate memory from the heap, the heap chooses the bucket that's able to hold the requested block with the least wasted space. The system uses the traditional heap for blocks greater than 16KB. To use the low-fragmentation heap, call HeapSetInformation, passing it the appropriate heap handle and flag value. All heaps default to "normal" Win32 heap behavior until HeapSetInformation is called. To determine which behavior a heap is using, call the HeapQueryInformation API.
In the area of system information, there's a motley collection of new interfaces, as you can see in Figure 9. GetSystemRegistryQuota provides you with the current size of the registry, as well as the maximum allowed size. GetSystemTimes returns the length of time that all processors have spent in the idle state, in kernel mode, and in user mode.
Figure 9 System Information APIs
| GetLargePageMinimum |
| GetSystemRegistryQuota |
| GetSystemTimes |
| GetNativeSystemInfo |
| TzSpecificLocalTimeToSystemTime |
| SetFirmwareEnvironmentVariable |
| GetFirmwareEnvironmentVariable |
| CreateMemoryResourceNotification |
| QueryMemoryResourceNotification |
| GetLogicalProcessorInformation |
The GetNativeSystemInfo is intended for 32-bit programs running under 64-bit Windows. It returns a SYSTEM_INFO structure filled in as if it were called from a native 64-bit program. For instance, running an x86 program on an Itanium machine, the SYSTEM_INFO.dwPageSize would be 8192 bytes, rather than the 4096 bytes that you'd get from calling GetSystemInfo. The SystemInfo program in Figure 10 shows GetSystemInfo being used, as well as a few of the other new system information APIs.
Figure 10 SystemInfo.cpp
//------------------------------------------------------------------- // Matt Pietrek // MSDN Magazine, 2003 // Program: SystemInfo // Purpose: A demo of the Windows XP/Windows Server 2003 System Info APIs //------------------------------------------------------------------- #include "stdafx.h" // Uncomment to get Windows Server 2003 APIs as well // #define W2K3SERVER 1 int main(int argc, char * argv[]) { // Show GetSystemRegistryQuota(); DWORD dwQuotaAllowed, dwQuotaUsed; GetSystemRegistryQuota( &dwQuotaAllowed, &dwQuotaUsed ); printf( "Quota allowed: %uK, Quota used: %uK\n", dwQuotaAllowed/1024, dwQuotaUsed / 1024 ); // Show GetSystemTimes() FILETIME IdleTime; FILETIME KernelTime; FILETIME UserTime; GetSystemTimes( &IdleTime, &KernelTime, &UserTime ); const DWORD filetimeDivisor = 1000000000 / 100; printf( "IdleTime: %I64u seconds\n", *(PDWORD64)&IdleTime / filetimeDivisor ); printf( "KernelTime: %I64u seconds\n", *(PDWORD64)&KernelTime / filetimeDivisor ); printf( "UserTime: %I64u seconds\n", *(PDWORD64)&UserTime / filetimeDivisor ); // Show GetNativeSystemInfo(). If you're running in a WOW session // (e.g., an X86 app running on IA64), it'll return the capabilities // of the actual OS, not the WOW info. If not running in WOW, you'll // get the same info as GetSystemInfo SYSTEM_INFO sysInfo; GetNativeSystemInfo( &sysInfo ); printf( "Native System Info - Processor type: %u page size: %X\n", sysInfo.dwProcessorType, sysInfo.dwPageSize ); #ifdef W2K3SERVER printf( "Large Page Minimum size: %IX\n", GetLargePageMinimum() ); #endif return 0; }
GetLogicalProcessorInformation returns information that is pertinent both to NUMA systems and to Intel's Hyperthreaded CPUs (wherein a single CPU has multiple execution units). The API returns an array of SYSTEM_LOGICAL_PROCESSOR_INFORMATION structures.
The CreateMemoryResourceNotification is a way for applications to receive a notification when the available physical memory is running low without having to constantly poll the value. The API creates a handle that can be passed to the WaitForXxx family of functions. The handle is signaled when free memory drops below a threshold. According to the documentation, the threshold is 32MB per 4GB of memory on the system. You can also check the memory status directly using the QueryMemoryResourceNotification. The system can also notify you when available free physical memory is high, but I doubt this will be a heavily used capability.
Debug APIs
In the realm of debugging, there's a small set of new APIs. The most exciting has to be DebugSetProcessKillOnExit. Up until now, if you were acting as a debugger for another process, there was no way to stop acting as a debugger. You simply couldn't detach from a process being debugged. When you act as a debugger for another process, one of your threads is the debug thread, and processes all the debug notification messages. Normally, if this thread terminates, the debuggee process terminates as well. The DebugSetProcessKillOnExit API changes this behavior. By passing FALSE, you tell the system to stop requiring the debug thread to process messages for the debuggee.
In a related vein, you'll find DebugActiveProcessStop, which tells the system to detach the specified process from whatever process is debugging it. It can only be called by the thread that called DebugActiveProcess or CreateProcess. Since a debugger thread can conceivably debug multiple processes at the same time, DebugActiveProcessStop needs the parameter that indicates which process to detach from.
The DebugBreakProcess is just like DebugBreak, except that it applies to the specified process rather than to the current thread. The API works by creating a thread in the target process, much like CreateRemoteThread does. The newly created thread invokes a breakpoint instruction, which causes the normal SEH mechanism to take over. For developers, this usually means that the Just-In-Time debugging dialog comes up.
The final new debugging API is CheckRemoteDebuggerPresent. It's similar to the IsDebuggerPresent API in that it tells you if a process is running under the control of a debugger process. IsDebuggerPresent tells you if your process is being debugged, while CheckRemoteDebuggerPresent lets you query about any process for which you have a handle.
Side-by-side Execution
Much has been made of the side-by-side installation and execution feature in the .NET Framework. However, these same capabilities are built into Windows Server 2003 and Windows XP. The key to these capabilities are the new activation context (ActCtx) APIs, shown in Figure 11.
Figure 11 ActCtx APIs
| CreateActCtx |
| AddRefActCtx |
| ReleaseActCtx |
| ActivateActCtx |
| DeactivateActCtx |
| GetCurrentActCtx |
| QueryActCtx |
| ZombifyActCtx |
| FindActCtxSectionString |
| FindActCtxSectionGuid |
An activation context is a set of system-managed data structures that contain information used to cause an application to use a particular DLL version or COM object instance, based upon a manifest file. Manifest files use the XML format (no surprise there!) and look very similar to .NET manifests. The use of activation contexts could easily be an article on its own, so I'll defer to the SDK documentation. However, it's interesting to note that certain system DLLs are enabled for side-by-side execution, with their corresponding .H files now using the activation context APIs. For a prime example, examine a recent version of COMMCTRL.H. There are a ton of inline functions with names like IsolationAwareImageList_Add. These inline functions show the activation context APIs in action. You'll also see some clever tricks with C++ macros to keep existing code compiling without any changes.
There's one final Kernel32 API that doesn't fit nicely into any other category. GetModuleHandleEx should have been included in the Win32 API years ago. The key capability it adds is finding an HMODULE when given an address within that module. If you've ever written debugging or diagnostic code, you've probably been in the situation where you had a code address and needed to determine which DLL it came from. It was possible to accomplish this in an awkward way using VirtualQuery, but GetModuleHandleEx is much more elegant.
Unlike its predecessor API (GetModuleHandle), GetModuleHandleEx affects the module's reference count, unless you explicitly tell it not to. Depending on the specified flags, it can increment the reference count, leave it unchanged, or pin the DLL in memory for the lifetime of the process. The GET_MODULE_HANDLE_EX_FLAG_PIN flag addresses one of my nagging concerns. Let's say you called GetModuleHandle to retrieve a module handle. In a multithreaded program, it's possible that another thread could unload the DLL and load another DLL at the exact same address. All this can happen between the time the first thread gets the HMODULE and when it starts using it. By pinning the module in memory, you're guaranteeing that the HMODULE you get will be valid when you use it later.
User Interface Additions
On the user interface side of things, the biggest new addition to USER32 is the raw input APIs. They offer an alternative to the keyboard and mouse as the only way to receive input from users. Using the raw input APIs, devices such as joysticks, microphones, or touch screens are on equal footing with the keyboard and mouse.
In the standard Windows input model, the keyboard and mouse drivers create low-level scan codes and motion events. The system takes these low-level events and translates them into higher-level messages, such as WM_CHAR or WM_APPCOMMAND. While this is great for keeping input capture simple, it's not very adaptable to other input devices.
The new APIs for raw input are shown in Figure 12. By default, applications don't receive raw input. Instead, you must register to receive the input through the RegisterRawInputDevices API, which takes an array of all the devices that you're interested in.
Figure 12 USER32 APIs
| Raw Input APIs |
|---|
| GetRawInputData |
| GetRawInputDeviceInfo |
| GetRawInputBuffer |
| RegisterRawInputDevices |
| GetRegisteredRawInputDevices |
| GetRawInputDeviceList |
| DefRawInputProc |
| Other New APIs |
| PrintWindow |
| IsGUIThread |
| GetWindowRgnBox |
| BroadcastSystemMessageEx |
| DisableProcessWindowsGhosting |
When a device has input, the system posts a WM_INPUT message to the program's message queue. The program reads the input in either an unbuffered mode (one message at a time) or in a buffered mode (where multiple messages are read at once). As you'd expect, there are APIs for enumerating all the raw input devices and querying for information about them.
There are also a handful of other new APIs in USER32 that I found interesting (see Figure 12). The PrintWindow API copies the contents of the specified HWND into the specified device context (DC). IsGUIThread returns (and optionally sets) a value that tells whether the calling thread is a GUI thread, meaning that the thread has called into Win32K.SYS and has a larger kernel-mode stack. BroadcastSystemMessageEx is like BroadcastSystemMessage, except that it returns more information about the window that had denied a request.
Although it's available as a redistributable for earlier operating systems, the Text Services Framework (TSF) makes its debut with Windows Server 2003 and Windows XP. The Text Services Framework is an extensible system for reading and writing text in a manner that's independent of the input/output device. The most readily accessible example of the TSF is as a way of allowing applications to receive text input from devices such as pens or microphones.
Each different type of text input/output device becomes a "text service." Between the text services and the applications is the TSF manager. To draw an analogy to databases, each text service is like an ODBC driver, and the TSF manager plays the role of the ODBC manager. The TSF is composed of a few APIs and many dozens of interfaces. An entire article—if not a book—could be devoted to them, so I won't attempt to describe them further here.
On a more graphical slant, Microsoft has put a friendly object-oriented face on the GDI. Sure, application frameworks like MFC have done this for years, but the new GDI+ API is part of the core operating system and doesn't require you to pull in all the baggage of an application framework. In broad strokes, the GDI+ functionality falls into the following four categories: two-dimensional vector graphics (lines and curves), imaging (bitmaps), typography (display of text), and matrix transformations.
Although the GDI+ APIs are technically just like any other Win32 APIs, it's unlikely that you'd call them directly (at least from C++ code). Instead, the Platform SDK has a set of header files (with names like GDIPlus.H) that define approximately 40 C++ classes. Typical of these classes are Bitmap, Font, and Region. The methods of these classes are usually inline functions that call the underlying API in GDIPlus.DLL.
Here's a typical use of the GDIPlus classes in C++ code:
VOID OnPaint(HDC hdc) { Graphics graphics(hdc); Pen pen(Color(255, 0, 0, 255)); graphics.DrawLine(&pen, 0, 0, 200, 100); }
Note that there's no BeginPaint/EndPaint tedium. Everything is very object-oriented. The Graphics object is the GDI+ equivalent of the device context handle (HDC). The Pen object takes care of the creation and destruction of the underling GDI pen for you.
The best way to get a feel for GDI+ is to browse through GDIPlus.H and the files it references. To use GDI+, you'll need to #include GDIPlus.H in your source file and add the GDIPlus.lib file to your linker line. As a side note, GDIPlus.DLL is side-by-side enabled. Thus, you won't find it in the \windows\system32 directory. Instead, you'll find the various versions of it in subdirectories under \windows\winsxs\.
Newly Documented Interfaces
Microsoft has recently documented several hundred APIs and COM interfaces that were in Windows 2000 or earlier. These APIs can be found at Settlement Program Interfaces. Although not technically new, there's documentation for them, so you can now safely call them in Windows XP and later. To be honest, when I looked at the APIs, I thought that with a few exceptions, there was really nothing scandalous or sizzling. Looking at them critically, you'll see that they're mostly rounding out existing API sets. In addition, some of the APIs in the list are already documented, but new details were included.
By far, the biggest subset of APIs fall within the SHELL32 library. There's approximately 110 APIs here, but judging from the names, many are of limited use. CDefFolderMenu_Create2 anybody? On a more promising note, there are more than 20 new COM interfaces, including IMenuBand, IShellItem, and IShellTaskScheduler.
The newly documented APIs also have a fair selection of new cryptography-related functions. WININET.DLL has five new APIs, mostly having to do with proxy support. The DirectShow functionality has a dozen or so new APIs added to it. Finally, some of the APIs such as NtQuerySystemTime and RtlUnwind are described in the WINTERNL.H file mentioned earlier.
New DEBUGHLP Additions
Long-time readers of my articles and columns for MSDN Magazine know that DBGHELP.DLL is one of my favorite DLLs. It's undergone quite substantial changes since its Windows 2000 incarnation. It does so many new things that it's really quite hard to know where to begin.
One of the coolest things in DbgHelp isn't an API that you call. Have you ever been frustrated when your debug symbols get out of sync with the DLLs on your system? Thanks to DBGHELP.DLL, this is (mostly) a thing of the past. The new feature is called the symbol server. When you ask DbgHelp to load the symbols for a module, it calls a symbol server DLL to locate the debug files if it can't find the debug file locally. The symbol server DLL is free to locate the debug files in whatever manner it sees fit. Conceptually, anybody can write a symbol server DLL, and DbgHelp will use it.
In reality, Microsoft has created the one symbol server DLL (SymSrv.DLL) that just about everybody will use. In addition, they've put debugging symbols for just about every relevant version of Windows on a publicly available Web server. The end result is that debuggers and tools can dynamically obtain debug files with no extra effort on their part. All that's necessary is to use DbgHelp.DLL to access the symbols. SymSrv.DLL is part of the Debugging Tools for Windows, which can be downloaded from the MSDN site (see Microsoft Debugging Tools).
SymSrv.DLL automatically downloads the appropriate PDB file the first time it's needed and stores it locally. It ensures that the PDB file it downloads is the correct version for the DLL. When storing the PDB files locally, it uses a directory naming scheme that allows the PDBs for multiple versions of the DLL to coexist.
The symbol server functionality is available to users of Visual Studio® .NET. All that's required is to put SymSrv.DLL in the same directory as the IDE (DevEnv.exe) and set an environment variable. DBGHELP.DLL by default uses the path in the _NT_SYMBOL_PATH to locate symbols. To indicate that the symbol server should be used, _NT_SYMBOL_PATH should be something like the following:
symsrv*symsrv.dll*c:\winnt\symbols* https://msdl.microsoft.com/download/symbols
Obviously, you'll want the path portion ("c:\winnt\symbols" in this example) to point at a valid directory on your hard drive. Assuming you have everything set up properly, this should work well. I've done this successfully on a number of machines, but unfortunately I can't provide support if you have problems.
The next big thing in DbgHelp is type support. My March 2002 Under The Hood column covers this in much more detail, so I'll just mention it in passing here. The type support extends beyond the primitive types to include user-defined types. Using the new SymFromAddr and SymFromName APIs, you can obtain a type index. This type index is then passed to SymGetTypeInfo to obtain information about the type. SymGetTypeInfo is a fairly difficult API to understand, so I again direct you to the aforementioned column on this subject. All the user-defined types in a given symbol table can be enumerated with the SymEnumTypes.
If you're a longtime DbgHelp user, you may notice that many of the new APIs run parallel to existing interfaces. For example, SymEnumSymbols would seem to do the same thing as SymEnumerateSymbols. The reason for the new APIs is that the old APIs didn't provide very complete information about the symbols. The newer APIs always use a SYMBOL_INFO structure, which has much more complete information about a symbol.
Another exciting addition to DbgHelp is an awareness of local variables and parameters. In the past, you could enumerate symbols for a module, but only global symbols. With the new SymEnumSymbols API, you can enumerate locals and parameters. The not-so-obvious trick to doing this is to call the SymSetContext function beforehand. The call to SymSetContext is where you specify an address within the particular function you're interested in. Under the hood, DbgHelp locates the enclosing function's local variables and parameters and enumerates only them.
The MiniDumpWriteDump API is exciting. With a single call, you can create your own dump files. These are the very same files that you'd get from a Dr. Watson crash or from the UserDump tool. These dump files can be loaded into WinDBG or Visual Studio .NET for post-mortem debugging. A typical reason to create a dump file would be if your program encounters some unexpected condition in the field. The user can send you back the file for perusal in the debugger of your choice.
There are many more new DbgHelp APIs, but I'll mention just a few more in this section. You're now able to enumerate source file lines via the SymEnumLines API. SymAddSymbol and SymDeleteSymbol let you dynamically extend the symbols defined in the symbol file. The SymFromToken API returns a SYMBOL_INFO, given a .NET metadata method token. This is an important first step for DBGHELP in supporting .NET Framework-based debug information.
With all these new features, it would be a shame not to show off some of them. The DBGHELP51 program (included with this month's download at the link at the top of this article) uses some of the new APIs just discussed, including SymEnumSymbols, SymEnumLines, and SymGetTypeInfo. To test it, run DBGHELP51.EXE, passing it the name of an EXE file that has debug information for it. If the debug information loads correctly, DBGHELP51.EXE first lists all the global symbols. If type information is available, the type name follows the symbol name. The second part of the output is any source file and line number information, if found.
Windows Error Reporting
A new DLL with only two exported APIs? That's what we have with the Windows error reporting API. As you've probably noticed over the years, Microsoft has made significant efforts to make application faults less scary to the typical user. For example, starting in Windows XP, when a program encounters an unhandled exception, a dialog pops up asking if you want to send a report to Microsoft. The new error reporting API lets applications integrate better with the system in this regard.
The first API is ReportFault, which is used in applications that catch their own exceptions with try blocks. It lets them tie into the system's fault reporting mechanism. ReportFault takes an EXCEPTION_POINTERS structure as a parameter. Calling ReportFault should cause the same system actions that would occur if you hadn't caught the exception. The return value from ReportFault is a code indicating what the system did. For example, one return code, frrvLaunchDebugger, indicates that a debugger was launched to attach to the program.
The second API is AddERExcludedApplication, which prevents the system from reporting faults on the specified executable. For example, if you called the API with the string foo.exe, any program called foo.exe that had an unhandled exception wouldn't be reported as such. The parameter to AddERExcludedApplication should be just a simple EXE name, without any path information.
ADVAPI32
Given the emphasis on security at Microsoft, it's not surprising that there's a new set of interfaces known as the credential APIs. They obtain and manage information such as user names and passwords. They can request Windows XP account information to be used in place of the credentials established while logging on. Such requests typically occur when the logon credentials do not have permissions required by the application.
Figure 13 shows the credential APIs, which come from the new WINCRED.H file. To show some basic functionality, I wrote the Credential program which enumerates all the current credentials and displays basic information for each one (see Figure 14). Try running it on your system. What you see might surprise you.
Figure 14 Credential
//------------------------------------------------------------------- // Matt Pietrek // MSDN Magazine, 2003 // Program: Credential // Purpose: A demonstration of the Windows XP/ Windows Server 2003 // Credential APIs //------------------------------------------------------------------- #include "stdafx.h" int _tmain(int argc, _TCHAR* argv[]) { PCREDENTIAL * pCreds; DWORD cCreds; CredEnumerate( NULL, 0, &cCreds, &pCreds ); for ( DWORD i = 0; i < cCreds; i++ ) { PCREDENTIAL p = pCreds[i]; printf( "Flags:%X Type:%X Target: %s User: %s\n", p->Flags, p->Type, p->TargetName, p->UserName ); } CredFree( pCreds ); return 0; }
Figure 13 Credential APIs
| CredRead |
| CredWrite |
| CredRename |
| CredEnumerate |
| CredDelete |
| CredFree |
| CredGetSessionTypes |
| CredGetTargetInfo |
| CredMarshalCredential |
| CredIsMarshaledCredential |
| CredReadDomainCredentials |
| CredWriteDomainCredentials |
| CredProfileLoaded |
| CredUnmarshalCredential |
| CredUIPromptForCredentials |
| CredUICmdLinePromptForCredentials |
| CredUIParseUserName |
| CredUIConfirmCredentials |
| CredUIStoreSSOCred |
Another new set of security APIs in ADVAPI32 is the safer APIs. The goal is to make it easy for programs which launch other programs to query security policy for approval before an executable is launched. This capability isn't limited to executables, as other sorts of active content such as scripts can also be verified. The APIs are especially useful for handling e-mail attachments. Figure 15 shows the safer APIs, which are defined in WinSafer.H. As it stands now, the documentation for these APIs is lacking in practical examples of how to use the functions.
Figure 15 More APIs
| Safer APIs |
|---|
| SaferGetPolicyInformation |
| SaferSetPolicyInformation |
| SaferCreateLevel |
| SaferCloseLevel |
| SaferComputeTokenFromLevel |
| SaferIdentifyLevel |
| SaferGetLevelInformation |
| SaferSetLevelInformation |
| SaferRecordEventLogEntry |
| Event Tracing APIs |
| TraceMessage |
| TraceMessageVA |
| EnumerateTraceGUIDs |
| FlushTrace |
Also new in ADVAPI32 are a few new event tracing APIs (see Figure 15). Event tracing was introduced in Windows 2000. As you'd expect, TraceMessage sends an event to the designated trace session. TraceMessageVA is essentially the same API, except that it takes a variable number of arguments. EnumerateTraceGUIDs returns information about the system's event trace providers, while FlushTrace does just what its name implies.
OLE Gets Old
Traditionally, the OLE32 and OLEAUT32 DLLs have been hotbeds of new APIs and interfaces. Since Windows 2000, it's slowed significantly due to the focus on the .NET Framework. One new API that looks interesting is CoRegisterInitializeSpy. You provide an interface implementation of type IInitializeSpy, and its methods are called before and after CoInitialize(Ex) and CoUninitialize on the thread for which the spy is registered.
The CoGetContextToken API returns the IObjContext for the current context. It's interesting primarily because this value is stored in the ReservedForOle field in the TEB, which is finally documented in WINTERNL.H.
The CoFreeUnusedLibrariesEx function is like its predecessor function, with the added capability of freeing unused libraries immediately rather than waiting the default 10 minutes. Finally, the new CoInvalidateRemoteMachineBindings API tells the OLE Service Control Manager to flush any cached remote procedure call binding handles for the specified machine. Beyond these few APIs, there's nothing else new in OLE and nothing at all new in OLEAUT32.
Clustering
The clustering API has a small number of new functions worth mentioning. Clustering is the ability to keep resources (such as applications, disks, and file shares) highly available by using more than one physical resource to represent a single logical resource to the outside world. The majority of the new clustering APIs are what I call the "EnumCount" subset. In a nutshell, there are five types of cluster objects: Group, Network, Node, Resource, and Resource Type. These objects can be enumerated via handle-based APIs. The new APIs here (for instance, ClusterNodeGetEnumCount) return the number of objects represented by the enumeration handle.
The two remaining new APIs are EvictClusterNodeEx and SetClusterServiceAccountPassword. EvictClusterNodeEx is like its predecessor function, with the addition of a timeout capability. The SetClusterServiceAccountPassword changes the password for the cluster service user account on all online nodes.
Real Time Client API
The real time client (RTC) API has one of those highfalutin names that doesn't really convey what it does. Although the RTC API encompasses quite a lot of functionality, I find it easiest to think of it primarily as the instant messaging (IM) API. The documentation paints a very broad picture, however, so it's definitely worth reprinting as follows:
The RTC Client API enables you to build applications that can make PC-to-PC, PC-to-phone, or phone-to-phone calls or create IM sessions over the Internet. Both voice and video calls can be established on PC-to-PC calls. Presence information on a list of contacts is also supported. Application sharing and whiteboard can be added to enhance the communication capabilities of any type of session.
So what does the RTC API look like? No surprises here in that it's COM-based. The root interface for working with the RTC API is the IRTCClient, which you obtain using CoCreateInstance. From that interface, you can create (or provide) other interfaces such as IRTCSession, IRTCParticipant, IRTCBuddy, IRTCProfile, and so on. Altogether, there are more than two dozen RTC interfaces. If you were so inclined, you could create your own custom instant messaging client using the RTC interfaces.
Conclusion
With all I've shown, it's pretty clear that Windows Server 2003 is a significant improvement over Windows 2000. I've focused on the user-mode programming APIs, but there have also been major changes and additions under the hood, both in performance and reliability. I'm personally very excited to see functionality like vectored exception handling, side-by-side execution, and better support for debug symbols making their way into Windows.
Consider this: Windows XP is the newest client operating system from Microsoft. Since Windows Server 2003 is a superset of Windows XP in all ways, it makes sense to at least consider using the new APIs that Windows XP brings to the table. I'm personally running Windows XP on some of my machines and Windows Server 2003 on others, and I can't tell the difference in my everyday work. I hope that you'll take the jump and explore Windows Server 2003 for yourself.
For background information see:
Windows Server 2003
Matt Pietrekis a software architect and writer. He works for the Compuware/NuMega lab as a lead architect for the BoundsChecker and Distributed Analyzer products. He has authored three books on Windows system programming and is a contributing editor for MSDN Magazine. His Web site (https://www.wheaty.net) has an FAQ and information on previous articles and columns.