Win32 I/O Cancellation Support in Windows Vista

  • Article

 

Microsoft Corporation

September 2005

Gerald Maffeo and Paul Sliwowicz
Microsoft Corporation

Abstract

This whitepaper is intended for Independent Software Vendors and others interested in leveraging new cancellation support to improve the robustness of their Windows applications. This enables improved customer experience through direct cancellation of and recovery from slow or blocked file input/output.

Windows Vista provides kernel and driver enhancements, as well as new Win32 application programming interfaces, to enable extended cancellation of native Win32 file management functions. These include new support for canceling synchronous operations.

This whitepaper provides developers with sufficiently detailed information to start building support for file I/O cancellation into their applications.

Contents

Introduction
Background
Cancellation Enhancements for Vista
Using the New File I/O Cancellation APIs
Canceling Asynchronous I/O
Canceling Synchronous I/O
Summary
Related Links
Appendix A: New I/O Cancellation APIs
Appendix B: Cancelable File I/O APIs

Introduction

We discuss in this whitepaper new cancellation support for Win32 file input/output (I/O) operations in Windows Vista and provide detailed information to enable their use in native (unmanaged) applications.

Note that the discussions in this paper are only applicable to native code. Cancellation in the .Net Framework libraries (WinFX) is not currently available to support cancellation of low-level file I/O.

Background

Application Termination Failures

Windows applications sometimes fail to terminate completely after users try to close them (for instance, by clicking the Red-X on the title bar). An application in these cases appears to close, but further inspection shows that the application's process persists in the process list (this is known as a "zombie" process). Attempts to kill the zombie are unsuccessful. This often blocks attempts to re-launch the application or may cause unpredictable behavior. It is often necessary to reboot the computer in order to recover.

We had many anecdotal reports of this behavior and suspected that faulty drivers were the cause, but we needed hard data to determine the underlying causes and how to address fixing them. In order to gather this data, we developed a specially instrumented version of Windows XP that would collect and report kernel data, including driver requests that were still pending for the stuck process. We deployed this instrumentation to both Microsoft internal users and to selected customers through a special Technical Beta Program.

These reports confirmed not only what we expected, but also that a large fraction of termination failures and also application unresponsiveness resulted from the lack of Window's and drivers' ability to cancel driver create requests. Changes would be needed in Windows to address these failures.

Cancellation Support for Applications

File operations can legitimately block an application's thread from continuing. A well-designed, responsive application comprises at least two threads, one that services the user interface (UI) and that does not call, directly or indirectly, any APIs that might cause it to become unresponsive. One or more "worker" threads are invoked to carry out blocking operations.

This then enables an application to support cancellation through a Cancel or Stop button as well as a progress indicator. In many cases, it is enough to simply ignore (abandon) the worker thread if the user decides to abort the operation. For instance, a call to the domain name service (DNS) resolver or other network call will eventually time out, so a worker thread can subsequently clean up after itself.

There are, however, other situations for which it is necessary or desirable to support actual I/O cancellation. As an example, stopping an OpenFile call to a very slow device, especially if it is necessary to quickly attempt the call again, perhaps with different parameters. The users' only alternative to waiting—perhaps indefinitely—is to try to terminate the application (which, as we saw above, may itself fail). Clearly, providing customers with the ability to cancel the request is far more preferable! This again requires Windows and its drivers to fully support cancellation, including support for synchronous cancellation.

Cancellation Enhancements for Vista

Kernel, I/O Manager Additions

The ability to cancel driver create requests (IRP_MJ_CREATE) is supported in Vista. Win32 APIs that map to driver create requests include CreateFile and OpenFile.

Note that this does not guarantee that these requests will be honored by underlying drivers. Drivers that can take a long time to complete create requests must implement cancellation support. We developed driver I/O completion/cancellation guidelines as well as a new option for Vista's driver verifier v2 to support testing completion and cancellation behaviors (see Driver Hang Verifier in Related Links).

Microsoft-provided drivers support create-request cancellation where needed, notably the Microsoft Multi-UNC Provider (MUP) and Redirector.

New File I/O Cancellation APIs

Limited I/O cancellation support exists in Windows XP and Windows Server 2003 File I/O APIs using the CancelIo function, which cancels pending asynchronous I/O issued by the calling thread for a specified file handle. However, CancelIo is very limited, since it does not support canceling

  • I/O operations issued for the file handle by other threads;
  • Specific I/O operations;
  • Synchronous I/O operations, since they (by definition) block in the operating system;
  • I/O issued to a completion port or associated with an IOSB range.

New Win32 cancellation APIs have therefore been implemented for Longhorn. Unlike CancelIo, the new CancelIoEx and CancelSynchronousIo APIs only mark pending I/O requests for cancellation and then return immediately. Some key points arising from this:

  • There is an inherent race condition between the cancellation and completion, since it is possible for the I/O request to complete normally, despite the request to cancel it;
  • It is necessary to verify whether the operation completed successfully or was cancelled (when you cancel a request using cancellation APIs, it ultimately completes);
  • The status code of the completion will indicate whether the operation completed successfully, was canceled (ERROR_OPERATION_ABORTED), or failed with some other error.

Prototypes for these new APIs are available—see Related Links. More complete information is provided in Appendix A: New I/O Cancellation APIs.

Cancelable File I/O APIs

Most file I/O APIs are cancelable.

A small number of complex Win32 APIs, such as CopyFile are not cancelable. Cancelable forms of these APIs are provided (e.g. CopyFileEx) and should be used. In addition to supporting cancellation, these APIs also have built-in callbacks to support tracking the call's progress.

Please refer to Appendix B for a list of cancelable file I/O APIs.

Using the New File I/O Cancellation APIs

Canceling Asynchronous I/O

You can cancel asynchronous I/O from any thread in the process that issued the I/O. You simply need to supply the handle that the I/O was performed on and, optionally, the overlapped structure that was used to perform the I/O. The following sample shows a routine that takes a timeout and attempts a read, canceling it with CancelIoEx if the timeout expires. Note that calling CancelIoEx does not guarantee that the read will be cancelled; the driver that is handling it must support it and it must be in a cancelable state.

In the asynchronous case you can tell if a cancel actually took place by examining the status when you retrieve the overlapped structure or in the completion callback. A status of ERROR_OPERATION_ABORTED will indicate the operation was cancelled.

BOOL DoCancelableRead(HANDLE hFile,
                 LPVOID lpBuffer,
                 DWORD nNumberOfBytesToRead,
                 LPDWORD lpNumberOfBytesRead,
                 LPOVERLAPPED lpOverlapped,
                 DWORD dwTimeout,
                 LPBOOL pbCancelCalled)
//
// Parameters:
//
//      hFile - An open handle to a readable file or device.
//
//      lpBuffer - A pointer to the buffer to store the data being read.
//
//      nNumberOfBytesToRead - The number of bytes to read from the file or
//          device. Must be less than or equal to the actual size of
//          the buffer referenced by lpBuffer.
//
//      lpNumberOfBytesRead - A pointer to a DWORD to receive the number
//          of bytes read after all I/O is complete or cancelled.
//
//      lpOverlapped - A pointer to a preconfigured OVERLAPPED structure that
//          has a valid hEvent.
//          If the caller does not properly initialize this structure, this
//          routine will fail.
//
//      dwTimeout - The desired time-out, in milliseconds, for the I/O read.
//          After this time expires, the I/O is cancelled.
//
//      pbCancelCalled - A pointer to a Boolean to notify the caller if this
//          routine attempted to perform an I/O cancel.
//
// Return Value:
//
//      TRUE on success, FALSE on failure.
//
{
    BOOL result;
    DWORD waitResult;
    DWORD waitTimer;
    BOOL bIoComplete = FALSE;
    const DWORD PollInterval = 100; // milliseconds

    // Initialize "out" parameters
    *pbCancelCalled = FALSE;
    *lpNumberOfBytesRead = 0;

    // Perform the I/O, in this case a read operation.
    result = ReadFile(hFile,
                  lpBuffer,
                  nNumberOfBytesToRead,
                  lpNumberOfBytesRead,
                  lpOverlapped );

    if (result == FALSE)
    {
        if (GetLastError() != ERROR_IO_PENDING)
        {
            // The function call failed. ToDo: Error logging and recovery.
            return FALSE;
        }
    }
    else
    {
        // The I/O completed, done.
        return TRUE;
    }

    // The I/O is pending, so wait and see if the call times out.
    // If so, cancel the I/O using the CancelIoEx function.

    for (waitTimer = 0; waitTimer < dwTimeout; waitTimer += PollInterval)
    {
        result = GetOverlappedResult( hFile, lpOverlapped, lpNumberOfBytesRead, FALSE );
        if (result == FALSE)
        {
            if (GetLastError() != ERROR_IO_PENDING)
            {
                // The function call failed. ToDo: Error logging and recovery.
                return FALSE;
            }
            Sleep(PollInterval);
        }
        else
        {
            bIoComplete = TRUE;
            break;
        }
    }

    if (bIoComplete == FALSE)
    {
        result = CancelIoEx( hFile, lpOverlapped );

        *pbCancelCalled = TRUE;

        if (result == TRUE || GetLastError() != ERROR_NOT_FOUND)
        {
            // Wait for the I/O subsystem to acknowledge our cancellation.
            // Depending on the timing of the calls, the I/O might complete with a
            // cancellation status, or it might complete normally (if the ReadFile was
            // in the process of completing at the time CancelIoEx was called, or if
            // the device does not support cancellation).
            // This call specifies TRUE for the bWait parameter, which will block
            // until the I/O either completes or is cancelled, thus resuming execution,
            // provided the underlying device driver and associated hardware are functioning
            // properly. If there is a problem with the driver it is better to stop
            // responding, or "hang," here than to try to continue while masking the problem.

            result = GetOverlappedResult( hFile, lpOverlapped, lpNumberOfBytesRead, TRUE );

            // ToDo: check result and log errors.
        }
    }

    return result;
}

Canceling Synchronous I/O

You can cancel synchronous I/O from any thread in the process that issued the I/O. To cancel synchronous I/O, supply a handle to the thread that is currently performing the I/O.

The following sample shows two routines:

  • SynchronousIoWorker is a worker thread that implements some synchronous file I/O, starting with a call to CreateFile. If successful, it is followed by additional operations (not shown). Global variable gCompletionStatus can be used to determine whether all operations succeeded or whether an operation failed or was canceled. Global variable, dwOperationInProgress, indicates whether the file I/O is still in progress. (In this example, the UI thread could also check the existence of the worker thread.) Note that additional manual checks (not shown in this example) are required in SynchronousIoWorker to ensure that if the cancellation was requested during the brief periods between file I/O calls, the rest of the operations will be canceled.
  • MainUIThreadMessageHandler simulates the message handler within a UI thread's window procedure. The user requests a set of synchronous file operations by clicking a control, which generates user-defined window message WM_MYSYNCOPS. The code for this results in creation of a new thread with the routine CreateFileThread, which starts SynchronousIoWorker. The UI thread continues to process messages while the worker thread does the requested I/O. If the user decides to cancel the unfinished operations (usually by pressing a cancel button), the code for WM_MYCANCEL simply calls CancelSynchronousIo using the thread handle returned by CreateFileThread. CancelSynchronousIo returns immediately after attempting cancellation; there is no guarantee the create will be cancelled. Finally, the user or application may later request some other operation that depends on whether the file operations have completed. In this case, the sample code at WM_PROCESSDATA first verifies that the operations completed and then executes the clean-up code. In this example, since the cancellation could have occurred anywhere in the sequence of operations, it may be necessary for the caller to ensure that the state is consistent or at least understood before proceeding.
DWORD gCompletionStatus;      // Completion status for last operation
DWORD dwOperationInProgress;   // TRUE if operation is in progress

VOID
SynchronousIoWorker( VOID * pv )
{
    LPCSTR lpFileName = (LPCSTR) *pv;
    HANDLE hEvent = lpCreateParams->hEvent;
    HANDLE hFile;

    dwOperationInProgress = TRUE;
    gCompletionStatus = ERROR_SUCCESS;
    hFile = CreateFileA( lpFileName,
                         GENERIC_READ,
                         0,
                         NULL,
                         OPEN_EXISTING,
                         0,
                         NULL );


    if (hFile != INVALID_HANDLE_VALUE) {
  
        /* TODO: CreateFile succeeded. Do more synchronous calls with hFile */
        
        if (result == FALSE) {
            // Some operation failed or was cancelled. If cancelled,
            // GetLastError() will return ERROR_OPERATION_ABORTED

            gCompletionStatus = GetLastError();            
        }
             
        CloseHandle(hFile);

    } else {
        // CreateFile failed or was cancelled. If cancelled,
        // GetLastError() will return ERROR_OPERATION_ABORTED
        gCompletionStatus = GetLastError();
    }

    dwOperationInProgress = FALSE;
}



LRESULT
CALLBACK
MainUIThreadMessageHandler(
    HWND hwnd,
    UINT uMsg,
    WPARAM wParam,
    LPARAM lParam
)
{
    HANDLE syncThread;

    /* TODO: Miscellaneous initializations, etc. */

    switch (uMsg) {

    case WM_MYSYNCOPS:   // User requested an operation on a file
        /* TODO: Retrieve filename from parameters */

        syncThread = CreateThread( NULL,
                                0,
                                (LPTHREAD_START_ROUTINE)SynchronousIoWorker,
                                &lpFileName,
                                0,
                                NULL );

        if (syncThread == INVALID_HANDLE_VALUE) {
            /* TODO: Handle failure */
        }
        break;
   
    // User clicked a cancel button
    case WM_MYCANCEL:
        if (syncThread != INVALID_HANDLE_VALUE) {
            CancelSynchronousIo(syncThread);
        }
        Break;

    case WM_PROCESSDATA:
        if (!dwOperationInProgress) {
            if (gCompletionStatus == ERROR_OPERATION_ABORTED) {
                /* TODO: Do appropriate cleanup */
            } else if (...) {
                /* TODO: Handle other cases */
            }
        }
        break;
   
    case ...:
        /* TODO: Handle any other cases */
    }

    return TRUE;
}

Cancellation Issues

Faulty Drivers

There is unfortunately no guarantee that underlying drivers correctly support cancellation. Even though a cancellation is correctly requested, the driver could block the requested operation from completing. To make matters worse, an application with correctly functioning cancellation support might suddenly stop responding to cancellation requests after a system reconfiguration, for instance after installation or update of drivers, such as anti virus or firewall filter drivers.

Potential Race Conditions

When canceling I/O operations on other threads, there are synchronization issues that a developer needs to be aware of.

In the asynchronous case, where no overlapped structure is supplied, a call to CancelIoEx will attempt to cancel all outstanding I/O on the file on all threads in the process. Note that it processes each thread individually, so after a thread has been processed it may start another I/O on the file before all the other threads have had their I/O for the file cancelled.

In addition, developers need to be careful when combining reuse of overlapped structures with targeted cancellation. It is not safe to reuse the structure right after calling CancelIoEx. Once the I/O operation completes (either successfully or with a cancelled status) then the overlapped structure is no longer in use by the system and can be reused.

In the synchronous case, a call to CancelSynchronousIo will attempt to cancel any current synchronous call on the thread. Without careful synchronization, the wrong call in a series of calls could get cancelled. These are very difficult to diagnose, since they are unpredictable and tend to occur rarely. A possible scenario is that CancelSynchronousIo is called. Although intended for a given synchronous operation, it only happens to start just after that operation completed (normally or with an error). If the thread that called the first operation then starts another synchronous call, the cancel call could interrupt this new I/O. This can have enormous consequences!

Another similar race condition can exist for synchronous I/O whenever a thread is shared between different parts of an application, for instance with a thread pool thread.

As an example, assume sub-app 1 calls CreateFile on thread 2 and issues cancellations from thread 1. Sub-app 2 also shares thread 2 and uses it to call Flush, which it cancels from another thread. If these are uncoordinated, then the cancellation requests could result in cancellation of the wrong operation.

Summary

Cancellation support has been improved in the Windows Vista kernel and many drivers. New Win32 application programming interfaces are available that enable synchronous file requests to be canceled. These are designed to improve Vista customer satisfaction by improving the reliability of application terminations. They are further designed to enable improved user experience through support of direct cancellation of and recovery from slow or blocked file I/O.

This whitepaper, especially its references and appendices, provides developers with sufficiently detailed information to start building support for file I/O cancellation into their applications.

See the following resources for further information:

Appendix A: New I/O Cancellation APIs

The information contained in this section provides additional information on the new cancellation APIs in Vista (beyond what is available in the Related Links). Note: This information is preliminary and subject to change.

CancelIoEx

Note This documentation is preliminary and is subject to change.

The CancelIoEx function causes all pending I/O operations for the specified file handle in the current process to be marked as canceled, regardless of which thread issued it.

BOOL CancelIoEx(
  HANDLE hFile,
  LPOVERLAPPED lpOverlapped
);

Parameters

  • hFile
    [in] Handle to the file whose operations are to be canceled.

  • lpOverlapped
    [in] Pointer to an OVERLAPPED data structure that supplies data used for asynchronous (overlapped) I/O.

    If lpOverlapped is not NULL, only requests that were issued with the supplied overlapped structure are marked as canceled; otherwise all I/O requests for the file handle are canceled.

Return Values

If the operation succeeds, the return value is nonzero.

If the function fails, the return value is zero. To get extended error information, call GetLast Error.

Remarks

CancelIoEx differs from CancelIo in that CancelIo will only mark requests canceled that were issued by the same thread that calls CancelIo.

If there are any pending I/O operations in progress for the specified file handle, the CancelIoEx function marks them for cancellation. Most types of operations can be canceled immediately, while others may continue toward completion before they are actually canceled and the caller is notified. CancelIoEx does not wait for all the canceled operations to complete.

The operation being cancelled will ultimately complete. It is necessary to check the completion status to determine whether the operation completed normally (which can occur despite the cancellation request); whether the operation was cancelled (with completion status ERROR_OPERATION_ABORTED); or whether the operation failed with some other error.

Requirements

Client Requires Windows Vista.
Server Requires Windows Server Vista.
Header Declared in Winbase.h.
Library Link to Kernel32.lib.
DLL Requires Kernel32.dll.

CancelSynchronousIo

Note This documentation is preliminary and is subject to change.

The CancelSynchronousIo function causes pending synchronous I/O operations issued by the specified thread to be marked as canceled.

BOOL CancelSynchronousIo(
  HANDLE hThread
);

Parameters

  • hThread
    [in] Handle to the thread that is blocked on synchronous I/O.

Return Values

If the operation succeeds, the return value is nonzero.

If the function fails, the return value is zero. To get extended error information, call GetLast Error.

Remarks

Pending synchronous I/O operations issued by a specified thread are marked as canceled. Most types of operations can be canceled immediately, while others may continue toward completion before they are actually canceled and the caller is notified. CancelSynchronousIoEx does not wait for all the canceled operations to complete.

The operation being cancelled will ultimately complete. It is necessary to check the completion status to determine whether the operation completed normally (which can occur despite the cancellation request); whether the operation was cancelled (with completion status ERROR_OPERATION_ABORTED); or whether the operation failed with some other error. This service is most useful when a create request has been issued that needs to be canceled, since the user will not yet have a handle to the file.

Requirements

Client Requires Windows Vista.
Server Requires Windows Server Vista.
Header Declared in Winbase.h.
Library Link to Kernel32.lib.
DLL Requires Kernel32.dll.

Appendix B: Cancelable File I/O APIs

API Name Cancelable? Cancellation Mechanism
WalkTree Partially WalkTree itself does not support cancellation, but it uses an enumeration callback to do the "walking." This callback could be used to support cancellation and stop enumeration if canceled.
DeleteTree No Callback for WalkTree that does not support cancellation.
AreFileApisANSI N/A  
CheckNameLegalDOS8Dot3 N/A  
CloseHandle N/A  
CopyFile No Use CopyFileEx.
CopyFileEx Yes Can cancel with progress routine by returning PROGRESS_CANCEL or can pass in variable pbCancel and set to TRUE elsewhere to cause cancellation.
CreateFile Yes  
CreateHardLink Yes  
DeleteFile Yes  
FindClose Yes  
FindFirstFile Yes  
FindFirstFileEx Yes  
FindFirstStreamW Yes  
FindNextFile Yes  
FindNextStreamW Yes  
GetBinaryType Yes  
GetCompressedFileSize Yes  
GetFileAttributes Yes  
GetFileAttributesEx Yes  
GetFileInformationByHandle Yes  
GetFileSize Yes  
GetFileSizeEx Yes  
GetFileTime Yes  
GetFileType Yes  
GetFullPathName Yes  
GetLongPathName Yes  
GetShortPathName Yes  
GetTempFileName Yes  
GetTempPath Yes  
MoveFile No Use MoveFileWithProgress.
MoveFileEx No Use MoveFileWithProgress.
MoveFileWithProgress Yes Progress routine can return PROGRESS_STOP or PROGRESS_CANCEL to stop the operation; does not support Boolean cancel like CopyFile though.
ReplaceFile No Does a handful of stream copies that are not cancelable.
SearchPath Yes  
SetFileApisToANSI N/A  
SetFileApisToOEM N/A  
SetFileAttributes Yes  
SetFileSecurity Yes  
SetFileShortName Yes  
SetFileTime Yes  
SetFileValidData Yes  
CreateIoCompletionPort N/A  
FlushFileBuffers Yes  
GetQueuedCompletionStatus Yes User-mode wait can be interrupted.
LockFile Yes  
LockFileEx Yes  
PostQueuedCompletionStatus N/A  
ReadFile Yes  
ReadFileEx Yes  
ReadFileScatter Yes  
SetEndOfFile Yes  
SetFilePointer Yes  
SetFilePointerEx Yes  
UnlockFile Yes  
UnlockFileEx Yes  
WriteFile Yes  
WriteFileEx Yes  
WriteFileGather Yes  
CreateFileMapping Yes  
FlushViewOfFile Yes  
MapViewOfFile Yes  
MapViewOfFileEx Yes  
OpenFileMapping N/A  
UnmapViewOfFile N/A  

 

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred. © 2004 Microsoft Corporation. All rights reserved.

Microsoft, Windows, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.