NTFS

Enhance Your Apps With File System Transactions

Jason Olson

This article discusses:

  • The Kernel Transaction Manager
  • The Distributed Transaction Coordinator
  • Using Transactional NTFS from native code
  • Using Transactional NTFS from managed code
This article uses the following technologies:
Windows Vista, .NET Framework

Code download available at: TxF 2007_07.exe(159 KB)

Contents

The Benefits of Transactional NTFS
When to Avoid Transactional NTFS
Getting Started
Secondary Resource Managers
Management
What Happened to the Transacted Command Line?
In Closing

It’s a revolutionary step for Windows®-based applications. And it can help you develop applications that are more robust than they could ever have been before. It’s Transactional NTFS, also known as TxF. In this article, I explain Transactional NTFS, why it matters to developers, and how you can use this new technology in your applications.

What is TxF? Conceptually, it is quite simple and can be expressed in a logical equation: Transactional NTFS = Transactions + NTFS. A new feature in Windows Vista™ and the next version of Windows Server®, code-named "Longhorn," TxF introduces the concept of transacting file operations.

When I say transacted file operations, I am talking about transactions with fully ACID semantics that can be exposed directly to the developer. This means the ability to perform actions that are fully atomic, consistent, isolated, and durable is now enabled directly for file operations. These ACID capabilities are something that database developers have benefited from for years, and having this power for file operations is a very welcomed addition.

You’re probably wondering if all these transactional capabilities have been built into NTFS, what is the performance overhead of Transactional NTFS? Well, TxF has a strictly pay-to-play model. If you aren’t using transacted file operations, there is no overhead. And even if you are using transacted file operations, the overhead is pretty minimal (in the 1 to 2 percent range). For an application that is not I/O-bound, this performance impact probably would not even be noticeable.

Here’s another aspect of performance to be aware of: Transactional NTFS is optimized for commit. When you write new pages to disk within a transaction, they are written directly in place while the previous pages are saved for retrieval by other readers if necessary. Because of this optimization, when the transaction is committed, the altered pages are already in place on disk. As a result, the overhead of committing the transaction is minimal.

The Benefits of Transactional NTFS

Transactional NTFS sounds nice, in theory. But why should you, as a developer, use TxF? There are a number of distinct reasons. Specifically, TxF achieves three goals that help you develop powerful applications:

  • Improved application stability
  • Improved platform stability
  • Increased innovation

Let’s take a closer look at how TxF achieves these goals.

Transactional NTFS enables better application stability by reducing or eliminating the amount of error-handling code that needs to be written and maintained for a given application. This ultimately reduces application complexity and makes the application easier to test. Say, for instance, you are developing a document management system where a SQL data source needs to be kept consistent with a file store on disk. Ensuring this consistency can be tricky and non-trivial in a non-transactional system. Without transactional file operations, it would be nearly impossible to account for every possible failure scenario, up to and including the operating system crashing at any imaginable point during the process.

One of the ways this was handled in the past was by storing the new version of the file with a temporary file name, writing the new data to the SQL database, and then renaming the temporary file to the real file name when committing the SQL transaction. But consider what happens if the application crashes or there is a power outage right after committing to the SQL database but before the file is renamed. Not only would you have an inconsistent data set, but you would also have an artifact on the file system that you would have to clean up at some point. As usual, the extremely difficult part lies in the details of how many different ways the process can fail.

Some solutions will implement their own entire two-phase commit protocol, which commits changes to the database and the file system at the same time. While this gets you a step closer to achieving the goal of application stability, implementing a custom two-phase commit system in your own application can be very difficult. And this requires code that you have to write, test, and maintain yourself.

Wouldn’t it be better if transaction management was built into the platform rather than having to be implemented by each developer who needs this type of behavior? With the introduction of TxF in Windows Vista and Windows Server "Longhorn," these capabilities are now built directly into the platform. And because Transactional NTFS is embedded in the system itself, it is capable of providing a level of integration that would not otherwise be possible for your applications.

One of the coolest parts about Transactional NTFS is that it can work with a large number of other transactional technologies. Because TxF uses the new Kernel Transaction Manager (KTM) features, and because the new KTM can work directly with the Microsoft® Distributed Transaction Coordinator (DTC), any technology that can work with DTC as a transaction coordinator can use transacted file operations within a single transaction. This means that you can now enlist transacted file operations within the same transaction as SQL operations, Web service calls via WS-AtomicTransaction, Windows Communication Foundation services via the OleTransactionProtocol, or even transacted MSMQ operations. You can even enlist transacted file operations directly with other technologies that are using XA-Transactions since DTC can also work with XA-Transactions. Simply put, if DTC can work with something, you can incorporate transacted file operations into it.

Using the document management example, how can you ensure consistency within a document management application more easily with TxF? This is where the DTC comes in handy. To absolutely ensure consistency between your SQL database and your file store, you can start a transaction, perform your SQL statements and file operations within that same transaction, and then commit or roll back the complete transaction based on the outcome. If your SQL call fails, your file will never be written. If your file system call fails, your SQL is rolled back. Everything remains consistent, and all of this is handled automatically by the platform since the operations are enlisted within a transaction.

When using this approach in my own application, I no longer have to worry about handling and accounting for every possible exception that could occur. I simply incorporate both operations within the same transaction, guaranteeing that if any exception is thrown, the platform will roll back the operations in question. Less code, more robust, very cool!

That leaves two more goals. As for platform stability, this is achieved by some of the ways that Microsoft is using TxF in its own technologies. There are three core features in Windows Vista and Windows Server "Longhorn" that now make use of Transactional NTFS: Windows Update, System Restore, and Task Scheduler. All of these use TxF to write files to the file system within the scope of a transaction in order to handle rollback/commit in case of any exceptions, such as a system reboot due to a loss of power. If you’ve ever experienced a loss of power right in the middle of Windows Update, you know this can leave your system dead and in need of a fresh installation. But now that Windows Update uses TxF, if this same situation occurs on your Windows Vista system, the computer can recover by rolling back the transaction when the system reboots. By adopting TxF internally, Microsoft has helped make its own operating system more stable.

Finally, TxF drives innovation by providing a framework for using transactions outside of SQL calls. Ultimately, Transactional NTFS can fundamentally change the way developers write applications, allowing them to build more robust code. By incorporating transactions into your design, you can write code without having to account for every single possible failure that can occur. The operating system will take care of those mundane details!

When to Avoid Transactional NTFS

Despite all the benefits offered by TxF, there are some scenarios in which you should think about not using it. These are the same scenarios that are discussed when talking about avoiding traditional database transactional systems.

You shouldn’t use TxF if you have a system that will have long-running transactions. The definition of "long-running" here is relative, however. A long-running transaction is any transaction that has been alive longer than many other transactions within the same log. This could mean a few seconds if there are thousands of transactions happening and lasting for very brief periods of time. On the other hand, a day or two might not be a long time if there are very few transactions happening in the system.

You should also be wary of using TxF if you are going to have multiple writers against the same file. Transacted file operations only allow for one writer at a time (even shared writers), meaning that a second writer won’t be allowed to open the file while it is being held open by another writer, unless that second writer is within the same transaction. Because of this, TxF can introduce contention into a system that has multiple writers against a single file.

Finally, you need to be careful about auto-started services and mixing transaction resources. If the service is using Transactional NTFS and SQL, the files it needs access to could remain locked until recovery completes (since your service has a direct dependency on DTC and SQL, those services need to be started first, so you might not be able to fully recover your files until DTC and SQL finish recovering first).

Getting Started

Now let’s take a look at how you can use TxF. If you are at all familiar with the existing file I/O Win32® APIs, the Transactional NTFS APIs should come naturally. The new Transactional NTFS APIs are unchanged from their non-transacted counterparts aside from the addition of a couple of parameters at the end, which are specifically for transactions. Figure 1 gives some examples of Win32 file APIs and their transacted counterparts. The complete list is much longer; nearly every file API in Win32 has a corresponding transacted version.

Figure 1 Win32 APIs and Their Transacted Counterparts

Non-Transacted API Transacted API
CreateFile CreateFileTransacted
CopyFileEx CopyFileTransacted
MoveFileWithProgress MoveFileTransacted
DeleteFile DeleteFileTransacted
CreateHardLink CreateHardLinkTransacted
CreateSymbolicLink CreateSymbolicLinkTransacted
CreateDirectoryEx CreateDirectoryTransacted
RemoveDirectory RemoveDirectoryTransacted

 

From an application perspective, there are three primary vehicles you can use when developing a TxF-aware application. You can use the new KTM directly as the transaction coordinator. This lets you develop applications that are solely using TxF file operations within their transactions.

The second option is to use DTC as the transaction coordinator. When you use this method, you need to go through DTC to get the handle for the kernel-level transaction object that you can then pass into the Transacted APIs.

The third way is actually an extension to the second approach where you are using System.Transactions from managed code to make use of DTC (or just the Lightweight Transaction Manager if there is only a single durable enlistment). In a moment, I’ll look at the challenges of this approach and what it really means to managed developers.

Right now, let’s take a closer look at these three approaches. Figure 2 illustrates all the major moving parts in the transaction platform in Windows Vista and Windows Server "Longhorn."

Figure 2 The Transaction Platform

Figure 2** The Transaction Platform **(Click the image for a larger view)

At the API level, there are three main interfaces into TxF. The first way is to go directly to KTM via the new ktmw32.h header file. At the most basic level, this contains the transaction coordinator Win32 calls for KTM: CreateTransaction, CommitTransaction, and RollbackTransaction. In this approach, you use these calls directly and have KTM itself do the transaction coordination if you are only doing transacted file operations within the single transaction (see Figure 3).

Figure 3 TxF from Native Code with KTM

hTx = CreateTransaction(NULL, NULL, 0, 0, 0, 0, L”MyTxn”); 
hFile = CreateFileTransacted(myFileName,
            GENERIC_READ | GENERIC_WRITE,
            FILE_SHARE_READ | FILE_SHARE_WRITE,
            NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL,
            hTx, NULL, NULL);

... // work with the file using its file handle hFile

CommitTransaction(hTx);

The second interface is via DTC. With this approach, you use DTC as the transaction coordinator. To use TxF through DTC, you make use of COM and QueryInterface to grab the IKernelTransaction interface directly from DTC’s ITransaction interface. Once you have a reference to the new IKernelTransaction COM interface, you can call its GetHandle method in order to retrieve the kernel-level transaction handle that can then be used to pass to the various Transacted APIs in Win32 (see Figure 4).

Figure 4 TxF from Native Code with DTC

hr = DtcGetTransactionManagerEx( NULL, NULL,
         IID_ITransactionDispenser, OLE_TM_FLAG_NONE, 
         NULL, (void**) &pITransactionDispenser);
hr = pITransactionDispenser->BeginTransaction(
         NULL, ISOLATIONLEVEL_READCOMMITTED,
         ISOFLAG_RETAIN_BOTH, NULL, &pITransaction);
hr = pITransaction->QueryInterface(
         IID_IKernelTransaction, 
         (void**) &pKernelTransaction);
hr = pKernelTransaction->GetHandle(&hTransactionHandle);
hAppend = CreateFileTransacted(TEXT(“test.txt”),
              FILE_APPEND_DATA, FILE_SHARE_READ,
              NULL, OPEN_ALWAYS,
              FILE_ATTRIBUTE_NORMAL, NULL,
              hTransactionHandle,    NULL, NULL);     

... // work with the file using its file handle hAppend

hr = pITransaction->Commit(FALSE, XACTTC_SYNC_PHASEONE, 0);

If you are a managed developer, the solution isn’t quite as elegant. Currently, to use TxF from managed code, you have to use P/Invoke and COM interop. Don’t let that scare you away, though. For a large number of uses, the P/Invoke and COM interop code isn’t too bad. If you’re still too timid, you can grab some sample code from the MSDN® Magazine Web site. The sample has a simple managed wrapper around Transactional NTFS that you can freely use in your own applications. (While you’re playing with this sample code, you should also check out the screencast. It demonstrates using TxF with just KTM, with SQL, and with Windows Communication Foundation.) In the short term, a more robust version of this wrapper will be included in an upcoming release of the Windows VistaBridge samples as part of the Windows SDK. In the longer term, the product team is currently working on a plan to integrate these APIs directly into the Microsoft .NET Framework.

To use TxF from managed code, you can continue to use the functionality of the System.Transactions namespace. To dig down into TxF, you are essentially following the same path that you would when using DTC in native code (see Figure 4), but with the managed code equivalents. From an object level, it looks something like Figure 5.

Figure 5 TxF from Managed Code

Figure 5** TxF from Managed Code **

The first thing you need to do is grab the IDtcTransaction COM interface from the current Transaction (this can be done using the TransactionInterop helper class in System.Transactions). Once you have the IDtcTransaction, you can cast it to an IKernelTransaction COM interface, which under the hood will QueryInterface for the IKernelTransaction COM interface. Then, once you have a reference to the currently running kernel-level transaction via IKernelTransaction, you call GetHandle on it to get a handle that you can pass to the various new Transacted Win32 APIs. For example, you can pass this transaction handle to CreateFileTransacted in order to obtain a SafeFileHandle for a new file. This SafeFileHandle can then be passed to the constructor of the StreamWriter class, which is then used to write to the file (see Figure 6).

Figure 6 TxF from Managed Code with System.Transactions

NativeMethods.cs

    // IKernelTransaction COM Interface
[Guid('79427A2B-F895-40e0-BE79-B57DC82ED231')]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IKernelTransaction
{
    int GetHandle(out IntPtr pHandle);
}

[DllImport(KERNEL32, 
   EntryPoint = 'CreateFileTransacted', 
   CharSet = CharSet.Unicode, SetLastError = true)]
internal static extern SafeFileHandle CreateFileTransacted(
   [In] string lpFileName,
   [In] NativeMethods.FileAccess dwDesiredAccess,
   [In] NativeMethods.FileShare dwShareMode,
   [In] IntPtr lpSecurityAttributes,
   [In] NativeMethods.FileMode dwCreationDisposition,
   [In] int dwFlagsAndAttributes,
   [In] IntPtr hTemplateFile,
   [In] KtmTransactionHandle hTransaction,
   [In] IntPtr pusMiniVersion,
   [In] IntPtr pExtendedParameter);
...

Program.cs

...
using (TransactionScope scope = new TransactionScope())
{
   // Grab Kernel level transaction handle
   IDtcTransaction dtcTransaction = 
      TransactionInterop.GetDtcTransaction(managedTransaction);
   IKernelTransaction ktmInterface = (IKernelTransaction)dtcTransaction;

   IntPtr ktmTxHandle;
   ktmInterface.GetHandle(out ktmTxHandle);

   // Grab transacted file handle
   SafeFileHandle hFile = NativeMethods.CreateFileTransacted(
      path, internalAccess, internalShare, IntPtr.Zero,
      internalMode, 0, IntPtr.Zero, ktmTxHandle,
      IntPtr.Zero, IntPtr.Zero);

   ... // Work with file (e.g. passing hFile to StreamWriter constructor)

   // Close handles
}
 

You should also know that TxF has read-committed consistency. This has impact on both writing and reading a file transactionally. When writing a file, only a single transaction can be writing to the file (note that we are talking about a single transaction, not a single writer). Non-transacted writers are blocked by the transacted writer, even if opened with share-write access. Additionally, a non-transacted writer prevents any transacted reader or writer from being opened on the file.

When reading from a file transactionally, there is an isolated view of the file for the life of the file handle. This isolated view of the file is based on the most recent contents of the file when the file handle was created. If a transacted writer is in the middle of modifying the file, those modifications won’t be picked up by a new transacted reader until the changes are actually committed. In addition, because of the read-committed consistency of Transactional NTFS, this isolated view of the transacted reader remains even if a transacted writer changes a file and commits those changes anytime during the life of the transacted reader. These transacted readers block non-transacted writers but only while their file handle is open. They do not block modification of the file for the life of the transaction. If you need a consistent view of the file throughout the life of the transaction, you need to keep the transacted reader’s file handle open.

Finally, developers should be aware that when using file handles for transacted file operations, there is a distinct difference between transacted and non-transacted file operations with regard to file handles. If you get a file handle from one of the new Transacted APIs, the file handle could become invalid at any time, whether from a transaction committing or a transaction rolling back.

Secondary Resource Managers

Secondary resource managers allow applications to manage their own resources running within a transaction, particularly their own transaction logs, as well as other possible file operations within a directory (and all subdirectories) specific to the application.

I have a good reason for discussing secondary resource managers in this article. When TxF was developed, there was an emphasis placed on availability over consistency. Since it is now possible within a transaction to modify files that are necessary to boot, the operating system has to guarantee that these files are always available. In order to guarantee these files are always available, the system drive’s transaction manager is required to be the transaction coordinator. Because of this, you cannot use a different transaction coordinator, like DTC for instance, as the "root" or superior transaction coordinator for these files. As a result, a transaction that needs to use DTC with other resource managers (such as SQL) cannot use TxF on the system drive. This means that if you want to use Transactional NTFS on the system drive within the same transaction as operations against SQL Server™, for example, you must use a different strategy.

The simplest way to get around this limitation is to place all of your data on a separate partition. If that is not possible, or you cannot guarantee that the data will be on a non-system partition, you can use a secondary resource manager. When using a secondary resource manager to manage transactional file operations for a given directory (and any subdirectories), you are informing KTM that the application will control resource management for any files residing there. This allows the default resource manager on the system volume to continue to be available and, hence, allows a transaction coordinator other than KTM to become the superior transaction coordinator.

At the command line, you can create and manage secondary resource managers using the FSUtil tool and the commands shown in Figure 7. However, for applications that need to use a secondary resource manager, it is better to control the secondary resource manager programmatically. To do this, you can use the DeviceIoControl Win32 API with specific dwIoControlCodes, outlined in Figure 8.

Figure 8 Manage Secondary Resource Managers Programmatically

Control Codes Purpose
FSCTL_TXFS_LIST_TRANSACTIONS Lists all transactions currently involved in the specified resource manager.
FSCTL_TXFS_LIST_TRANSACTION_LOCKED_FILES Lists all files currently locked by the specified transaction.
FSCTL_TXFS_GET_METADATA_INFO Retrieves Transactional NTFS metadata for a file, as well as the ID of the transaction that has the file locked.

Figure 7 Manage Secondary Resource Managers from the Command Line

Command Purpose
fsutil resource create Creates a secondary transactional resource manager.
fsutil resource info Displays information relating to a transactional resource manager.
fsutil resource setautoreset Sets whether a transactional resource manager will clean its transactional metadata the next time it starts.
fsutil resource setlog Changes characteristics of a running transactional resource manager.
fsutil resource start Starts a transactional resource manager.
fsutil resource stop Stops a transactional resource manager.

 

When using secondary resource managers, the startup and recovery of the resource manager must be done by your application. The act of starting the resource manager creates the KTM objects to manage transactions, and opens up the Common Log File System (CLFS) log and metadata that is stored locally with your directories; the act of recovering the resource manager recovers all files to a consistent state within the secondary resource manager. The resource manager will stop when the last handle to the resource manager root goes away. As a result, files within a secondary resource manager are not automatically made consistent at system startup. Instead, that is done when your application starts and recovers the secondary resource manager. In addition, files are not automatically made consistent when the system is backed up with any software that uses the volume shadow copy service (VSS) unless you provide a VSS writer. Of course, if you restore an inconsistent secondary resource manager and then start and recover it, the files will be made consistent. Make sure that your backup software also backs up the TxF metadata using the FSCTL_TXFS_READ_BACKUP_INFORMATION and FSCTL_TXFS_WRITE_BACKUP_INFORMATION dwIoControlCodes.

As I mentioned, the use of secondary resource managers is only required if the files participating in the transaction are on the system drive and you want a technology other than KTM to be the superior transaction coordinator or you want to control the consistency and availability of the resource manager. Secondary resource managers become very powerful in this regard if you wish to have more direct control over the behavior of the resource manager in your application. If the files you are operating on are on a non-system drive, you do not need to worry about the use of secondary resource managers. We are currently working to lessen this restriction in Windows Server "Longhorn" so this requirement won’t be applied to the entire system drive, but only to the parts of the system drive that Windows absolutely needs to boot.

Management

For any application that is going to live in a production environment, it is important to think about how the application will be managed. Therefore, you need to know how you can monitor TxF itself. Figure 9 shows a list of control codes that can be used with DeviceIoControl to help you do just that.

Similar to the information on resource managers provided previously, you can get the same information as these control codes from the FSUtil command-line tool. The commands are as follows:

  • fsutil transaction list
  • fsutil transaction fileinfo
  • fsutil transaction query

What Happened to the Transacted Command Line?

If you were involved in the early betas for Windows Vista, you may recall seeing a transaction command available from the command line. This allowed an administrator to start a transaction from the command line and run an application, and then any file operations within that application would implicitly be transacted.

While testing application compatibility in Windows Vista, the team found that this implicit model was difficult for COM+ and managed developers to use safely. As a result, the model was changed to be opt-in and explicit to avoid this confusion. The downside to this change is that if you wish to use TxF in your app, your code will have to change to call the new APIs.

In Closing

Transactional NTFS is a truly innovative technology. It helps developers create more reliable applications on a more powerful platform. With TxF, developers can have atomic file operations that result in consistent file data, easing the test burden by reducing the number of error conditions to test for, and assisting in multi-user environments by providing isolation via ACID transaction semantics. Consider using TxF when developing apps that target Windows Vista and Windows Server "Longhorn." For more information, see the "Transaction Resources" sidebar.

I would like to thank Dana Groff, Christian Allred, and Jon Cargille for their help with this article.

Transaction Resources

Here’s a list I’ve compiled of resources relating to Transactional NTFS.

Web Sites:

Blogs:

Media:

Jason Olson is a Technical Evangelist at Microsoft working on Windows Server "Longhorn" in the Developer and Platform Evangelism division. Outside of work, he is a loving and dedicated husband, hobbyist game developer. He can be reached at jason.olson@microsoft.com.