CCR Failure Handling

Glossary Item Box

CCR Failure Handling

Traditional failure handling schemes follow these patterns:

1. For synchronous invocations of methods, the caller checks one or more return values from the method. The method uses the callers execution context, most often this is a thread, to run.
2. Using structured exception handling, the caller wraps synchronous method invocations in try/catch/finally statements and either relies exclusively on the catch{} block for error handling. It may also use a combination of the catch{} block plus explicit checks on the results from the method.
3. Transactions which rely on a variety of compensation implementations by the components being called, OS infrastructure and usually expensive mechanisms to flow context across threads.

All methods above can not easily apply to concurrent, asynchronous execution, especially the first two which are simply not available for asynchronous programming. Methods execute in arbitrary context, potentially in parallel with the caller. Unless there is a blocking operation to collect the result of the asynchronous operation, thus blocking a thread in the callers context. In this case, failure or success can not easily be determined. The asynchronous programming model in Common Language Runtime (CLR), using the Begin/End calls introduces additional complexity since failure handling happens through additional calls and exception handling in the single callback that executes regardless of the result of the operation. The code that started the asynchronous operations is not the place that failure is handled, making code hard to read.

The CCR addresses failure handling with two approaches:

1. Explicit or local failure handling using the Choice and MultipleItemGather arbiters. Combined with iterator support they provide a type safe, robust way to deal with failure since they force you to deal with the success and failure case in two different methods and then also have a common continuation. Examples 8,13 and 20 show explicit error handling using Choice, MultipleItemGather and Choice in an iterator, respectively.
2. Implicit or distributed failure handling, referred to as Causalities that allows for nesting and extends across multiple asynchronous operations. It shares in common with transactions the notion of a logical context or grouping of operations and extends it for a concurrent asynchronous environment. This mechanism can also be extended across machine boundaries

Causality refers to a sequence of operations, that can span multiple execution contexts, fork and join, creating a tree of execution logically rooted in one source, one cause. This logical grouping is called the causality context and implicitly flows from the sender of a message to the receiver. Any further messages sent by the code activated on the receiver, carry with them this causality and propagate it to any new receivers.

Causalities are an extension of the structured exception mechanism implemented for threads. They allow for nesting but also deal with multiple failures happening concurrently and with merges with other causalities, due to joins for example.

Example 22

void SimpleCausalityExample()
{
    Port<Exception> exceptionPort = new Port<Exception>();

    // create a causality using the port instance
    Causality exampleCausality = new Causality("root cause", exceptionPort);

    // add causality to current thread
    Dispatcher.AddCausality(exampleCausality);

    // any unhandled exception from this point on, in this method or
    // any delegate that executes due to messages from this method,
    // will be posted on exceptionPort.

    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandler)
    );

    // causalities flow when items are posted or Tasks are scheduled
    portInt.Post(0);

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine(ex);
        })
    );
}

void IntHandler(int i)
{
    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine(c.Name);
    }

    // expect DivideByZeroException that CCR will redirect to the causality
    int k = 10 / i;
}

Example 22 demonstrates a simple scenario where causalities help with error handling across asynchronous operations. The code performs the following key steps:

1. An instance of Port<Exception> is created to store any exceptions thrown within the causality context.
2. A new Causality instance is created using the exception port plus a friendly name.
3. The causality is added to the list of causalities the Dispatcher keeps for the current execution context, the OS thread.
4. An item is posted on a port. Because the post happens within the context of a causality, it will implicitly attach its causality to the item being posted. When a handler executes with this item, the causalities attached with the item will be added to the thread executing the receiver. Any exceptions in the receiver, will be posted on the exception port associated with the causality
5. A receiver is activated on the exception port, so any exception thrown and not handled by any asynchronous operations within the causality can be received.
6. The handler scheduled due to the item being posted on the portInt instance, throws a DivideByZeroException. The CCR scheduler redirects the exception to the exception port associated with the causality.

The example demonstrates that causalities are the most robust way to deal with failure. If you add additional asynchronous operations, that cause other handlers to execute in parallel, the failure handling is not modified. Further, any operations these handlers do, will also be covered within the scope of the original causality flowing exceptions all the way back to the exception port

It is often appropriate to nest causalities, so different failure handlers compensate for failure at different levels. Similar to structured exceptions, but generalized to a concurrent setting. Causalities can nest, giving you the choice of either dealing with an exception in the inner causality, or posting it explicitly to any parent causality.

Example 23

public void NestedCausalityExample()
{
    Port<Exception> exceptionPort = new Port<Exception>();
    Causality parentCausality = new Causality(
        "Parent",
        exceptionPort);
    Dispatcher.AddCausality(parentCausality);

    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandlerNestingLevelZero)
    );
    portInt.Post(0);

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false,
        exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Parent:" + ex);
        })
    );
}

void IntHandlerNestingLevelZero(int i)
{
    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine("Before child is added:   " + c.Name);
    }

    // create new child causality that will nest under existing causality
    Port<Exception> exceptionPort = new Port<Exception>();
    Causality childCausality = new Causality(
        "Child",
        exceptionPort);
    Dispatcher.AddCausality(childCausality);

    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false,
        exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Child:" + ex);
        })
    );

    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine("After child is added:    " + c.Name);
    }

    // attach a receiver and post to a port
    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandlerNestingLevelOne)
    );
    portInt.Post(0);
}

void IntHandlerNestingLevelOne(int i)
{
    throw new InvalidOperationException("Testing causality support. Child causality will catch this one");
}

Before child is added:   Parent
After child is added:    Child
Child:System.InvalidOperationException: Testing causality support. Child causality will catch this one
   at Examples.Examples.IntHandlerNestingLevelOne(Int32 i) in C:\mri\main\CCR\testsrc\UnitTests\ccruserguideexamples.cs:line 571
   at Microsoft.Ccr.Core.Task`1.Execute() in C:\mri\main\CCR\src\Core\Templates\GeneratedFiles\Task\Task01.cs:line 301
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecuteTaskHelper(ITask currentTask) in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1476
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecuteTask(ITask& currentTask, DispatcherQueue p) in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1376
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecutionLoop() in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1307

The example above shows a nested asynchronous sequence:

1. Method NestedCausalityExample adds a causality and issues post to a port with a receiver.
2. Method IntHandlerNestingLevelZero executes asynchronously and in parallel with method NestedCausalityExample, adding a new causality that nests under the causality that flowed from step 1.
3. Method IntHandlerNestingLevelZero again creates a new port with a different receiver and posts a message.
4. Method IntHandlerNestingLevelOne executes, indirectly caused from the original post in method NestedCausalityExample, and with two active causalities. The bottom of the stack, is the child causality created by method IntHandlerNestingLevelZero.
5. Exception is thrown in method IntHandlerNestingLevelOne, which is handled by the child causality handler, since it "hides" exceptions from its parent.

Using the static property Dispatcher.ActiveCausalities, a causality exception handler can explicitly post exceptions to its parent causalities, propagating failure selectively to higher levels. The example uses this collection to print out all the active causalities at the top of method IntHandlerNestingLevelZero.

A unique feature of the CCR causality implementation is that it can combine causalities that come from two different executions paths, but "meet" on a handler that executed due to a join or multiple item receive being satisfied. In this scenario the CCR does not nest causalities, but adds them as peers in the thread handler with the joined items. If an exception is thrown, it propagates concurrently along two independent causality stacks, one for each item passed in the handler.

Example 24

public void JoinedCausalityExample()
{
    Port<int> intPort = new Port<int>();
    Port<int> leftPort = new Port<int>();
    Port<string> rightPort = new Port<string>();

    Port<Exception> leftExceptionPort = new Port<Exception>();
    Port<Exception> rightExceptionPort = new Port<Exception>();

    // post twice so two handlers run
    intPort.Post(0);
    intPort.Post(1);

    // activate two handlers that will execute concurrently and create
    // two different parallel causalities
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, intPort,
        delegate(int i)
        {

            Causality leftCausality = new Causality("left", leftExceptionPort);
            Dispatcher.AddCausality(leftCausality);
            // post item on leftPort under the context of the left causality
            leftPort.Post(i);
        })
    );

    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, intPort,
        delegate(int i)
        {
            Causality rightCausality = new Causality("right", rightExceptionPort);
            Dispatcher.AddCausality(rightCausality);
            // post item on rightPort under the context of the right causality
            rightPort.Post(i.ToString());
        })
    );

    // activate one join receiver that executes when items are available on
    // both leftPort and rightPort

    Arbiter.Activate(_taskQueue,
        Arbiter.JoinedReceive<int, string>(false, leftPort, rightPort,
        delegate(int i, string s)
        {
            throw new InvalidOperationException("This exception will propagate to two peer causalities");
        })
    );

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, leftExceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Left causality:  " + ex);
        })
    );

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, rightExceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Right causality:  " + ex);
        })
    );
}

Left causality:  System.InvalidOperationException: This exception will propagate to two peer causalities
Right causality:  System.InvalidOperationException: This exception will propagate to two peer causalities

The example above demonstrates how anonymous methods are used heavily to keep all the logic within one method, for readability. The example demonstrates:

• Two independent handlers create two causalities, named left and right
• Post one message each to two different ports
• A third handler executes due to a join being satisfied across these two ports, leftPort and rightPort
• The join handler throws an exception in the context of both causalities

The key point is that when the join handler executes and throws an exception, two peer causalities are active and thus both independently get the exception posted on their respective exception ports.

Since causalities use regular CCR ports to receive exceptions, you can use CCR coordination primitives to combine exception handling across multiple causalities. Joins, interleave and choice are all appropriate and powerful ways to combine failure handling across concurrent, multi-tiered asynchronous operations

© 2012 Microsoft Corporation. All Rights Reserved.