Traditional failure handling schemes follow the following patterns:

1. For synchronous invocations of methods, the caller checks one or more return values from the method. The method uses the callers execution context (most often this is a thread) to run
2. Using structured exception handling, the caller wraps synchronous method invocations in try/catch/finally statements and either relies exclusively on the catch{} block for error handling or uses a combination of the catch{} block plus explicit checks on the results from the method
3. Transactions relying on a variety of compensation implementations by the components being called, OS infrastructure and usually expensive mechanisms to flow context across threads

All methods above can not easily apply to concurrent, asynchronous execution, especially the first two which are simply not available for asynchronous programming. Methods execute in arbitrary context, potentially in parallel with the caller. Unless there is a blocking operation to collect the result of the asynchronous operation, thus blocking a thread in the callers context, failure or success can not easily be determined. The asynchronous programming model in CLR, using the Begin/End calls introduces additional complexity since failure handling happens through additional calls and exception handling in the single callback that executes regardless of the result of the operation. The code the started the asynchronous operations is not the place that failure is handled, making code hard to read.

The CCR addresses failure handling with two approaches:

1. Explicit or local failure handling using the Choice and MultipleItemGather arbiters. Combined with iterator support they provide a type safe, robust way to deal with failure since they force the programmer to deal with the success and failure case in two different methods and then also have a common continuation. Examples 8,13 and 20 show explicit error handling using Choice, MultipleItemGather and Choice in an iterator, respectively.
2. Implicit or distributed failure handling, referred to as Causalities that allows for nesting and extends across multiple asynchronous operations.It shares in common with transactions the notion of a logical context or grouping of operations and extends it for a concurrent asynchronous environment. This mechanism can also be extended across machine boundaries

Causalities

Causality refers to a sequence of operations, that can span multiple execution contexts, fork and join, creating a tree of execution logically rooted in one source, one cause. This logical grouping is called the causality context and implicitly flows from the sender of a message to the receiver. Any further messages sent by the code activated on the receiver, carry with them this causality and propagate it to any new receivers.

Causalities are an extension of the structured exception mechanism implemented for threads. They allow for nesting but also deal with multiple failures happening concurrently and with merges with other causalities (due to joins for example).

Example 22

void SimpleCausalityExample()
{
    Port<Exception> exceptionPort = new Port<Exception>();

    // create a causality using the port instance
    Causality exampleCausality = new Causality("root cause", exceptionPort);

    // add causality to current thread
    Dispatcher.AddCausality(exampleCausality);

    // any unhandled exception from this point on, in this method or
    // any delegate that executes due to messages from this method,
    // will be posted on exceptionPort.

    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandler)
    );

    // causalities flow when items are posted or Tasks are scheduled
    portInt.Post(0);

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine(ex);
        })
    );
}

void IntHandler(int i)
{
    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine(c.Name);
    }

    // expect DivideByZeroException that CCR will redirect to the causality
    int k = 10 / i;
}
        

In the example above we demonstrate a simple scenario where causalities help with error handling across asynchronous operations. The code performs the following key steps:

1. An instance of Port<Exception> is created to store any exceptions thrown within the causality context.
2. A new Causality instance is created using the exception port plus a friendly name.
3. The causality is added to the list of causalities the Dispatcher keeps for the current execution context (OS thread).
4. An item is posted on a port. Because the post happens within the context of a causality, it will implicitly "attach" its causality to the item being posted. When a handler executes with this item, the causalities attached with the item will be added to the thread executing the receiver. Any exceptions in the receiver, will be posted on the exception port associated with the causality
5. A receiver is activated on the exception port so any exception thrown and not handled by any asynchronous operations within the causality, can be received.
6. The handler scheduled due to the item being posted on the portInt instance, throw a DivideByZeroException. The CCR scheduler redirects the exception to the exception port associated with the causality.

The example demonstrates that causalities are the most robust way to deal with failure. If the programmer adds addition asynchronous operations, that cause other handlers to execute in parallel, the failure handling is not modified. Furthermore, any operations these handlers do, will also be covered within the scope of the original causality, flowing exceptions all the way back to the exception port

Nested Causalities

It is often appropriate to nest causalities, so different failure handlers compensate for failure at different levels. Similar to structured exceptions, but generalized to a concurrent setting, causalities can nest, giving the programmer the choice of either dealing with an exception in the inner causality, or posting it explicitly to any parent causality.

Example 23

public void NestedCausalityExample()
{
    Port<Exception> exceptionPort = new Port<Exception>();
    Causality parentCausality = new Causality(
        "Parent",
        exceptionPort);
    Dispatcher.AddCausality(parentCausality);

    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandlerNestingLevelZero)
    );
    portInt.Post(0);

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false,
        exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Parent:" + ex);
        })
    );
}

void IntHandlerNestingLevelZero(int i)
{
    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine("Before child is added:   " + c.Name);
    }

    // create new child causality that will nest under existing causality
    Port<Exception> exceptionPort = new Port<Exception>();
    Causality childCausality = new Causality(
        "Child",
        exceptionPort);
    Dispatcher.AddCausality(childCausality);

    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false,
        exceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Child:" + ex);
        })
    );

    // print active causalities
    foreach (Causality c in Dispatcher.ActiveCausalities)
    {
        Console.WriteLine("After child is added:    " + c.Name);
    }

    // attach a receiver and post to a port
    Port<int> portInt = new Port<int>();
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, portInt, IntHandlerNestingLevelOne)
    );
    portInt.Post(0);
}

void IntHandlerNestingLevelOne(int i)
{
    throw new InvalidOperationException("Testing causality support. Child causality will catch this one");
}
        

Console output from example:

Before child is added:   Parent
After child is added:    Child
Child:System.InvalidOperationException: Testing causality support. Child causality will catch this one
   at Examples.Examples.IntHandlerNestingLevelOne(Int32 i) in C:\mri\main\CCR\testsrc\UnitTests\ccruserguideexamples.cs:line 571
   at Microsoft.Ccr.Core.Task`1.Execute() in C:\mri\main\CCR\src\Core\Templates\GeneratedFiles\Task\Task01.cs:line 301
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecuteTaskHelper(ITask currentTask) in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1476
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecuteTask(ITask& currentTask, DispatcherQueue p) in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1376
   at Microsoft.Ccr.Core.TaskExecutionWorker.ExecutionLoop() in C:\mri\main\CCR\src\Core\scheduler_roundrobin.cs:line 1307

In the example above we show a nested asynchronous sequence:

1. Method NestedCausalityExample adds a causality and issues post to a port with a receiver
2. Method IntHandlerNestingLevelZero executes asynchronously and in parallel with method NestedCausalityExample, adds a new causality that nests under the causality that flowed from step 1
3. Method IntHandlerNestingLevelZero again creates a new port with a different receiver and posts a message
4. Method IntHandlerNestingLevelOne executes, indirectly caused from the original post in method NestedCausalityExample, and with two active causalities. The bottom of the stack, is the child causality created by method IntHandlerNestingLevelZero
5. Exception is thrown in method IntHandlerNestingLevelOne, which is handled by the child causality handler, since it "hides" exceptions from its parent

Using the static property Dispatcher.ActiveCausalities, a causality exception handler can explicitly post exceptions to its parent causalities, propagating failure selectively to higher levels. The example uses this collection to print out all the active causalities at the top of method IntHandlerNestingLevelZero.

Joins and Causalities

A unique feature of the CCR causality implementation is that it can combine causalities that come from two different executions path, but "meet" on a handler that executed due to a join or multiple item receive being satisfied. In that scenario the CCR does not nest causalities, but adds them as peers in the thread the handler with the joined items. If an exception is thrown, it propagates concurrently along two independent causality stacks, one for each item passed in the handler.

Example 24

public void JoinedCausalityExample()
{
    Port<int> intPort = new Port<int>();
    Port<int> leftPort = new Port<int>();
    Port<string> rightPort = new Port<string>();

    Port<Exception> leftExceptionPort = new Port<Exception>();
    Port<Exception> rightExceptionPort = new Port<Exception>();

    // post twice so two handlers run
    intPort.Post(0);
    intPort.Post(1);

    // activate two handlers that will execute concurrently and create
    // two different parallel causalities
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, intPort,
        delegate(int i)
        {

            Causality leftCausality = new Causality("left", leftExceptionPort);
            Dispatcher.AddCausality(leftCausality);
            // post item on leftPort under the context of the left causality
            leftPort.Post(i);
        })
    );

    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, intPort,
        delegate(int i)
        {
            Causality rightCausality = new Causality("right", rightExceptionPort);
            Dispatcher.AddCausality(rightCausality);
            // post item on rightPort under the context of the right causality
            rightPort.Post(i.ToString());
        })
    );

    // activate one join receiver that executes when items are available on
    // both leftPort and rightPort

    Arbiter.Activate(_taskQueue,
        Arbiter.JoinedReceive<int, string>(false, leftPort, rightPort,
        delegate(int i, string s)
        {
            throw new InvalidOperationException("This exception will propagate to two peer causalities");
        })
    );

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, leftExceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Left causality:  " + ex);
        })
    );

    // activate a handler on the exceptionPort
    // This is the failure handler for the causality
    Arbiter.Activate(_taskQueue,
        Arbiter.Receive(false, rightExceptionPort,
        delegate(Exception ex)
        {
            // deal with failure here
            Console.WriteLine("Right causality:  " + ex);
        })
    );
}
        

Console output from example.

 Left causality:  System.InvalidOperationException: This exception will propagate to two peer causalities
Right causality:  System.InvalidOperationException: This exception will propagate to two peer causalities

In the example above anonymous methods are used heavily to keep all the logic within one method, for readability. The example demonstrates:

•  two independent handlers create two causalities (named "left" and "right")
• post one message each to two different ports
• a third handler executes due to a join being satisfied across these two ports (leftPort and rightPort)
• The join handler throws an exception in the context of both causalities

The key point is that when the join handler executes and throws an exception, two peer causalities are active and thus both independently get the exception posted on their respective exception ports.

Since causalities use regular CCR ports to receive exceptions, you can use CCR coordination primitives to combine exception handling across multiple causalities. Joins, interleave and choice are all appropriate and powerful ways to combine failure handling across concurrent, multi-tiered asynchronous operations