How to: Fix 'Application is Busy' and 'Call was Rejected By Callee' Errors

Visual Studio add-ins are deprecated in Visual Studio 2013. You should upgrade your add-ins to VSPackage extensions. For more information about upgrading, see FAQ: Converting Add-ins to VSPackage Extensions.

If you programmatically call into Visual Studio automation from an external (out-of-proc), multi-threaded application, you may receive the following errors:

  • Application is busy (RPC_E_CALL_REJECTED 0x80010001)

  • Call was rejected by callee (RPC_E_SERVERCALL_RETRYLATER 0x8001010A)

These errors occur due to threading contention issues between external multi-threaded applications and Visual Studio. They can be eliminated by implementing IOleMessageFilter error handlers in your Visual Studio automation application. (Do not confuse IOleMessageFilter with System.Windows.Forms.IMessageFilter.)

To fix errors

  1. Add the following class to your application.

  2. Add a COM reference to "Microsoft Development Environment 8.0." This adds references to EnvDTE and EnvDTE80 to your solution.

    Add a reference to System.Windows.Forms.

  3. In the code, create an instance of EnvDTE80, as outlined in the following example.

  4. Call Message.Register to handle thread errors.

  5. Call your automation code as usual.

  6. When your automation code is finished, call Message.Revoke to remove the thread error handlers.

Example

using System;
using System.Collections.Generic;
using System.Text;
using EnvDTE;
using EnvDTE80;
using System.Runtime.InteropServices;
using System.Windows.Forms;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            EnvDTE80.DTE2 dte;
            object obj = null;
            System.Type t = null;

            // Get the ProgID for DTE 8.0.
            t = System.Type.GetTypeFromProgID("VisualStudio.DTE.10.0",
              true);
            // Create a new instance of the IDE.
            obj = System.Activator.CreateInstance(t, true);
            // Cast the instance to DTE2 and assign to variable dte.
            dte = (EnvDTE80.DTE2)obj;

            // Register the IOleMessageFilter to handle any threading 
            // errors.
            MessageFilter.Register();
            // Display the Visual Studio IDE.
            dte.MainWindow.Activate();

            // =====================================
            // ==Insert your automation code here.==
            // =====================================
            // For example, get a reference to the solution2 object
            // and do what you like with it.
            Solution2 soln = (Solution2)dte.Solution;
            System.Windows.Forms.MessageBox.Show
              ("Solution count: " + soln.Count);
            // =====================================
            
            // All done, so shut down the IDE...
            dte.Quit();
            // and turn off the IOleMessageFilter.
            MessageFilter.Revoke();
            
        }
    }

    public class MessageFilter : IOleMessageFilter
    {
        //
        // Class containing the IOleMessageFilter
        // thread error-handling functions.

        // Start the filter.
        public static void Register()
        {
            IOleMessageFilter newFilter = new MessageFilter(); 
            IOleMessageFilter oldFilter = null; 
            CoRegisterMessageFilter(newFilter, out oldFilter);
        }

        // Done with the filter, close it.
        public static void Revoke()
        {
            IOleMessageFilter oldFilter = null; 
            CoRegisterMessageFilter(null, out oldFilter);
        }

        //
        // IOleMessageFilter functions.
        // Handle incoming thread requests.
        int IOleMessageFilter.HandleInComingCall(int dwCallType, 
          System.IntPtr hTaskCaller, int dwTickCount, System.IntPtr 
          lpInterfaceInfo) 
        {
            //Return the flag SERVERCALL_ISHANDLED.
            return 0;
        }

        // Thread call was rejected, so try again.
        int IOleMessageFilter.RetryRejectedCall(System.IntPtr 
          hTaskCallee, int dwTickCount, int dwRejectType)
        {
            if (dwRejectType == 2)
            // flag = SERVERCALL_RETRYLATER.
            {
                // Retry the thread call immediately if return >=0 & 
                // <100.
                return 99;
            }
            // Too busy; cancel call.
            return -1;
        }

        int IOleMessageFilter.MessagePending(System.IntPtr hTaskCallee, 
          int dwTickCount, int dwPendingType)
        {
            //Return the flag PENDINGMSG_WAITDEFPROCESS.
            return 2; 
        }

        // Implement the IOleMessageFilter interface.
        [DllImport("Ole32.dll")]
        private static extern int 
          CoRegisterMessageFilter(IOleMessageFilter newFilter, out 
          IOleMessageFilter oldFilter);
    }

    [ComImport(), Guid("00000016-0000-0000-C000-000000000046"), 
    InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
    interface IOleMessageFilter 
    {
        [PreserveSig]
        int HandleInComingCall( 
            int dwCallType, 
            IntPtr hTaskCaller, 
            int dwTickCount, 
            IntPtr lpInterfaceInfo);

        [PreserveSig]
        int RetryRejectedCall( 
            IntPtr hTaskCallee, 
            int dwTickCount,
            int dwRejectType);

        [PreserveSig]
        int MessagePending( 
            IntPtr hTaskCallee, 
            int dwTickCount,
            int dwPendingType);
    }
}

Robust Programming

When your external, multi-threaded application calls into Visual Studio, it goes through a COM interface. COM sometimes has problems dealing properly with threads, especially with respect to timing. As a result, occasionally the incoming thread from the external application cannot be handled by Visual Studio at the very moment it arrives, resulting in the previously mentioned errors. This does not occur, however, if you are calling from an application that is running inside Visual Studio (in-proc), such as a macro or an add-in. For a more detailed explanation about the reasons behind this, see Threading Support in Office.

To avoid these errors, implement an IOleMessageFilter handler function in your application. When you do this, if your external application thread calls into Visual Studio and is rejected (that is, it returns SERVERCALL_RETRYLATER from the IOleMessageFilter.HandleIncomingCall method), then your application can handle it and either retry or cancel the call. To do this, initiate the new thread from your Visual Studio application in a single-threaded apartment (STA) and surround your automation code with the IOleMessageFilter handler.

See Also

Tasks

Walkthrough: Debugging an Add-In Project