DirectShow Movies in Direct3D Worlds

  • Article

Mike Borozdin

February 2003

Applies to:
    Microsoft DirectShow
    Microsoft Direct3D
    Microsoft DirectX 9.0

Summary: Learn how to display video in a Direct 3D environment from a DirectShow or Direct3D application developer's perspective. (13 printed pages)

Contents

Introduction
Architecture of the VMR-9
Implementing the Allocator-Presenter
Configuring the VMR and the Filter Graph
For More Information

Introduction

One of the biggest new features in Microsoft® DirectX® 9.0 is the long-awaited merging of the video and graphics rendering pipelines. Until recently, Microsoft® DirectShow® rendered video primarily through Microsoft DirectDraw® on the hardware overlay surface, or else through GDI in system memory.

The Video Mixing Renderer 7 (VMR-7) was introduced in Microsoft Windows® XP. It enabled rendering to off-screen DirectDraw 7 surfaces, but was not compatible with the Microsoft Direct3D® 8.0 interfaces. Most recently, DirectX 9.0 introduced the VMR-9, which is available for all platforms supported by DirectX, and which uses Direct3D surfaces to render video frames.

The VMR-9 has several advantages. First, you can use all of the processing transforms that Direct3D offers. For example, you can easily run a video stream though a pixel shader for custom video effects. In fact, the VMR-9 is a powerful real-time digital signal processor (DSP) at your fingertips. Video can also be rendered onto a texture—in the simplest scenario, imagine a video playing on one surface of a rotating cube. If you're a game developer, video elements are no longer limited to clunky cutscenes. Now, you can combine video with 3D graphics, to create video clips as dynamic and interactive as your graphics.

Second, the VMR-9 makes it easier to integrate video with the user interface (UI). Now the UI can be dynamic—you are no longer limited to using color keying to overlay static UI elements onto video.

In addition to its integration with Direct3D, the VMR offers other important new features:

  • The ability to render multiple video streams simultaneously.
  • Support for the latest deinterlacing hardware. When interlaced sources, such as television or DV video, are displayed on a progressive-scan monitor, the interlaced fields must be deinterlaced to form a progressive image. Sophisticated deinterlacing techniques can be provided in real time by the graphics processing unit (GPU). The VMR enables the application to choose from the available techniques.
  • Support for new hardware that controls hue, saturation, brightness, and contrast, called process amplification or ProcAmp. ProcAmp is no longer a global setting that users have to tweak on their monitors. In 3D worlds, a global contrast and brightness feature is usually not desirable. Combined with the mixing, alpha blending, and color-keying capabilities of the VMR-9, support for ProcAmp opens a wide range of possibilities for fading movies in and out.

This article covers the following topics:

  • Architecture of the VMR-9 describes the plug-in components of the VMR-9 filter.
  • Implementing the Allocator-Presenter describes how to create a custom allocator-presenter object for the VMR-9.
  • Creating the Direct3D Device describes the first step in the 3-D rendering process: creating the Direct3D device object.
  • Allocating Surfaces examines the next step in the 3-D rendering process: allocating one or more Direct3D surfaces to hold the video image.
  • Presenting the Image examines the last step in the 3-D rendering process: presenting the image.
  • Configuring the VMR-9 and the Filter Graph describes how the application configures the VMR-9 filter and builds the filter graph to render video.

Architecture of the VMR-9

If you are familiar with the VMR-7, you will find the VMR-9 to be quite familiar, because the VMR-9 architecture is virtually identical to that of the VMR-7. This architecture is described in detail in the SDK documentation, so it will only be briefly mentioned here.

The VMR-9 is a renderer filter; it sits in the filter graph at the end of the video stream. It contains two plug-in components that can be replaced by custom components in your application:

  • Allocator-Presenter. Allocates the Direct3D surfaces used by the VMR-9, and presents those surfaces when the VMR-9 fills them with video frames.
  • Image Compositor. Controls how the video input streams are mixed. (This article does not discuss the image compositor.)

This article shows how easy it is to replace the default allocator-presenter with a custom allocator-presenter that draws the video stream onto a D3D texture surface, which is then rendered onto a rotating square plane.

Implementing the Allocator-Presenter

To use a custom allocator-presenter, the application:

  • Creates an instance of the custom allocator-presenter object.
  • Configures the VMR-9.
  • Builds the filter graph.

Before describing these steps, however, let's examine the implementation of the allocator-presenter object itself.

In the VMR9Allocator SDK sample, the custom allocator-presenter is implemented as a C++ class named CAllocator. This class performs three basic tasks:

  • Creating the Direct3D device
  • Allocating surfaces
  • Presenting the image
Creating the Direct3D Device

The first task for the custom allocator-presenter is setting up the Direct3D environment. If you are familiar with Direct3D programming, this process is relatively straightforward. First, create the Direct3D object, and then use this object to create the Direct3D device. The CAllocator class performs these steps inside its class constructor, which calls the CreateDevice member function, as shown in the following code:

CAllocator::CAllocator(HRESULT& hr, HWND wnd, IDirect3D9* d3d,
    IDirect3DDevice9* d3dd)
: m_refCount(1), m_D3D(d3d), m_D3DDev(d3dd), m_window( wnd )
{
    CAutoLock Lock(&m_ObjectLock);
    hr = E_FAIL;

    if ( IsWindow( wnd ) == FALSE )
    {
        hr = E_INVALIDARG;
        return;
    }

    if ( m_D3D == NULL )
    {
        ASSERT( d3dd ==  NULL ); 
        m_D3D.Attach( Direct3DCreate9(D3D_SDK_VERSION) );
        if (m_D3D == NULL) {
            hr = E_FAIL;
            return;
        }
    }

    if ( m_D3DDev == NULL )
    {
        hr = CreateDevice();
    }
}

HRESULT CAllocator::CreateDevice()
{
    m_D3DDev = NULL;

    D3DPRESENT_PARAMETERS pp;
    ZeroMemory(&pp, sizeof(pp));
    pp.Windowed = TRUE;
    pp.hDeviceWindow = m_window;
    pp.SwapEffect = D3DSWAPEFFECT_COPY;
    return m_D3D->CreateDevice(0, D3DDEVTYPE_HAL, m_window,
        D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_MULTITHREADED,
        &pp, &m_D3DDev.p);
}

In this example, the flags given to IDirect3D9::CreateDevice specify windowed mode and D3DSWAPEFFECT_COPY. You can use different flags if appropriate. The D3DCREATE_MULTITHREADED flag is required, however, because the VMR-9 runs on a separate thread.

Allocating Surfaces

The next major task for the allocator-presenter is allocating surfaces for the VMR-9. Surface allocation can be tricky, because decoders output a wide range of video formats, and graphics cards provide varying support for these formats. When you allocate surfaces, the goal is to create a video image that can be drawn onto a 3-D primitive. To do this, you must consider the upstream filter, the hardware, and the format conversion capabilities of Direct3D.

Surfaces are created in the allocator-presenter object's InitializeDevice method, which is called when the VMR-9 negotiates the pin connection with the upstream decoder. This method has the following signature:

  STDMETHODIMP InitializeDevice( 
    DWORD_PTR dwUserID,
    VMR9AllocationInfo *lpAllocInfo,
    DWORD *lpNumBuffers
);

The dwUserID parameter identifies the VMR-9 instance. For this example, there is only one instance, so dwUserID can be ignored. The VMR9AllocationInfo structure contains several pieces of information; the VMR fills them in based on the format proposed by the upstream filter, which is typically a video decoder. The dwWidth and dwHeight members contain the video dimensions, the Format member is a Direct3D format type, and the dwFlags member describes the surface.

For example, the VMR9AllocFlag_TextureSurface flag indicates a texture surface, and the VMR9AllocFlag_3DRenderTarget flag indicates a render target. The lpNumBuffers parameter specifies the number of surfaces to create.

The VMR-9 needs a surface that the upstream decoder will accept. To create Direct3D surfaces that are compatible with video decoders and with the Direct3D device, the following steps are suggested:

  1. Notify the VMR-9 about the Direct3D device and the monitor, by calling IVMRSurfaceAllocatorNotify::SetD3DDevice.
  2. Add the VMR9AllocFlag_TextureSurface flag to the VMR9AllocationInfo structure, to request a texture surface.
  3. Allocate surfaces. If this operation succeeds, return S_OK.
  4. If the previous step failed, and the upstream filters did not request the surfaces to be render targets, try to create off-screen surfaces.

The following code shows how the CAllocator class implements these steps:

HRESULT CAllocator::InitializeDevice(DWORD_PTR dwUserID,
    VMR9AllocationInfo *lpAllocInfo, DWORD *lpNumBuffers)
{
    if (lpNumBuffers == NULL)
    {
        return E_POINTER;
    }
    if ( m_lpIVMRSurfAllocNotify == NULL )
    {
        return E_FAIL;
    }
    HRESULT hr = m_lpIVMRSurfAllocNotify->SetD3DDevice( m_D3DDev,
        MonitorFromWindow( m_window, MONITOR_DEFAULTTOPRIMARY ) );
    if (FAILED(hr))
    {
        return hr;
    }
    // Make sure to create textures. Otherwise, the surface cannot be
    // textured onto our primitive. Therefore, add the "texture" flag.
    lpAllocInfo->dwFlags |= VMR9AllocFlag_TextureSurface;

    DeleteSurfaces(); // Release any surfaces that we allocated 
      previously.

    // Resize the array of surface pointers.
    m_surfaces.resize(*lpNumBuffers); 

    // Ask the VMR-9 to allocate the surfaces for us.
    hr = m_lpIVMRSurfAllocNotify->AllocateSurfaceHelper(
        lpAllocInfo, lpNumBuffers, & m_surfaces.at(0) );
    
    // If this call failed, create a private texture. We will copy the
    // decoded video frames into the private texture. 
    if (FAILED(hr) && 
        !(lpAllocInfo->dwFlags & VMR9AllocFlag_3DRenderTarget))
    {
        DeleteSurfaces();
        // Is the format YUV?
        if (lpAllocInfo->Format > '0000')
        {           
            D3DDISPLAYMODE dm;
            hr = m_D3DDev->GetDisplayMode(NULL,  & dm );
            if (SUCCEEDED(hr))
            {
               // Create the private texture.
               hr = m_D3DDev->CreateTexture(
                   lpAllocInfo->dwWidth, 
                   lpAllocInfo->dwHeight,
                   1, 
                   D3DUSAGE_RENDERTARGET, 
                   dm.Format, 
                   D3DPOOL_DEFAULT, 
                   & m_privateTexture.p, NULL );
            }
            if (FAILED(hr))
            {
               return hr;
            }
        }
        lpAllocInfo->dwFlags &= ~ VMR9AllocFlag_TextureSurface;
        lpAllocInfo->dwFlags |= VMR9AllocFlag_OffscreenSurface;
        hr = m_lpIVMRSurfAllocNotify->AllocateSurfaceHelper(
            lpAllocInfo, lpNumBuffers, & m_surfaces.at(0) );
        if (FAILED(hr))
        {
            return hr;
        }
    }
    return m_scene.Init(m_D3DDev);
}

Translating the information in the VMR9AllocationInfo structure into a series of Direct3D calls to allocate the correct surfaces is not trivial. Fortunately, the VMR-9 provides a helper function, IVMRSurfaceAllocatorNotify9::AllocateSurfaceHelper, which does much of the work for you. Due to the variety of available hardware, it is a good idea to use one or more fallback algorithms, to ensure that at least some surfaces get created. If the format of the surface is a YUV format, the allocator-presenter must create an RGB surface that will get textured onto the Direct3D primitives.

Presenting the Image

For each input stream, the video decoder writes the video frames into the VMR-9 input buffers. The VMR-9 applies the ProcAmp values to each stream, composites the streams, applies color keying and alpha blending if necessary, and finally passes the composited image to the allocator-presenter. The VMR-9 then calls the allocator-presenter object's GetSurface method to get the surfaces for the decoders, and calls the PresentImage method when the image should be presented.

The correct way to present the image depends largely on the characteristics of the surfaces that were allocated. For this reason, it is best to implement the allocator-presenter as a single object that performs both allocation and presentation. In addition to the obvious tasks of putting together a 3-D scene, your application must handle situations when the device is lost or the window has been moved to another monitor.

To render the image onto a 3-D scene, the following steps are suggested:

  1. Check whether a display change is in progress. If the device has changed, recreate the surfaces on the new device.
  2. If the device does not support textures in the native video format and you created a private texture, do the following:
    • Get the surface by calling IDirect3DTexture9::GetSurfaceLevel.
    • Stretch and color convert the surface from the VMR-9 into your private texture by calling IDirect3DDevice9::StretchRect.
  3. If you did not create a private texture, get the texture from the surface.
  4. Draw the scene, texturing the video onto some set of vertices in the scene.
  5. Present the image.
  6. If the device was lost, quit rendering the scene, restore the device, and recreate the surface.

The following code shows how the CAllocator class implements the PresentImage method:

HRESULT CAllocator::PresentImage(
    DWORD_PTR dwUserID, 
    VMR9PresentationInfo *lpPresInfo)
{
    CAutoLock Lock(&m_ObjectLock);
    // Check to see if we are in the middle of a display change.
    if ( NeedToHandleDisplayChange() )
    {
        //  Switch the Direct3D device. (Code not shown.)
    }
    HRESULT hr = PresentHelper( lpPresInfo ); // Code shown below.
    if ( hr == D3DERR_DEVICELOST)
    {
        // Try to restore the device.
        if (m_D3DDev->TestCooperativeLevel() == D3DERR_DEVICENOTRESET)
        {
            DeleteSurfaces();
            hr = CreateDevice();
            if (FAILED(hr))
            {
               return hr;
            }
            hr = m_lpIVMRSurfAllocNotify->ChangeD3DDevice( m_D3DDev,
                MonitorFromWindow( m_window,MONITOR_DEFAULTTOPRIMARY ) );
            if (FAILED(hr))
            {
               return hr;
            }
        }
        hr = S_OK;
    }
    return hr;
}
HRESULT CAllocator::PresentHelper(VMR9PresentationInfo *lpPresInfo)
{
    // Parameter validation.
    if ( ( lpPresInfo == NULL ) || ( lpPresInfo->lpSurf == NULL ) )
    {
        return E_POINTER;
    }
    CAutoLock Lock(&m_ObjectLock);
    // Get the device from the surface. 
    CComPtr<IDirect3DDevice9> device;
    HRESULT hr = lpPresInfo->lpSurf->GetDevice(& device.p );
    if (FAILED(hr))
    {
        return hr;
    }
    // If we created a private texture, blit the decoded image onto it.
    if ( m_privateTexture != NULL )
    {   
        CComPtr<IDirect3DSurface9> surface;
        hr = m_privateTexture->GetSurfaceLevel( 0 , & surface.p );
        if (FAILED(hr))
        {
            return hr;
        }
        // Copy the full surface onto the texture's surface.
        hr = device->StretchRect( lpPresInfo->lpSurf, NULL, surface,
            NULL, D3DTEXF_NONE );
        if (FAILED(hr))
        {
            return hr;
        }
        hr = m_scene.DrawScene( device, m_privateTexture ) ;
        if (FAILED(hr))
        {
            return hr;
        }
    }
    else // The texture was allocated by the VMR-9.
    {
        // Get the texture from the surface.
        CComPtr<IDirect3DTexture9> texture;
        hr = lpPresInfo->lpSurf->GetContainer( IID_IDirect3DTexture9,
           (LPVOID*) & texture.p ) );
        if (FAILED(hr))
        {
            return hr;
        }
        hr = m_scene.DrawScene (device, texture );
        if (FAILED(hr))
        {
            return hr;
        }
    }
    hr = device->Present( NULL, NULL, NULL, NULL );
    return hr;
}

Configuring the VMR and the Filter Graph

The VMR-9 supports three distinct modes of operation:

  • Windowed

    In this mode, which is the default, the VMR-9 creates a separate window and renders the video onto that window.

  • Windowless

    This mode gives the application greater control, by rendering the video directly onto an application window, in effect making the video into a control on the window.

  • Renderless

    This mode uses a custom allocator-presenter object.

To use the CAllocator class with the VMR-9, therefore, you must configure the VMR-9 for renderless mode by performing the following steps:

  1. Create the VMR-9 filter.
  2. Query the VMR-9 for the IVMRFilterConfig9 interface.
  3. Set renderless mode.
  4. Notify the VMR-9 filter about your custom allocator-presenter object, and vice versa.

The following code shows Steps 1 through 3:

CComPtr<IBaseFilter> g_filter;
g_filter.CoCreateInstance(CLSID_VideoMixingRenderer9);
CComQIPtr<IVMRFilterConfig9> filterConfig(g_filter);
if (filterConfig)
{
    filterConfig->SetRenderingMode( VMR9Mode_Renderless )
}

Step 4 is shown in the following code:

HRESULT SetAllocatorPresenter(IBaseFilter *filter, HWND window)
{
    HRESULT hr;
    CComQIPtr<IVMRSurfaceAllocatorNotify9> lpIVMRSurfAllocNotify(filter);
    if (!lpIVMRSurfAllocNotify)
    {
        return E_FAIL;
    {
    // Create the allocator-presenter object.
    g_allocator.Attach(new CAllocator( hr, window ));
    if ( FAILED( hr ) )
    {
        g_allocator = NULL;
        return hr;
    }
    // Notify the VMR-9.
    hr = lpIVMRSurfAllocNotify->AdviseSurfaceAllocator(g_userId,
       g_allocator );
    if (FAILED(hr))
    {
        return hr;
    }
    // Notify the allocator-presenter.
    hr = g_allocator->AdviseNotify(lpIVMRSurfAllocNotify);
    return hr;
}

DirectShow is designed to support a variety of formats. Therefore, the DirectShow graph-building mechanism is quite flexible. An application can build the graph by connecting filters one-by-one, or it can rely on Intelligent Connect and let DirectShow build the best possible graph. For this example, the best approach is to add the VMR-9 filter to the graph manually, but let the Filter Graph Manager render the rest of the graph automatically. This approach is compatible with the majority of formats that are supported by DirectShow.

After you create the VMR-9 and configure it, as shown in the previous code, call IFilterGraph::AddFilter to add the filter to the graph. Then call IGraphBuilder::RenderFile to render the file:

hr = g_graph->AddFilter(g_filter, L"Video Mixing Renderer 9");
if (SUCCEEEDED(hr))
{
    hr = g_graph->RenderFile( wszPath, NULL )
}

The RenderFile method favors renderers already in the graph. It adds new ones only if the renderers in the graph cannot render the file. For details about the graph building in this example, refer to the code in the VMR9Allocator sample.

For More Information

To learn more about Microsoft DirectShow, see the DirectShow SDK documentation.