Architecture Guidance for Building Federated Search Connectors

  • Article

Microsoft Search Server 2008 provides you with the ability to include items from content repositories that are not indexed by the search server’s crawler through federated locations.

Search Server 2008 supports the following types of federated locations:

  • **Local Search Index   **Search results from the local search server.

  • **Sites that support the OpenSearch 1.0 or 1.1 standard   **The RSS feed for any remote search server's results page, or any searchable RSS feed that supports the OpenSearch standard and returns search results in a structured XML format (for example, RSS or Atom results).

In certain scenarios, you might want to include federated search results from other types of content repositories but do not want Search Server 2008 to crawl the content. Scenario examples include repositories that:

  • Cannot be crawled, possibly due to technology, resources, or content size.

  • Already have a search engine that works well.

  • Are secure, and can be accessed only by search queries.

To include federated results in these repository scenarios, you must create a federated connector, which is an application that provides an interface to the repository to expose results from the repository’s search engine. You must expose the results in the same way that they are exposed by OpenSearch federated locations.

To decide whether to crawl a repository’s content directly, or add it as a federated location, consider the following advantages to crawling the content directly:

  • You can sort results by relevance.

  • You can control how frequently the content index is updated.

  • You can modify crawled metadata.

  • You can perform a single backup operation for crawled content.

This topic discusses the requirements and provides some architecture guidance for creating a federated connector for Search Server 2008.

Federated Connector Requirements

This section describes the minimum requirements for a basic federated connector, starting with a review of how Search Server 2008 connects to an OpenSearch location and retrieves results, and then a discussion of how to implement similar functionality in a federated connector.

Federating Search from an OpenSearch Location

Before you build a federated connector for Search Server 2008, you should understand how the search server queries an OpenSearch federated location, and how it processes the returned results:

  1. The search server sends a GET request to the federated location by using the URL specified in the query template for the location definition information. The parameterized URL points to the location’s Web interface, and contains the search query and any parameters that are required by the search engine.

  2. The Web interface returns a response containing the search results in a structured XML format (for example, RSS or Atom).

    Note

    Search Server 2008 does not support results returned in HTML or XHTML format.

  3. Search Server 2008 formats the results based on the search results display metadata that is specified for the OpenSearch federated location, and displays it in the search results UI.

Based on these steps and the Search Server 2008 extensions to OpenSearch, the OpenSearch federated location, at a minimum, must be able to do the following:

  • Send queries to the search engine as URLs.

  • Return results as XML.

Required Functionality for a Federated Connector

Based on the requirements for an OpenSearch location, as described earlier, the federated connector must be able to do the following:

  • Provide a Web page or Web service for the search server’s query request, sent as a parameterized URL.

  • Retrieve the search query from the request’s URL parameters.

  • Submit the query to the repository’s search engine in the appropriate format.

  • Convert the repository’s search results into a structured XML format.

  • Return the search results XML in the response to the search server.

Federated Connector Sample Scenarios

A federated connector is required in certain search scenarios. This section describes some relevant sample scenarios, and offers implementation details for sample federated connector solutions to these scenarios.

Note

The following implementation details, provided for informational purposes, represent only a few of many possible solutions.

Scenario 1: Connecting to a SQL Server Database

Scenario background: Connections to the database require Windows authentication.

Possible solution: A Web application within a SharePoint site that contains a lightweight ASPX page that does the following:

  1. Constructs a SQL query by using the search terms passed in the request URL.

  2. Connects to the database as the user who submitted the query.

  3. Executes the SQL query.

  4. If records are returned, formats them as XML, and returns them in the ASPX page’s response to the search server.

Note

In this scenario, NTLM authentication works only if the database and the SharePoint site are located on the same server. If the database is located on a remote server, you must use Kerberos authentication.

For more information about Web applications within SharePoint sites, see How to: Create a Web Application in a SharePoint Web Site and Getting Started with Programmatically Customizing a SharePoint Web Site in Visual Studio.

Scenario 2: Connecting to an External Search Site That Returns Results in HTML Format

Scenario background: The site is configured to use Anonymous access.

Possible solution: Use a Web application outside of the context of a SharePoint site, which contains a lightweight ASPX page that does the following:

  1. Submits a search request to the site by using the search terms passed in the initial request URL.

  2. Converts the results in the HTML response received from the external search site to RSS format.

  3. Returns the RSS XML in the response to the search server.

In this scenario, the federated connector’s Web application could be located on a remote server; however, a simpler solution is to create the Web application within the _layouts folder for the SharePoint site. For more information about creating this type of Web application, see How to: Modify Configuration Settings for an Application to Coexist with Windows SharePoint Services.

In a variation for this federated connector solution, you can add support for multiple external search sites by modifying the ASPX page to include details for more than one site within a case statement. The query template specified for these locations could then include a custom parameter that specifies which site in the case statement receives the federated query. Another variation is to combine the results for multiple external search providers, incorporating logic to order the results based on relevance.

Scenario 3: Connecting to a Back-End Server Application That Exposes Search Results through an Object Model

Scenario background: The back-end server application is configured to use forms authentication, and the same user name and password are required for all queries.

Possible solution: Configure a federated location to use Corporate forms authentication, specifying the required user name and password in the location definition.

The federated connector is a Web application running on a remote server that contains an ASPX page that does the following:

  1. Retrieves the authenticated user credentials passed with the request.

  2. Invokes the object model method to query the back-end server application for the search terms passed on the request URL.

  3. Formats results as XML, and returns them in the ASPX page’s response to the search server.

Scenario 4: Connecting to a Back-End Server Application That Exposes Search Results through a Web Service

Scenario background: The Web service requires the user’s credentials for each query that is submitted, and the authentication mode is Windows NTLM.

Possible solution: Configure a federated location for User NTLM authentication.

The federated connector is a Web application running on the remote server that contains an ASPX page that does the following:

  1. Retrieves the authenticated user credentials passed with the request.

  2. Connects to a single sign-on (SSO) service to get the user’s credentials.

  3. Calls the Web service method to query the back-end server application for the search terms passed on the request URL, passing the credentials from the previous step in the request.

  4. Formats results as XML, and returns them in the ASPX page’s response to the search server.

Guidelines for Creating a Federated Connector

This section contains guidelines and recommendations for determining:

  • The scope for the federated connectors you are creating.

  • How to format and customize the display of the search results.

Scoping the Creation of the Federated Connector

When you are planning to create and implement your federated connector, consider how you would answer the following questions about the federated location’s content repository:

  • What authentication modes does the repository support?

  • Should the authentication credentials that are used be based on the user who submits the query, or should the same credentials be used for all queries to a specific federated location regardless of the submitting user?

  • Where is the physical location of the repository? On the same server as the SharePoint site, or on a remote server?

  • How do you access the repository’s search functionality? For example, through one of the following:

    • Web service

    • Object model

    • ASPX page

    • HTML content

  • Should the results for the federated location include any custom fields from the repository?

  • What results format customization (XSLT transformation) is required?

Displaying the Search Results

This section provides details about the format required for search results, and ways you can customize the display of the search results.

Generating the Search Results XML

Before returning the response that contains the results to the search server, you must convert the results into a structured XML format. We recommend converting to RSS format for a simple customization scenario, as this requires little to no change to the default formatting for the XSL and Properties fields in the federated location’s search results display metadata.

Returning Custom Properties

If the search results XML that is returned by a federated connector includes any custom properties, they are not displayed in the search results UI unless you override the default formatting of the federated location’s display metadata for the following fields:

  • Properties. Add the name of the custom property in the list of metadata. You must include the custom property in this list, or it will not be available in the XSL. The custom property must be a managed property.

    Note

    This only applies to SharePoint federated locations; for OpenSearch federated locations, all properties are available in the XSL.

  • XSL. Update the XSL to display the custom property.

The More Results link is the URL of an HTML page that displays results for the federated search location. To include this link when search results are displayed for the federated location, either include the location in this HTML page, or create your federated connector to provide the search results in an HTML page.

See also

Federated Search SQL Server Connector Sample [Search Server 2008]
Federated Search Overview [Search Server 2008]

Microsoft Search Server 2008
OpenSearch Specification