Creating Custom Error Pages with Live Search

For large web sites with extensive amounts of content, 2 to 10 percent of traffic is looking for pages that either don’t exist or have been moved. Most web servers return a generic 404 error page or a sitemap when a user’s desired page cannot be found. These unhelpful pages often result in a dead end for users.

With Microsoft’s Web Page Error Toolkit, you can create dynamic 404 pages that contain customized error messages along with search results seeded with relevant keywords to help your users move past the missing page and find the information they need.

The Toolkit is a customizable ASP.net application that replaces the default error page on your IIS server. The Toolkit enables you to use Bing (or any search engine) to return results for the specified domain and locale, control the number of results returned on your page, choose whether to offer spelling corrections, and customize your error message.

You also have the option of choosing from several keyword extraction strategies that are included with the install, or providing your own implementation.

What Users Typically See

When most people encounter an error page, they see a static page with a simple error message. Some sites try to help by offering instructions on how to correct the user's input, while others throw up a sitemap to make the user wade through all the content on the site, hoping they will find what they were looking for. None of these approaches is particularly helpful to the user, leaving them stranded and frustrated.

With the Bing Webpage Error Toolkit, you can help your users even when they encounter an error page on your site by providing contextual search results based the user's search terms.

Following is an example from the Microsoft website, utilizing the Webpage Error Toolkit to show a useful error page after the user typed in Offfice with an extra “f”:

In this example, the site’s search page was modified slightly to create the error page. The newly created page was then installed into the web server as the error page, providing the correct link to Microsoft Office.

Why You Should Care

Users usually encounter a missing page for one of the following three reasons:

• a broken link from an external web site

• a broken link from an internal page

• direct user navigation (the user typed in the URL incorrectly)

While each case has unique aspects, the common denominator is that a user was interested in specific content and your website did not deliver it. In each of these cases, the user is interested in your site, ready to be engaged, and is ultimately disappointed. Until now, an important opportunity has often been missed: to deliver an error page that can re-engage your users.

When a user has typed in a URL and arrived at an error page, we know the user was interested enough to remember (or copy) the URL. In this case, the URL probably contained enough text that applying a search query against that text could result in highly relevant results, which could keep the user engaged.

The error page is encountered often enough that content owners need to provide resources for understanding and dealing with it. On Microsoft web sites, error pages can be delivered on as many as 10 percent of all incoming requests. On www.microsoft.com and other large web sites, this can be millions of impressions per month. This level of traffic is worth understanding, especially when most scenarios involve an interested user. If a site delivers a relevant error page, it can engage the user with a meaningful and lengthy session (with multiple page views).

Measuring the number of times a site's error page is displayed, along with the referral page, is illuminating information for any content owner. The error page usage information provides excellent data for both site quality and for search engine monitoring.

How to Build a Better Error Page

A successful error search page is built through a number of steps:

1. Configuring the web server to use the error search page for errors

2. Extracting relevant information about the intended destination from the web server error information

3. Sending the information extracted as a query to a search engine, optionally scoping it to the domains you want to explore.

4. Creating the error search page and displaying the results returned by the search engine

5. Ensuring the error search page returns proper HTTP status codes

6. Monitoring error page usage information

These steps are the blue print for building a successful error search page for your website. Each web server responds differently to errors and your site’s engineering team will best understand how to configure errors for the server, including any custom processing done by your content management system.

When creating the error search page, integration into your content management system is essential. While many web sites use URLs that contain folders, and pages using words and phrases (such as www.microsoft.com/office/word), others use internal document identifiers that are harder to extract useful information from. Integrating with the content management system will allow programmatic extraction of useful search terms from the content database for mis-typed document identifiers. Other options include using information from the referring link text to provide search terms.

When creating the error search page it is important to make it fit into your site's overall design, but also to distinguish it such a way that users recognize it. The error message should be as straight forward as possible and should direct your users to the page’s search results. Complex error messages on this page often distract users from their original intent (to find specific content they are interested in). Another crucial aspect of the error page is to ensure that it returns suitable HTTP status codes and search engine meta tags that will enable search engines to recognize that this page should not be indexed.

Error page traffic should be monitored on a regular basis to ensure that the overall site is operating properly. A small amount of time spent with your site’s error page data will often return useful insights, such as finding internal pages with broken links, or pointing to errors in search engine indexing that you may want to research.

Summary

Improving a site’s error page can have many benefits. As part of Microsoft’s commitment to helping publishers maximize engagement on their sites through innovative technology solutions spanning search and advertising, we’ve created a “Webpage Error Toolkit” that can be used to integrate an error search page based on Bing (or the engine of your choice) into your website. The toolkit can be downloaded from Customize your 404 error pages with the Web Page Error Toolkit.