Middleware Company Study: .NET vs. IBM WebSphere

The Middleware Company recently published a detailed study comparing .NET 1.1 on Windows Server 2003 to IBM WebSphere 5.1 on RedHat Linux.  The study compares developer productivity, application performance, manageability and reliability of the two platforms.  As such, the study is the most comprehensive hands-on comparison published on these two platforms to date, and was unique in its design.  To complete the study, Middleware created an application specification for a connected system known as ITS, an automated facilities management application with both B2C and B2B integration via Web Services and Message Queuing, web application front ends and mobile device integration.  The Middleware Company then assembled two teams of senior developers (three developers on each team), one to develop the ITS application with WebSphere using J2EE, one to develop the application with Microsoft Visual Studio.NET.  Each team was similarly skilled on their respective platforms.  The full Middleware Company report can be downloaded from:

http://www.microsoft.com/windowsserversystem/facts/analyses/middleware.mspx

Team Makeup

The J2EE/WebSphere team consisted of three senior developers from the Middleware Company, which operates the largest third party J2EE architect/developer forum and training site on the Internet (TheServerSide.com).

* Includes experience with Java servlet API predating the introduction of J2EE in 1999.

Additionally, the J2EE team used two independent, IBM-certified WebSphere consultants at different times during the deployment and tuning phase. 

• One had three years’ experience as a WebSphere administrator on various Unix platforms, including Linux.
• The other had over four years’ experience installing, configuring and supporting IBM WebSphere on multiple platforms, including Linux.

The .NET team consisted of three senior developers from Vertigo Software, a California-based Microsoft Solution Provider, with the following credentials:

Developer Productivity

During this phase of the study, developer productivity was carefully measured in a laboratory setting for each subsystem of the application.  Unlike past comparisons of developer productivity based on lines-of-code counts, this study actually compares real man-hours of development to complete the application to the specification.  The Middleware report also documents the development process and development logs which provide interesting technical details about technical roadblocks, workarounds, architecture chosen, etc. for each platform.  The team developing for IBM WebSphere used IBM’s Rational Rapid Developer (RRD) to build the application (a model-driven tool that enhances developer productivity); while the team developing for .NET used Visual Studio.NET and C# to build their implementation.

Application Tuning and Configuration

During this phase of the study, each team was given the opportunity to tune and configure their developed ITS implementation(s) on production hardware consisting of 4 backend 4-processor servers (1.4 GHz CPUs with 2GB RAM) connected to the two independent ITS databases running on 8-processor servers (900 MHz CPUs with 4GB RAM).  The teams were given advance knowledge that each implementation would be put through a series of performance, reliability and manageability tests while the application operated under load.  Hence, this was their opportunity to prepare for these tests using Mercury LoadRunner scripts that drove high user loads on the system.  Performance, failover, reliability and manageability were all issues the teams had to address during this phase of configuration tuning prior to the actual tests being performed by the Middleware Auditor.  It should be noted that based on the performance results of the IBM WebSphere application initially developed using Rational Rapid Developer (RRD) to enhance developer productivity, Middleware subsequently developed a second IBM WebSphere implementation using WebSphere Studio Application Developer (WSAD).  The results for each WebSphere implementation are shown below.

Application Performance Under Load

Next, the ITS system was placed under a series of performance load tests to determine the overall peak throughput of each implementation.  These tests included both modular tests of the individual subsystems, as well as an integrated test of the entire connected system as it would operate in a production environment.  Mercury LoadRunner was used to drive load across the system.  In this test, the subsystems were fully integrated via Web Service requests and message queuing technologies.  Details of the results for other performance tests can be found in the complete Middleware report.

Application Manageability

After testing the performance of the implementations, Middleware constructed a series of manageability tests to test how quickly the teams could develop and deploy changes to the system.  Also, teams were requested to deploy the changes to the servers while they operated under load, to simulate a production system.  The deployment process, error rates, system downtime and the ability to maintain user session state was measured during deployment under load.

For this part of the test, three change scenarios were tested:

1. Work Order Subsystem Database Query Change. Results generated from the Ticket Search Page needed to be ordered in descending order by date with most recent tickets displayed first.

On the ITS Customer Service subsystem, the following change was mandated:

2. Adding a new data-driven page to the Customer Service application. This page allows administrators for each company to generate new news bulletins that are displayed for their company.  This part requires 2 changes: adding the web form to allow news items to be submitted and stored in the database, and adding a column to the database table allowing news items to be tracked by unique customer id. 

3. Add databinding to a form element in the Customer Service Application.  For this change, a dropdown list that was hardcoded needed to be data-bound to a new database table on the backend.

Next, the changes were deployed (in separate deployments) to the backend hardware while running under load.

Application Reliability

During this phase of the study, the teams were given the task to operate the system under moderate load while responding to various conditions typical of a live system to test overall application reliability.  The test involved testing graceful shutdowns of clustered servers for maintenance, re-integrating new servers into the cluster, and failover scenarios where servers in the cluster were powered down.  Teams were instructed to operate the application with minimal interruption to users.  Error rates, session state and site downtime/app restarts were tracked and reported in this part of the study.

Conclusions

The Middleware Study is a detailed study of various characteristics of IBM Websphere 5.1 deployed to Linux vs. Microsoft .NET 1.1 deployed to Windows Server 2003.