NetWare on One CPU Outperforms Windows NT Server on Four CPUs
Articles and Tips: article
Senior Research Engineer
Novell Systems Research
DREW MAJOR
Chief Scientist
Novell Inc.
01 Jun 1996
Unlike general-purpose operating systems such as UNIX and Windows NT, NetWare is a special-purpose operating system that is specifically designed to host network services. As such, it is a superior platform for special-purpose network applications such as databases and Web servers. Using the NetWare Web Server as an example, this AppNote describes why NetWare, running on a single-processor machine, significantly outperforms Microsoft's Internet Information Server/Windows NT Server running on a machine with four processors. Results from these tests clearly show NetWare's strengths as a special-purpose application server.
This AppNote is an update of the handout entitled "Benchmark Results Show Off NetWare's Strengths as a Special-Purpose Application Server" that was distributed at Novell's BrainShare conference in March 1996.
Introduction
NetWare is often directly compared to general-purpose operating systems such as UNIX and Microsoft's Windows NT. However, this is an apples-to-oranges comparison. NetWare is a special-purpose operating system that is specifically designed to host network services. As a platform for high-performance network services such as directory services, and for special-purpose applications such as databases and Web servers, NetWare provides strengths that general-purpose operating systems cannot.
This AppNote presents the results of tests performed on two Web server platforms: Novell's NetWare Web Server running on NetWare 4.1, and Microsoft's Internet Information Server (IIS) running on Windows NT Server 3.51. These results clearly show NetWare's strengths as a special-purpose application server--in this case, as a Web server. The results are even more impressive since the NetWare Web Server was running on a single-processor machine, while Microsoft's IIS/Windows NT Server was running on a machine with four 133MHz processors.
Much has been said recently in the press and in competitors' marketing documents concerning the importance of performance in a Web server. As we will see, NetWare's special-purpose design allows it to provide superior performance as a platform for a Web server application.
Performance Analysis
In the past, Web server benchmark results showed Microsoft's implementation outperforming NetWare Web Server by a margin of 2 to 1. That was before Novell tuned the NetWare Web Server to take advantage of NetWare's efficiencies. These latest results demonstrate the magnitude of difference available to developers of specialized network services and applications on the NetWare platform.
Novell tested two Web server implementations:
Novell NetWare 4.1 and NetWare Web Server 2.5 (running on one CPU)
Microsoft NT Server 3.51 and Internet Information Server(running on four CPUs)
Both implementations used a Zenith Z Server MX with 128MB of RAM, dual Adaptec AIC7870 SCSI adapters, and Intel EtherExpress PRO/100 Smart LAN adapters, running on a 100Mbps Ethernet LAN. The sole difference between the two implementations was NT's use of four 133MHz CPUs in a symmetrical multiprocessing (SMP) system.
We used the WebStone benchmark to produce these results because WebStone's measurements closely resemble the performance of real Web server configurations. In fact, WebStone uses the actual service protocol (HTTP) and back-end service provider (Web server) that will run in the final production system. Since WebStone issues HTTP requests to the real Web server running on the system under test, its measurements are much more accurate than artificial benchmarks such as NetBench or ServerBench in measuring the performance of Web servers.
Two of the most significant performance numbers produced by Web-Stone describe the total capacity of a Web server implementation: throughput and connections per second. We have added our own CPU utilization measurements to give you a more accurate picture of how the Web server platforms scale while providing Web services.
Throughput
WebStone measures Web server data throughput in megabits per second (Mbps). This measurement shows the average per-second throughput capacity of the entire system. Figure 1 charts the throughput results for both the NetWare and NT implementations. (In this chart, larger numbers show higher levels of throughput.)
Figure 1: NetWare running on one CPU produces greater throughput than NT running on four CPUs.
The results in Figure 1 show the NetWare implementation on one CPU outperforms the NT implementation on four CPUs by a significant margin. These results also indicate that Microsoft's claims that MS IIS scales well have yet to be substantiated. The Microsoft solution on one CPU can only handle half the throughput of a NetWare server. When scaled up to four CPUs, IIS's performance is only marginally improved. Thus the Microsoft solution doesn't make efficient use of additional processors.
Connections per Second
WebStone also measures Web server connection performance in connections per second (CPS). Figure 2 compares the CPS capabilities of NetWare and NT on one CPU, along with NT on four 133MHz CPUs in an SMP system.
Figure 2: NetWare running on one CPU handles more connections per second than NT running on four CPUs.
As with the throughput results in Figure 1, Figure 2 shows the NetWare implementation on one CPU services a greater number of connections than NT--even when NT has the advantage of four CPUs.
CPU Utilization
Both NetWare and NT measure CPU utilization. We recorded the CPU utilization during each test. NetWare's implementation on one CPU stayed in the mid-90% range of CPU Utilization throughout the test. The sum of NT's CPU utilization, running on four CPUs, stayed in the 225% to 250% range. These results show that NetWare required less than one CPU to do the same work NT required two and a half CPUs to do--that's efficiency!
In a marketing document published on the World Wide Web (see http://www.microsoft.com/intdev/server/iisperf.htm), Microsoft lists multi-purpose servers as one of the reasons contributing to the increasing focus on performance for Web servers and states that "high performance Web server software leaves more CPU and memory resources available for other applications...." Based on the above test results, Microsoft's solution is very inefficient in its use of CPU and memory resources, and it certainly doesn't justify the added expense of multi-processor hardware.
Why NetWare Is Faster
On the face of it, it appears counter-intuitive that any piece of code running on a single-processor machine could outperform similar code running on a multiple-processor machine. A multi-processor system has more than one CPU to throw at the task, so it seems it should have an inherent advantage.
The truth is that although an SMP system offers the potential of harnessing multiple processors, the performance of the entire system--hardware, software, and workload--has more to do with the efficiency of the underlying software than with the quantity of hardware alone. LAN drivers, transport protocols, service protocols, operating system, and even the file system all play an integral part in the performance and scalability of the total system.
A Web server is a good example of a specialized service that can take advantage of efficiencies in the underlying software. A Web server is focused on data delivery, making file services its primary function. In fact, Web server benchmarks do nothing but request file services. There are no GUI tests or video tests--only tests to exercise the Web server's ability to service file requests to as many connections as possible with the smallest latency. The ideal platform for a high-performance Web server, then, is a NOS that provides efficient, high performance file services.
Novell has continually streamlined NetWare to be an efficient and scalable platform not only for file services, but for other value-added network services as well. Because NetWare is a specialized network operating system, Novell has been able to make two important assumptions regarding its design:
Core processes will run more efficiently if they are allowed to run to completion without being interrupted.
Incoming requests can be handled more efficiently if they have equal priority.
These two assumptions reduce the overhead of extensive process switching and scheduling, allowing NetWare's core code paths to be highly optimized. It is primarily due to these inherent efficiencies within the NetWare operating system that the NetWare Web Server can service HTTP requests several times faster than other implementations.
General-purpose operating systems can be coerced into operating as platforms for specialized network services. However, because they are not optimized to provide efficiencies that scale well, general-purpose operating systems typically require additional resources such as CPUs and memory to bring performance up to an acceptable level. This additional investment in hardware is difficult to justify when a NetWare-based system on one CPU does the job at a significantly lower cost.
Recent tests conducted by Ziff-Davis Labs comparing Windows NT Server and NetWare found that the maximum throughput for NT was less than half that of NetWare. These tests were run on a Fast Ethernet LAN, which essentially moved the performance bottleneck from the network to the server itself. According to their published report, "providing a larger pipe to the server helped NetWare tremendously, but Windows NT Server could not use this extra bandwidth, simply because it does not handle read and write request as efficiently as NetWare." (See "Server Power," PC Magazine, March 26, 1996, p. 161.)
Other value-added network services such as messaging and database engines will see the same performance gains when optimized for NetWare. For those applications that are efficient and scale well, NetWare SMP extends NetWare's high-performance platform across multiple CPUs. These performance characteristics position NetWare as the ideal special-purpose application server for specialized network services and applications.
How We Tested
The following test configuration is similar to the one set up at Novell's BrainShare technical conference held in March 1996 in Salt Lake City, Utah. However, the testing described in this AppNote is based on the latest shipping version of the NetWare Web Server software, not on pre-release versions as was the case for the BrainShare demonstration.
The WebStone Configuration
For our testing of NetWare Web Server and Microsoft Internet Information Server, we used the WebStone benchmark. WebStone is a client-server benchmark for HTTP servers published as freeware by Silicon Graphics, Inc. (SGI).
We ran WebStone on 96 real clients, testing in turn with 16, 32, 64, and 96 rather than simulating the client workload by spawning Web children on one to four Unix workstations. We tested 128 clients by configuring each of 64 real clients to run as two. Each test ran for 5 minutes; all of the tests were repeated several times.
WebStone measures two types of throughput: aggregate and per-client. Both measurements are averaged over the entire test time and client base. Aggregate throughput is the total bytes (body + header) transferred throughout the test, divided by the total test time. WebStone divides the aggregate throughput by the number of clients to arrive at a per-client figure.
A copy of WebStone, along with README and Frequently-Asked-Questions (FAQ) documents, are available at:
http://www.sgi.com/products/WebFORCE/WebStone
System Configurations
The NetWare hardware platform was a Zenith Z Server-MX with one Pentium/133 CPU with 512KB of L2 cache. The Zenith included 128MB of RAM, dual Adaptec AIC7870 SCSI adapters, one 1GB Seagate Hawk drive, three 2GB Quantum drives, and one Intel EtherExpress PRO/100 Smart adapter.
The Novell NetWare operating system was NetWare 4.1 with all released patches loaded. We also used the released version (version 2.5) of the NetWare Web Server.
|
For more information on the use of Polling Mode, see "The Benefits of Using Intelligent LAN Adapters in NetWare Servers"in the May 1995 Novell Application Notes. |
Additional settings were made as follows:
The ODI-compliant LAN driver was loaded with a command-line setting of MODE=1 to enable Polling Mode (the default is Interrupt Mode).
SET Minimum Packet Receive Buffers = 512
SET Maximum Packet Receive Buffers = 640
HTTP logging was enabled (the default is enabled).
The NT hardware platform was also a Zenith Z Server-MX but with additional CPUs for the multi-processor tests. The NT Zenith included four Pentium/133 CPUs each with 512KB of L2 cache, 128MB of RAM, dual Adaptec AIC7870 SCSI adapters, one 1GB Seagate Hawk drive, three 2GB Quantum drives, and one Intel EtherExpress PRO/100 Smart LAN adapter.
The Microsoft NT operating system was NT Server 3.51 with Service Pack 3 loaded. In addition, we purchased the Microsoft Value Pack ten-license pack, which includes Microsoft IIS 1.0 free of charge.
Additional settings were made as follows:
Using REGEDIT, we increased Listen Backlog from 0xF to 0x96, the equivilant of increasing the setting from 15 to 150.
We also added a registry entry that enabled us to set ObjectCacheTTL = FFFFFFFFh. This setting was suggested by Microsoft to optimize server cache for IIS.
The NDIS-compliant driver was used for the Intel EtherExpress PRO/100 Smart PCI LAN adapter.
HTTP logging was enabled (the default is enabled).
Both server hardware platforms were identical, with the one exception being NT's use of four Pentium/133 CPUs in a symmetrical multi-processing configuration where NetWare used only one. Microsoft's CPU utilization figures are the sum of the CPU utilization from all four CPUs.
The distribution of Web pages on the disk system had little to do with performance since the test ran entirely from server cache. However, we still used drive spanning when setting up both the drives for NetWare and NT. Both NetWare and NT used FAT file types.
The WebStone system was an HP Vectra XU 5/90 running SCO UnixWare v2.0. The Vectra included two Pentium/90 CPUs, 40MB of RAM, and one Intel EtherExpress PRO/100 PCI LAN adapter.
The Web clients were Compaq DeskPro XL 5133 systems. The Compaqs included one Pentium/133 CPU, 32MB of RAM, Microdyne NE10/100 PCI LAN adapters. The LAN driver was an ODI-compliant driver. Each client was running WEBCLIENT.EXE, Microsoft Windows 95, and Novell's Client 32 network client.
The LAN was 100Mbps Fast Ethernet using Category 5 UTP cabling. The LAN was connected into a star topology using 11 twelve-port Bay Networks BayStack 100BaseT hubs.
Conclusion
As shown by the benchmark results presented in this AppNote, NetWare is a superior platform for high-performance network services and special-purpose applications such as databases and Web servers. As a special-purpose application server, NetWare provides significant strengths that general-purpose operating systems cannot match.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.