Novell is now a part of Micro Focus

Head-to-Head Comparison

Articles and Tips: article

01 Nov 1999


This document details the results from a head-to-head comparison between Novell's Internet Caching System (ICS) and Inktomi's Traffic Server Web caching solutions. It discusses five performance-related criteria for comparing the two products: peak performance, scalability, price-performance, viability of value-added services, and Inktomi's InkBench performance benchmark.

Novell Internet Caching System ICS vs. Inktomi Traffic Server

What is Web Caching?

Web caching (also known as proxy caching) is a relatively new means of improving browser response times and reducing network traffic. In a typical scenario, a Web cache acts as an intermediary between potentially thousands of users and the Internet. The Web cache accepts user requests and satisfies them using Web content that has been cached, prefetched, or retrieved directly from an origin Web server. By intelligently caching popular Web content close to the user, Web caches dramatically improve Web response times and greatly reduce network traffic.

Peak Performance

The primary function of a Web cache is to accelerate the delivery of repeatedly-used Web content to users. Since users typically request Web pages consisting of many individual Web objects, the measurement of how quickly a cache can deliver Web objects is a powerful indicator of a cache's peak performance.

Figure 1 compares the peak performance ratings for several Novell ICS solutions with the peak performance of Inktomi's Traffic Server on a fully loaded Sun Enterprise 450 server. These results demonstrate each system's ability to handle "flash crowds"—bursts of user requests that are typical of Web traffic—as well as each system's capability to handle growth and support value-added services.

Figure 1: Novell ICS solutions from both Compaq and Dell dwarf Inktomi's performance on the Sun Enterprise 450 platform.

The Compaq and Dell performance results in Figure 1 were produced with NLANR's Web Polygraph PolyMix-1 workload and run rules. The Inktomi Traffic Server PolyMix-1 result is based on Inktomi's 1999 Traffic Server Audited Performance Benchmark Report. (Inktomi recently cut the average size of the Web objects used by their proprietary benchmark by more than half, effectively doubling their performance results overnight. We've made a similar adjustment to their advertised result to bring it in line with Web Polygraph's results that are based on a more realistic average object size. For more details, see "Web Polygraph vs. InkBench" below.)

Since Inktomi's solution can service approximately the same 500 Web Polygraph PolyMix-1 requests per second as Compaq's mid-tier C1500R, a comparison of the hardware used by both systems is enlightening. Figure 2 compares the two systems and provides a clear picture of Novell's efficiency advantage over Inktomi's Traffic Server.

Figure 2: Novell ICS uses considerably less hardware than Inktomi's Traffic Server to service an identical workload (Sources: NLANR Web Polygraph PolyMix_1 results and Inktomi Traffic Server Audited Performance Benchmark report results).


 
Novell ICS (Compaq TaskSmart C1500R)
Inktomi Traffic Server (Sun Ultra Enterprise 450 server)

Web Polygraph PolyMix_1 peak requests per second

500

500

Number of CPUs

1

4

CPU speed

500 MHz

400 MHz

L2 cache size

512 KB

2 MB

RAM size

512 MB

2 GB

Number of disks

2

12

Software Licenses

Included

4 ($24,995 per CPU)

List Price

$13,999 (Compaq TaskSmart C1500R)

$159,980 (4 Traffic Server licenses + 1 $60,000 Sun Enterprise 450)

Given that Inktomi's Traffic Server requires this much hardware and software to deliver 500 Web objects per second, it can hardly be characterized as an efficient platform for additional value-added services.

Scaling the Internet

Inktomi's marketing literature talks about "scaling the Internet." Scalability is the ability to easily increase the size or configuration of a system to suit your changing conditions-usually your requirement for more horsepower. Using their own InkBench benchmark, Inktomi claims they can scale the Traffic Server solution to service 3,488 InkBench hits per second and save 147.7Mbps of bandwidth. But at what cost?

Inktomi used 16 Sun dual-processor 300MHz UltraSPARC machines with 512MB RAM and 8 4GB SCSI drives to demonstrate this level of scalability (based on Inktomi's Sun/Inktomi Large Scale Benchmark Report). Using an estimated list price of $25,000 for each Sun system, and $25,000 as the Traffic Server licensing fee for each of the 32 CPUs, you'll come up with a staggering total cost of $1,200,000. It's no wonder they didn't publish the price.

In Figure 3, Novell's scalability picture looks quite different. A cluster of three Novell ICS caches outscales Inktomi's cluster of 16 Traffic Servers by nearly 2-to-1.

Figure 3: A cluster of three Novell ICS caches outscales Inktomi's cluster of 16.

The three Novell ICS systems were uniprocessor 550MHz Intel Pentium III systems with 4GB RAM and 13 9GB SCSI drives. With a total solution cost of under $200,000, the Novell ICS solution is 1/5 the cost of Inktomi's, 1/5 the platform management costs, and nearly double the performance capacity. Novell ICS simply out-scales Inktomi's Traffic Server, at a fraction of the cost.

Price-Performance

Figure 4 compares the price-performance of several Novell ICS solutions from Novell's initial partners-Compaq and Dell-to Inktomi's Traffic Server on their performance platform of choice. The performance numbers used for this chart are based on IRCache's PolyMix-1 results and benchmark results published by Inktomi. Price-performance is calculated by dividing each system's list price by its peak performance rating.

Figure 4: Novell ICS solutions easily outperform inktomi's in terms of price-performance.

Compared dollar-for-dollar, Novell ICS solutions are less than one-tenth the initial cost of Inktomi's Traffic Server.

When comparing the costs of an Inktomi Traffic Server solution to a Novell ICS solution, don't forget to figure in Inktomi's higher initial purchase price as well as the costs of ownership for ten times as much hardware than would be required with the Novell ICS solution to provide the same level of performance.

Value-Added Services

Value-added services such as content filtering, identity brokering, and content transformation are important to the future of the Web. However, Inktomi's marketing messages about value-added services are at odds with the performance and underlying architecture of their platform.

In operating system terms, Traffic Server is a user-level application running on top of a general-purpose OS. Independent research studies have shown that general-purpose operating systems lack the efficiencies necessary for large-scale Internet server applications. According to one study, "The performance of Internet server applications on a general-purpose operating system is often dismayingly lower than what one would expect from the underlying hardware." (See the related document "Better Operating System Features for Faster Network Servers" for more information). Inktomi's cache platform efficiency and performance are a direct result of this architecture.

Inktomi's application architecture is better suited to back-end services than real-time services. Aside from Traffic Server, Inktomi's other service offerings, including their search engine, search directory, and shopping service, are back-end services that are appropriate for Inktomi's application architecture. These services support significantly lower transaction rates than caches that operate in the flow of Web traffic. Caches and value-added services designed to run in the flow and render services in real-time must be capable of servicing transaction rates an order of magnitude greater than those serviced by Inktomi's back-end systems. The Traffic Server platform is simply not up to the task.

Novell ICS's affordable performance capacity and efficient use of hardware resources position it as the ideal platform for these services. Novell's high-efficiency cache platform is the result of well-known strategic advantages in the Novell microkernel, protocol engine, and file caching system, combined with several recent technological advances in our cache object store and cache heuristics. This special-purpose design provides the underpinnings for high-precision value-added services with very fine service tolerances (see the related document "High-Capacity Internet Middleware: Internet Caching System Architectural Overview" for more information).

Novell ICS's high-efficiency provisioning of content filtering and blocking services, with industry-leading partners such as N2H2 and Log-on Data (X-Stop), provides validation for these claims. Novell ICS systems support real-time URL filtering with less than a 10% drop in peak capacity, demonstrating how Novell ICS's extensible framework allows third-party value-added services to take advantage of the same efficiencies and precision enjoyed by the Novell ICS core engine. This capability will allow the Novell ICS platform to support and accelerate the Web's most important value-added services.

Web Polygraph vs. InkBench

Inktomi's strategy for reporting performance is unique in the industry. In addition to using a proprietary tool called InkBench, Inktomi customers must sign rigorous non-disclosure agreements that limit who can view the performance of their systems and that forbid public results reporting. This secretive strategy stands in stark contrast to the computing industry's long-standing support of standard performance measurement tools and open results reporting.

Novell supports the open-source Web Polygraph benchmark. This tool was developed by an Internet research team (IRCache) at the National Laboratory of Applied Network Research (NLANR) with funding from the National Science Foundation (NSF). The IRCache team has years of experience as the developers and maintainers of the Squid code base as well as significant Internet research on their experimental global cache hierarchy in conjunction with Internet researchers around the world. This expertise, combined with their non-commercial status, uniquely qualifies NLANR and IRCache to develop an open, standard Web cache benchmark.

The Polygraph PolyMix-1 workload was first used during the first IRCache Bake-Off in March 1999. More recent versions incorporate the use of persistent connections, a more realistic object lifecycle, and a variety of cache directives. Although both Network Appliance and CacheFlow declined to participate in the first IRCache bake-off, NLANR's significant Web Polygraph enhancements, along with an aggressive development schedule prior to the next bake-off in January, 2000, led both Network Appliance and CacheFlow to join a mid-summer Web Polygraph competition for Data Communications magazine. The industry is rallying around Web Polygraph as the de-facto standard Web cache benchmark.

On the other hand, vendors and Web caching experts have little information about Inktomi's InkBench "Large Scale" benchmark. Since Inktomi won't allow their customers to disclose their performance measurements or the tool, we're skeptical of its results as well as Inktomi's private results interpretations. To make matters worse, Inktomi is manipulating their workload to increase the perceived performance of their systems—a marketing ploy not allowed with NLANR's Web Polygraph.

In October 1997, Inktomi published a "Sun/Inktomi Large Scale Benchmark" report where they used 16 dual processor Sun systems to service a combined workload of 3488 requests per second with a throughput rating of 399Mbps. This workload's throughput rating is similar to the throughput ratings for a Web Polygraph workload when run at the same 3488 request per second load, because both tools used a realistic average object size range of approximately 13-15 KB.

However, in 1999, Inktomi cut their workload's average object size by more than half, from 15KB to 6.2KB, effectively doubling their InkBench results overnight. Using a quad-processor Sun Ultra Enterprise 450 server with 2.5GB RAM and 74GB of persistent storage, Traffic Server purportedly serviced 1059 requests per second. The problem is it only served 54.7Mbps throughput to achieve that result, which is less than one-half the throughput a typical production system would have to serve at the same request rate. 54.7Mbps is also less than one-half the throughput of a Web Polygraph workload running at the same request rate of 1059.

Figure 5: Internet average object size statistics measured by 12 caches in NLANR's National Cache Mesh validate NLANR's 13 KB average object size recommendation for Web Polygraph (printed with permission from NLANR).

Realistic average object sizes are an important ingredient in Web cache benchmarks if you want meaningful results. Web Polygraph's standard 13KB average object size is the result of formal research studies and active measurement points throughout the Internet. Figure 5 is a plot of 1400 days of measurement data for daily average object sizes at twelve different Internet caches in NLANR's National Cache Mesh. From this chart, it is clear that NLANR's 13KB average object size is a good average, and Inktomi's 6.2KB average object size has nothing to do with "realism."

Although more information is necessary for us to fully understand InkBench and its workloads, it is clear that Inktomi's latest InkBench results have to be cut by at least half to be in the same ballpark as Web Polygraph results. We've cut their 1059 requests per second InkBench result down to 500 in an effort to represent their potential PolyMix-1 performance.

Conclusion

In this head-to-head comparison, a $13,000 Compaq TaskSmart (Novell ICS) solution equaled the performance of a $160,000 Sun Enterprise 450 Traffic Server solution. Novell ICS also outscaled Inktomi's Traffic Server with double the throughput savings for one—fifth the cost. These Traffic Server performance characteristics are a direct result of Inktomi's inefficient application architecture, rendering their platform overpriced and underpowered for caching as well as value-added services.

Inktomi's Traffic Server solutions simply don't measure up to their marketing messages. Interestingly enough, Inktomi is the only cache vendor to not participate in the planning meeting for the second IRCache Web Cache Bake-Off in January 2000. In fact, they have stated they will never attend an open performance forum. We invite Inktomi to validate their marketing literature's claims of "extreme performance" by entering IRCache's Bake-Off events where Inktomi's performance can be openly measured and published for the benefit of all Web cache customers.

Related Reading

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates