Novell is now a part of Micro Focus

Head-to-Head Comparison

Articles and Tips: article

01 Nov 1999


This document details the results from a head-to-head performance comparison between Novell's Internet Caching System (ICS) and CacheFlow's Web caching solutions. It discusses four performance-related criteria for comparing the two products: peak performance, response time behavior, price-performance, and cache freshness.

Novell Internet Caching System (ICS) vs. CacheFlow

What is Web Caching?

Web caching (also known as proxy caching) is a relatively new means of improving browser response times and reducing network traffic. In a typical scenario, a Web cache acts as an intermediary between potentially thousands of users and the Internet. The Web cache accepts user requests and satisfies them using Web content that has been cached, prefetched, or retrieved directly from an origin Web server. By intelligently caching popular Web content close to the user, Web caches dramatically improve Web response times and greatly reduce network traffic.

Peak Performance

Performance comparisons between competing Web cache products are usually characterized as "apples-to-oranges" comparisons due to significant differences in their hardware platforms. However, CacheFlow's pre-packaged solutions contain Intel motherboards, which are a supported platform for Novell's Internet Caching System (ICS). This enabled us to first benchmark CacheFlow's system, then open up the box, install an image of the Novell ICS software, and benchmark ICS running on CacheFlow's hardware for an "apples-to-apples" comparison.

Figure 1 shows the results of a peak performance test of the two products at both ends of CacheFlow's product line. In the low-end test, Novell ICS running on the CacheFlow 100 platform achieved a peak performance of 115 requests per second, while CacheFlow reached a peak of 25 requests per second on the same platform. In the high-end test on the CacheFlow 5000, the CacheFlow software fully utilized the CPU when servicing its peak of 660 requests per second.

Figure 1: The peak performance of Novell ICS is more than four times that of CacheFlow on identical hardware.

After imaging the CacheFlow 5000 with the Novell ICS software and running a preliminary test, we found that the hardware was underutilized. By adding six more drives to the existing six and a Gigabit network card, we were able to balance the capabilities of the system and generate a true peak performance measurement of 2200 requests per second for Novell ICS on the CacheFlow 5000 platform. Even at this peak load, the Novell ICS solution utilized only 74% of the system's CPU—leaving significant CPU headroom for additional services. In both tests, Novell's ICS v1.0 software outperformed its CacheFlow counterpart by more than a four-to-one margin when run on identical hardware.

Note: Although the CacheFlow 100 successfully completed request rates of up to 35 requests per second in our tests, we discarded those results due to excessive latencies. (See the "Response Time Behavior" section below for details.)

These results were produced using IRCache's Web Polygraph PolyMix-1 workload and run rules. The CacheFlow 100 hardware was an $8,900 solution containing an Intel Pentium II 233 MHZ processor, 128 MB of RAM, a 2.38 GB EIDE drive, and an on-board Intel E100B network card. The CacheFlow 5000 features an Intel Pentium III Xeon 500 MHZ processor, 2 GB of RAM, six 9 GB SCSI-3 drives, and an Ethernet Gigabit network card.

To simulate real-world Internet usage patterns, the PolyMix-1 workload generates a realistic disk access pattern in the Web cache by accessing a very large set of working data. The resulting workload forces Web caches to access their persistent storage to service more than 90% of their cache hits. At peak levels, the disk naturally becomes the bottleneck for both Novell ICS and CacheFlow. With the disk bottlenecked at peak performance in both systems, these test results demonstrate the efficiency (or lack thereof) of each solution's persistent cache storage technologies.

Figure 2 shows the results of running the PolyMix-1 bench mark in NLANR's PolyLab facility on a variety of hardware plat forms from two of Novell's OEM vendors for ICS: Compaq and Dell. These results demonstrate the scalability of Novell ICS across the product range, contrasted with the results from similar tests conducted by CacheFlow.

Figure 2: Novell ICS solutions dwarf CacheFlow's performance capabilities (based on NLANR's Web Polygraph results at http://polygraph.ircache.net/Results/pmix-1.dell/ and numbers published in Cacheflow's Product Family Guide).

In a recent review (Sep. 1999), PC Magazine compared two Novell ICS solutions to the CacheFlow 500. However, their WebBench benchmark—originally designed to test Web servers rather than proxy caches—was flawed by several unrealistic characteristics:

  • PC Magazine used a small average object size of 8KB which has the effect of producing inflated results.

  • PC Magazine offered the caches an 80% cache hit rate, which also inflates their results.

  • PC Magazine used a very small working set of data, allowing the caches to serve far too much content from RAM rather than from disk.

When PC Magazine introduced an artificial T1 simulator to the testbed, these workload characteristics produced inflated results that can't be duplicated in the real world (compare PC Magazine's result of 120 for the CacheFlow 500 to the more realistic peak of 50 advertised by CacheFlow for the same system). This type of workload exaggerates the value of a system that is just barely able to support T1 capacity, while underestimating the value of more capable systems.

In Figure 3, the Dell results and curves are nearly ideal. Dell's throughput curve is virtually identical to CacheFlow's, topping out at approximately 120 requests per second. What this chart doesn't tell you is that the CacheFlow 500's CPU was saturated, while the Dell system was basically idling at approximately 20% of capacity. In the absence of the T1 bottleneck between the cache and the origin servers, the Dell system was capable of approximately 600 requests per second. However, if this system was deployed in a T1 environment where its performance was throttled by the T1's bandwidth capacity, the system's CPU headroom provides additional capabilities for cached services such as URL filtering and blocking.

Figure 3: Dell's ICS-130B and the CacheFlow 500 produce similar throughput ratings when constrained by a T-1 link in a recent PC Magazine review.

Note: The Compaq results in Figure 3 are misleading. Novell and Compaq provided PC Magazine an early release of Novell ICS Service Pack 1 to fix a TCP/IP bug, but the magazine neglected to publish any amended results. The improved results parallel the Dell and CacheFlow curves on the left side of the chart, then follow the Eolian curve on the right side of the chart. The difference between the resulting Dell and Compaq curves is due to the Compaq system having a single drive, whereas the Dell system had two.

Novell's consistent performance lead over CacheFlow is the result of well-known strategic advantages. Novell has a long history of high-performance network services and special-purpose OS research and development. The ICS microkernel is the result of this experience, combined with collaboration with the core Unix System 5 team from Bell Labs now working at Novell. The ICS file system is a patent-pending persistent storage technology called Cache Object Store (COS) that is finely tuned for Internet data types. This special-purpose design provides the underpinnings for high-precision caching services. (See the related document "High-Capacity Internet Middleware: Internet Caching System Architectural Overview" for more information).

Response Time Behavior

Figure 4 compares CacheFlow's response times, also called "latency," with those of the Novell ICS software running on identical hardware. While the low-end CacheFlow solution can handle up to 35 PolyMix-1 requests per second, the requests are serviced very slowly, with a sizable amount of latency being added to browser requests. At 25 requests per second, CacheFlow generates an acceptable mean response time of 1.35 seconds. At 30 requests per second, their response time is over 5 seconds—an eternity in a cached infrastructure. At 35 requests per second, CacheFlow's mean response time is over 15 seconds per request.

Figure 4: Novell's ICSs response time (latency) remains nearly unchanged at 1.3 seconds, while CacheFlow's response time climbs above 15 seconds per request under peak workload conditions.

These response times challenge the utility of CacheFlow's solution in even the smallest of caching applications. At 5 to 15 seconds per request under peak loads, it is arguably faster to bypass the cache altogether and make round trip requests to the origin Web servers.

Novell ICS's response time behavior on the same hardware is ideal, remaining nearly unchanged over the system's full range of performance capacity from 1 to a peak of 115 requests per second. This behavior produces a predictable quality of service (QoS) during the bursts of activity common to Internet traffic—a quality CacheFlow is unable to match.

The PC Magazine review mentioned earlier also published a latency chart (see Figure 5). This chart is interesting because the CacheFlow 500 exhibited some alarming delays, in the six to seven second range, during the test (seen on the left side of the chart).

Figure 5: PC Magazine uncovered significant latency problems in their CacheFlow 500 results.

Price-Performance

Novell's ICS product strategy integrates the fastest Web caching software with the latest hardware platforms available from Compaq, Dell, and other OEMs. These Novell ICS solutions easily outperform the price-performance of CacheFlow's systems. In fact, CacheFlow would have to reduce the price of their CacheFlow 100 to $625 to compete with Compaq's and Dell's Novell ICS solutions on a cost-per-request basis.

Figure 6 compares the price-performance of several Novell ICS solutions to CacheFlow's (based on PolyMix-1 results and CacheFlow's Product Family brochure, Feb. 1999). Price-performance is calculated by dividing each system's list price by its peak performance rating.

Figure 6: Novell ICS solutions easily outperform CacheFlow in terms of price-performance.

Note: Again, we used 25 requests per second for the CacheFlow solution in this chart, rather than their peak, due to CacheFlow's unacceptable response times at higher rates. Even at 35 requests per second, CacheFlow's cost per request is $254.

Based on these results, CacheFlow systems would take ten times as long as Novell ICS solutions to pay for themselves. At one-quarter of Novell ICS performance capacity, CacheFlow systems will also require more frequent upgrades and replacement to keep pace with Novell's solutions.

Conclusion

In this head-to-head comparison, the Novell ICS solution delivers more than four times the performance capacity of a comparable CacheFlow solution on identical hardware. In addition to poor peak performance ratings throughout their product line, CacheFlow response times climb to over 15 seconds per request under load, whereas Novell maintains a 1.3 second response time. Novell ICS solutions also cost as little as 1/26 of CacheFlow's solution on a cost-per-request basis.

We invite CacheFlow to validate their marketing literature's claims of "extremely fast system performance" and "the industry's highest-performance cache solution" by entering IRCache's Bake-Off events where their performance can be openly measured and published for the benefit of all Web cache customers.

Related Reading

—Ron Lee

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates