Permormance Analysis: Isolating the Real Bottleneck in a System
Articles and Tips: article
Senior Research Engineer
Systems Research Department
01 Jan 1996
A System Is Only As Strong As Its Weakest Point All systems are constrained by their weakest point. Networks are no different. In network computing systems, bottlenecks are weaknesses that limit the full potential of the system. In this discussion you'll see that every system has a bottleneck --some detrimental, others less so. The trick is identifying which component is the real bottleneck and understanding how the bottleneck "moves" from one component to another when you upgrade individual components.
A Well-Designed System
Figure 1 represents a system constructed with components a, b, c, d, and e. Each of these components is placed on a scale that represents performance potential or capability. Component a is on the low-performance end of the scale, meaning it has less capability than component b located further up the scale. Component b has less capability than component c, and so on, until we reach component e which has the highest performance capability.
Figure 1: A well-designed system with all components within the same performance range.
Because component a has the least performance capacity, it is the bottleneck in this system. The rest of the system, as a whole, can't operate any faster than component a's limitations will allow. Upgrading component a to the performance level of component e would provide a slight increase in performance. But the overall increase wouldn't be as much as you might expect because component b would become the new bottleneck and the system would then be constrained by its limitations. From a performance point of view, however, this is a well-designed system because all of the components are within the same performance range.
A real-world example of this type of system might be a Pentium Pro-based server platform with PCI disk and intelligent LAN adapters, multiprocessing capabilities, a tuned cache, and fast disk storage array.
A Poorly-Designed System
Figure 2 shows another system comprised of components f, g, h, i, and j. In this system, components f and g have considerably inferior performance characteristics than components h, i, and j. With component f as the bottleneck, components h, i, j, and even g are constrained by f's meager capabilities to run much slower than their designs would otherwise permit. In this system the bottleneck is significantly hampering the system's performance.
Figure 2: A poorly-designed system with an inefficient mix of low- and high-performance components.
This is a poorly-designed system because h, i, and j aren't able to perform anywhere near their full potential. In fact, this system may have cost more than the system in Figure 1 because of the high-performance characteristics of h, i, and j. But it cannot perform as well because of the greater disparity between its low- and high-performance components.
A real-world example of this type of system could be any high performance server that is using a 16-bit, dumb adapter (such as the NE2000) as its LAN adapter. The overall performance of such a server is severely limited by the LAN adapter.
Moving Your System Bottleneck
The unbalanced system in Figure 2 would benefit noticeably if you were to upgrade component f to the performance level of g. But as soon as f is upgraded beyond g (see Figure 3), all additional performance characteristics in f will be lost because the bottleneck moves to component g. Thus you derive only a partial return on your investment in the upgrade.
Figure 3: System bottlenecks can be moved, but other components take their place.
Both components f and g must be upgraded to h's level before a full return on investment in h, i, and j can be reached (see Figure 4).
Figure 4: Upgrade all lower-performance components for a full return on investment.
Bottlenecks Aren't Limited to Hardware Components
The examples above use plug-and-play components, such as LAN adapters and CPUs, to demonstrate the effects that each component's performance characteristics can have on the resulting system's performance. But in reality, bottlenecks can be created by a great number of subcomponents and component interactions. Protocols create bottlenecks. Specific software functionality creates bottlenecks. In fact, every part and process of a system is a potential bottleneck.
It's important to keep in mind that all systems have a bottleneck somewhere. You can't really eliminate bottle-necks; you can only move them from one component to another. So your goal, as shown in the examples above, is to match the performance capabilities of your system's components in such a way that no single component is hampering the performance of any other component.
Another point to remember is that only one bottleneck--the system's weakest point--can exist at one time. This sublety plays havoc with all kinds of performance analysis projects, including benchmarks. It is futile to try to test one component of a system to its fullest capacity without realizing that another component is actually bottlenecking system performance. You can spend a lot of money upgrading other system components, but you won't see any significant improvement until you isolate the real system bottleneck and take appropriate action.
The Bottom Line
Bottleneck isolation is critical if you're building high-performance systems. You need to know how and where your system is bottlenecking. Moreover, you need to know how tightly that bottleneck is coupled with the performance characteristics of the rest of the system. If the guilty component is in a different performance league than the other components, you may reap greater return on investment by making the necessary upgrades to move the bottleneck to the highest possible performance level.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.