Novell is now a part of Micro Focus

Using Packet Size Distributions to Uncover Hidden Network Utilization Bottlenecks

Articles and Tips: article

RON LEE
Senior Research Engineer
Systems Research Department

01 Jan 1996


Inefficient packet size distributions in production workloads create bottlenecks that are difficult to isolate. This AppNote describes this phenomenon and includes documentation for PSD.EXE, a freeware DOS utility designed to record the distribution of packet sizes on a LAN segment.

The Impact of Packet Size Distributions

Packet size distribution (PSD) is a characteristic of network traffic that describes the size of packets traveling across the network. If every packet in production workloads were the same size, we wouldn't have any need for a distribution statistic. But in reality, production networks are host to a wide range of packet sizes, and the packet size distributions of most production workloads are far from ideal.

Generally, you can characterize most networks by saying that about half of the packets are very large and the other half are very small. Packet size distribution plays a significant role in the performance of networks. It determines the efficiency of the workload--which, in turn, has a direct impact on how efficiently your network's bandwidth is used.

The Impact on Benchmarks

Packet size distribution has a marked effect on benchmark results. Figure 1 displays the results of an Ethernet stress test using the PERFORM3 benchmark utility. The chart shows that Ethernet's bandwidth is used most efficiently when the test was configured to generate packets equal to the maximum frame size of 1514 bytes. The chart then shows a diminishing amount of throughput as the packet size is decreased.


How We Tested

The results in Figures 1 through 4 were generated using PERFORM3.EXE with six identical PC workstations and one NetWareserver. We used the PERFORM3 defaults:

  • 12-secondtest interval

  • 4096 bytes maximum requestsize

  • 1 byte minimum request size

  • requestsize step of 32 bytes

For more information, see the PERFORM3 help screen by running PERFORM3 without any parameters. PERFORM3.EXE canbe found on Novell's Web Site

Figure 1: Ethernet throughput based on a decreasing packet size distribution.


Note: The specific configurations for each of the PERFORM3tests have been omitted. Results will vary slightly withdifferent LAN adapters.

You can see that Ethernet throughput measured with six PERFORM3 clients was near the theoretical bandwidth of Ethernet when the packets were the largest. But with the decreasing packet size throughput dropped until it was nearly one-tenth of its potential. This is not a chance observation. All Ethernet networks, including 100Mbps Ethernet, will show a similar drop in through-put as the packet size is decreased. Moreover, all networks, including Token-Ring and FDDI, suffer from this natural phenomenon. Figures 2 through 4 demonstrate similar results on 16Mbps Token-Ring, Fast Ethernet, and FDDI media.

Figure 2: Token-Ring throughput based on a decreasing packet size distribution.

Figure 3: Fast Ethernet (100baseT) throughput based on a decreasing packet size distribution.

Figure 4: FDDI throughput based on a decreasing packet size distribution.

These charts are familiar to network adapter vendors and benchmark experts because every new adapter is tested with a variety of packet sizes. But these charts aren't widely distributed. One reason is that the results are lengthy and sometimes difficult to interpret. It's easier for the tester to provide a single number as the result. More importantly, vendors use the highest number to demonstrate the high-performance capability of the adapter. Third-party performance labs and industry publications do the same and only publish one peak performance statistic for each adapter.

This one-number-fits-all approach is accurate only if you need to know how the adapter will perform using the maximum packet size. However, this same one-number-fits-all approach is misleading because it doesn't describe how the adapter will perform under other, more realistic conditions--such as your production workload.

The Impact on Production Workloads

Suppose that you standardize on 16Mbps Token-Ring and purchase an adapter the vendor says will operate at or near Token-Ring's 16Mbps theoretical maximum bandwidth. When you design your system on paper, you'll probably put a 16Mbps label next to the line that represents each Token-Ring.

Now suppose that your application mix and protocols produce a lot of small packets so that half of the packets are fairly large and the other half are very small. 16Mbps is still the theoretical maximum bandwidth of each ring and the vendor's tests have shown the adapter to perform near that level. However, your production workload's packet size distribution is less than ideal. Your workload's packet size distribution has reduced your network's efficiency, making its 16Mbps maximum rating unattainable.

As with the benchmark results above, this same scenario can be found in all production networks regardless of type or size. The obvious remedy for inefficient packet size distributions is to standardize on the maximum frame size. The problem is that you have little control over the packet sizes used by your production workload. Packet sizes are determined for you by the architecture of your applications, by network protocols, by management protocols, by intermediate network components, and by other components of the network.

But simply knowing the problem exists can save you hours of frustration. If you understand the characteristics of your production workload, particularly its packet size distribution, you'll understand how to trace bottlenecks that would otherwise be hidden from your view.

The most important step you can take is to characterize the packet size distribution of your production networks, using a protocol analyzer or the Novell Research PSD.EXE utility introduced in this AppNote. If most of the packets on a particular segment are large (with sizes greater than half the media's maximum frame size), you can place that segment in the "efficient" category. If, on the other hand, most of the packets are small (less than half the media's maximum frame size), your network's efficiency is in question. Further steps of investigation might include the following:

  • Identify the offending protocols. Analyze every type of traffic on your network, looking for traffic that relies heavily on small packets. Check into NetWare file service traffic, network management traffic, users' NET.CFG files that disable PacketBurst, and so on.

  • Identify the offending applications. Look for applications that standardize on small data requests, or an application built in-house that you can rearchitect to make better use of thenetwork's maximum frame size.

  • Identify the offending users. See if there is a particular user or workgroup that generates an inefficient packet size distribution.

Using PSD.EXE to Record Network Packet Size Distributions

PSD.EXE is a real-mode DOS utility that records the distribution of packet sizes on a LAN segment. Like a protocol analyzer, PSD uses a LAN adapter's promiscuous mode to process all packets on the attached network segment. While running, PSD displays a packet counter on the screen. When PSD exits it produces an output file called PSD.DAT that lists the recorded packet size distribution.

PSD is specifically written to provide a more detailed packet size distribution than commercial protocol analysis tools. Output from PSD can be used to isolate workload-related bottlenecks.

PSD is a freeware utility from Novell Research, available in the NOVUSER forum of NetWire. You do not need permission to copy or transfer the executable. PSD is offered on an as-is basis and is not supported by Novell.

System Requirements

To run PSD.EXE, you will need the following:

  • A personal computer with an 80386 or higher processor.

  • A LAN adapter that supports promiscuous mode.

  • A LAN driver written to ODI specification version 4 or greater.

  • LSL.COM version 2.11 or greater.

The following network software is optional:

  • IPX

  • NetWare Shell (NETX) or NetWare DOS Requester (VLM)

PSD does not require the presence of IPXODI.COM, NETX.EXE, or VLM.EXE. However, these components may be loaded without any effect to PSD.

Specifications

PSD is written in Microsoft Assembly Language (MASM) using the 80386 and floating-point instruction sets. It is implemented as a version 4.0 NetWare Open Data-Link Interface prescan protocol stack. In the ODI model (see Figure 6), prescan protocol stacks receive all incoming packets from a particular LAN adapter before the packet is routed to the appropriate protocol stack. Prescan protocol stacks have the option of either consuming the packet or allowing other bound protocols to process the packet. PSD uses this design to examine each packet on the wire before passing control of the packet back to the Link Support Layer (LSL) for proper distribution. PSD does not consume any packets.

Figure 5: The ODI Model with prescan stack functionality.

Use of Promiscuous Mode

Normally, a LAN adapter only receives packets that are addressed to it. However, some adapters support "promiscuous mode" operation. When promiscuous mode is enabled on a LAN adapter, the adapter receives every packet on the wire. PSD requires that the LAN adapter support this mode of operation so it can examine every packet on the wire.

To determine which LAN adapters support promiscuous mode, call Novell Labs' Fax Back System at 801-429-2776 (domestic and international). You can request a master index or you can request document 10047, "LAN Adapters which Support Promiscuous, Multicast, and Error Capturing Modes." This information is also available through the following forums:

  • Network Support Encyclopedia (NSEpro on CD-ROM)

  • NetWire online service via CompuServe

  • World Wide Web (URL to http://www.novell.com select PROGRAMS, select Novell Labs, select Bulletins, select LANAdapters and Drivers, and finally select Doc. 10047, LANAdapters which Support Promiscuous Mode.)

If your LAN adapter doesn't support promiscuous mode, PSD will abort with the error message "PS_ERROR5: No Promiscuous MLIDs are loaded."

Installation and Operation of PSD

PSD.EXE is a single executable with no supporting modules. Installation involves a simple copy of the .EXE file to the /NWCLIENT directory. The steps involved are:

  1. Copy PSD.EXE to the \NWCLIENT directory.

  2. Set up the NET.CFG file.

  3. Load LSL.COM.

  4. Load the required LAN driver.

  5. Load PSD.EXE.

Once you've set up your NET.CFG file, you can finish the load process with a batch file that includes the executables named in steps 3 through 5. An example is shown below:

Sample batch file to load PSD.EXE

c:\nwclient\LSL

c:\nwclient\3C589 // Substitute your LAN driver .EXE here

c:\nwclient\PSD

Each step is described in more detail below.

Step 1. Copy PSD.EXE to the \NWCLIENT directory on the workstation.

Step 2. Set up your NET.CFG file with the frame types that you want PSD to report distributions for.

Your NET.CFG file must be configured with all of the frame types you want to examine with PSD.EXE because PSD's results are limited to the frame types you list here. The following listing is an example NET.CFG file configured for all Ethernet frames:

Sample NET.CFG for 3Com 3C589 Ethernet adapter

Link Support

    MemPool 6192            // Optional

    Buffers 10 1580         // Optional

        Max Stacks 8        // Optional



Link Driver 3C589           // Substitute your driver

        Int 5               // Place your driver's config here

        Mem C8000

        Port 200

        Frame Ethernet_802.2    // Logical board 1

        Frame Ethernet_II       // Logical board 2

        Frame Ethernet_SNAP     // Logical board 3

        Frame Ethernet_802.3    // Logical board 4

Token-Ring clients should have NET.CFG files configured to load the following two frame types under the Link Driver heading:

Frame Token-Ring_802.2     // Logical board 1

Frame Token-Ring_SNAP        // Logical board 2

Step 3. Load LSL.COM.

As the LSL loads, it displays a "preamble" with the module name, version, date, and copyright notice. It then displays the Link Support configuration parameters from the NET.CFG file, followed by the resulting LSL configuration. The following is an example of the information displayed during the loading of LSL.COM:

Preamble

NetWare Link Support Layer v2.14 (941011)

 Copyright 1990-1994 Novell, Inc. All Rights Reserved.

NET.CFG parameters

MEMPOOL 6192

BUFFERS 10 1580

MAX STACKS 8

Resulting LSL config

The configuration file used was "C:\NWCLIENT\NET.CFG".

Max Boards 4, Max Stacks 8

Buffers 10, Buffer size 1580 bytes, Memory pool 6192 bytes.

As you load the LSL, notice the version number on the first line of the LSL preamble. PSD.EXE requires an LSL version number greater than or equal to v2.11.

Step 4. Load the necessary LAN driver for your LAN adapter.

PSD.EXE will register as a prescan protocol stack on all boards and frame types that support promiscuous mode. Therefore, you should only load drivers for LAN adapters (and their attached LANs) that you want to monitor. You can see the frame types the LAN driver is supporting during the loading of the LAN driver. The following example shows the information displayed for a 3Com 3C589 LAN driver with a NET.CFG configuration that includes all four Ethernet frame types:

3Com EtherLink III Parallel Tasking PC Card MLID w/ DME  v2.11 (950907)

 Copyright 1995 3Com Corporation.  All Rights Reserved.



     IRQ 5, Port 200, Mem C8000, Max Frame 1514 bytes, Line Speed 10 Mbps, Bus ID 3

     Board 1, Frame ETHERNET_802.2, LSB Mode

     Board 2, Frame ETHERNET_II, LSB Mode

     Board 3, Frame ETHERNET_SNAP, LSB Mode

     Board 4, Frame ETHERNET_802.3, LSB Mode

Step 5. Load PSD.COM.

If no errors are encountered, PSD begins running immediately and displays the number of packets seen.

Note: If an error occurs, see the error explanations in "Error Messages."

An example of the PSD.EXE display is shown below:

Novell Research Packet Size Distribution Utility    v1.0

Loaded successfully, press any key to unload. Results are in PSD.DAT.



123456 packets counted

PSD will count up to 65,535 packets before exiting. However, you need only run PSD for several minutes to get a good idea of your packet size distribution. We recommend that you monitor a segment during production hours and allow PSD to count 10,000 to 20,000 packets before you exit the program.

Exiting PSD. PSD will exit gracefully if one of two events occur:

  • You press any key.

  • The packet counter reaches 65,535 packets.

PSD.DAT File Output

The granularity of PSD's statistics is different than those produced by commercial protocol analyzers. PSD uses distribution bins in increments of 128 bytes to describe the packet size distribution. So in an Ethernet environment where the maximum frame size is 1518, you get twelve bins reported in your distribution (see Figure 6).

Figure 6: Sample PSD.DAT output for an Ethernet trace.

35% < 1536 Bytes

0% < 1408 Bytes

0% < 1280 Bytes

5% < 1152 Bytes

0% < 1024 Bytes

0% < 896 Bytes

0% < 768 Bytes

0% < 640 Bytes

7% < 512 Bytes

3% < 384 Bytes

3% < 256 Bytes

47% < 128 Bytes

PSD begins reporting at the highest packet size encountered. If you only have packets that are less than 128 bytes in size, for example, PSD will only display one line of output as shown below:

100% < 128 Bytes

Error Messages

The table in Figure 7 lists and describes the error messages you may encounter when loading PSD.EXE.

Figure 7: PSD.EXE error messages and explanations.


Error Message
Explanation

PS_ERROR1: The LSL's version is not 2.11 or greater.

PSD.EXE requires the LSL to be v2.11 or greater. When you load LSL.COM you can check the version number in the first line of the preamble, as described under Step 3 in the "Installation and Operation of PSD" section of this AppNote.

PS_ERROR2: The LSL isn't loaded.

PSD.EXE requires the presence of LSL.COM. The Link Support Layer (LSL) must be loaded prior to PSD.EXE.

PS_ERROR3: This system is not a 386-class machine.

PSD.EXE requires an 80386 processor or greater to support the 386 instruction set. The current system will not supportPSD.EXE.

PS_ERROR4: No MLIDs are loaded.

A Multiple-Link Interface Driver (MLID) LAN driver must be loaded prior to PSD.EXE. Contact your LAN controllermanufacturer for the appropriate LAN driver for your card.

PS_ERROR5: No Promiscuous MLIDs are loaded.

PSD requires your LAN controller and driver to support promiscuous mode. (For an explanation of promiscuous mode,see "Use of Promiscuous Mode").

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates