Understanding SCANTREE.EXE's Statistics

Articles and Tips: article

RON LEE
Senior Research Engineer
Systems Research Department

01 Mar 1996

SCANTREE.EXE is a DOS application designed to traverse a NetWare Directory Services (NDS) tree and return a series of statistics. These statistics describe the width and depth of the tree, as well as the variety and characteristics of leaf objects and attributes. This AppNote describes these statistics and gives suggestions for deriving additional statistics to further describe the characteristics of your tree.

RELATED APPNOTES Jan 96 "Using SCANTREE.EXE to Gather NDS Tree Statistics"

Introduction
Object Class Statistics
Summary Statistics
Calculating Additional Statistics
Conclusion
Where to Find SCANTREE.EXE

Introduction

Other useful SCANTREE options allow you to count only users, dump the entire tree's objects in logical order, or scan specific parts of a tree.

This AppNote describes SCANTREE's summary statistics and gives suggestions for deriving additional statistics to further describe the characteristics of your tree.

Object Class Statistics

After SCANTREE completes its traversal of a tree it prints two series of statistics. The first series is an alphabetical list of the object classes SCANTREE found in the tree. Each class name is followed by the total number of instances of that class in the tree.

If you run SCANTREE on a newly installed NetWare server, the class summary will look like the listing in Figure 1.

Figure 1: SCANTREE statistics for object classes found in a newly installed tree.

NCP Server      1               // A NetWare server object
Organization    1               // An "O=" object
Top             1               // A [ROOT] object
User            1               // A "CN=" object of type user
Volume          1               // A NetWare server volume object

This object class summary will grow as you add new objects to your tree. Mature trees typically have 15 or more classes represented.

The object class summary is also useful in determining the quantity of different types of objects in your tree. For example, you can quickly identify the number and types of objects created by third-party applications.

Another useful item in the list is the number of objects that DS and DSREPAIR have changed to the "unknown" class. These objects fall into one of two categories: (1) an object that is missing one or more mandatory attributes, or (2) a leaf object that contains subordinate objects. Both of these object states are invalid and both DS and DSREPAIR change their object class to unknown when they are encountered. If your tree contains "unknown" objects you should search the tree for those objects using NWADMIN and delete them.

Also useful, is the number of directory maps, organizational roles, profiles and aliases. Usually, a mature tree will include these object classes in direct proportion to the number of containers and the geographic dispersion of the tree. A lack of directory maps can suggest that an organization doesn't share a common file system and a standard application set, or it can suggest that the file system is not organized efficiently. A lack of organizational roles can suggest that all administration is handled centrally with no container admins. In systems that include larger numbers of container admins, the lack of Roles can suggest a lack of proper security administration.

A tree that have container admins and lack specific profiles for those admins suggests that tree designers are unaware of this NDS feature or that they don't require the additional environment variables and other things provided by Profiles. A lack of Aliases can suggest that the administrator isn't aware of the benefits of using Aliases to ease access to printers and other network resources for users.

Summary Statistics

The second series of statistics is a collection of 17 counters that describe the width and depth of the tree, as well as the variety and characteristics of containers, leaf objects, attributes, and errors. Each of these counter names are enclosed in braces to set them apart from the object class summary statistics.

SCANTREE's descriptive statistics include:

{ Attributes - # }
   { Attributes - size }
   { Collision renames }
   { Errors - NWDSList }
   { Errors - NWDSRead }
   { Errors - Subordinate }
   { Group memberships }
   { Largest container }
   { Objects }
   { Partitions }
   { Replica Subreferences }
   { Replicas - Master }
   { Replicas - R/O }
   { Replicas - R/W }
   { Streams - # }
   { Streams - size }
   { Tree Depth }

All of the summary statistics are optional -- only printing if the events or objects they represent are encountered in the tree. The following table lists these summary statistics and describes each one.

Summary Statistic	Description
Attributes - #	The total number of attributes read. An attribute is an individual piece of information that is attached to an object. NDS implements attributes as variable length data structures that contain information and a description of that information including its type, size, and a variety of other optional identifiers.
Attributes - size	The total number of bytes allocated for attributes. This total accounts for the entire structure allocated for each attribute including all information stored in the attribute.
Collision renames	The total number of renamed objects encountered in the tree. A collision rename occurs when a user creates two sibling objects with the samename but with different creation times. This typically occurs when the user iscreating similar objects in multiple partitions. DS finds name collisions duringsynchronization. When this occurs, DS removes the ambiguity of two objectshaving the same name by manufacturing a new name for one of the objects. Thenew object name is in the "<CollisionCount<<Replica#<" format. For example, the object name 123_5 would be considered by SCANTREE to be anobject that was previously involved in a collision rename process. SCANTREE identifies collision renames by looking for object names with integervalues on both sides of an underscore. If you purposefully create object names ofthis type, SCANTREE will mis-count them as collision renames and this summarystatistic will be inaccurate.
Errors - NWDSList	The total number of errors returned by the API call "NWDSList()."For a complete list of possible errors see Directory Services OS Errors (-001 to-255), Directory Services Client Library Errors (-301 to -399), or Directory ServicesAgent in the Server Errors (-601 to -699) located in the NetWare 4 Client SDK.
Errors - NWDSRead	The total number of errors returned by the API call "NWDSRead()." The most common NWDSRead() errors received by SCANTREE include: -150 DSERR_NO_ALLOC_SPACE. DS attempted to write to a file server that did not have enough free memory to process the request. Example: NWDSRead(OU=SED.OU=SERVICES.O=Novell) error -150 * -333 ERR_NO_CONNECTION. This is an internal DS error and requires you to call Novell Technical Support for assistance. Example: NWDSRead(CN=DUS-WAN-MPR1.OU=MPR.OU=IS.O=Novell) error -333 * -601 ERR_NO_SUCH_ENTRY. In this case, SCANTREE encountered an error when referencing the specified object. Example: * NWDSRead(CN=Hplaser3p S216-p1+Bindery Type=7.OU=gaa) error -601 *
Errors - NWDSRead	-626 ERR_ALL_REFERRALS_FAILED. The server had no objects matching the request and attempted to contact other servers to find the object. None of those servers responded. Example: * NWDSRead(CN=DUS-CD.OU=MANAGED TREE SERVERS.OU=PRV.OU=IS.O=Novell) error -626 * -649 ERR_INSUFFICIENT_BUFFER. If you receive this error, the object is too large for SCANTREE's buffer. This is a problem with SCANTREE, not your tree. Example: * NWDSRead(CN=Chey Vsvr Mbdnw410.O=MBA) error -649 * For a complete list of possible errors see Directory Services OS Errors (-001 to-255), Directory Services Client Library Errors (-301 to -399), or Directory ServicesAgent in the Server Errors (-601 to -699) located in the NetWare 4 Client SDK.
Errors - Subordinate	The number of times SCANTREE found fewer subordinates in a container than the container reported. This can be caused by NWDSList() errors. Subordinate errors will also occur when a portion of the tree is hidden from Admin by an exclusive container administrator.
Group memberships	The total number of group memberships held by all objects in the tree.
Largest container	The total number of objects in the largest container (the container with more subordinates than all other containers).
Objects	The total number of objects from all object classes found in the tree.
Partitions	The total number of partitions in the tree.
Replica Subreferences	The total number of subordinate references in the tree.
Replicas - Master	This is the total number of master replicas. The master is the first instance of a partition.
Replicas - R/O	This is the total number of read-only (R/O) replicas in the tree.
Replicas - R/W	This is the total number of read-write (R/W) replicas in the tree.
Streams - #	This is the total number of stream attributes in the tree. A stream attribute is an attribute that is large enough to justify separate storage in its own file, such as login scripts and graphic images. The use of stream files is at the developer's discretion. Stream files are allocated by DS and located in the same hidden directory with the main DS files on volume SYS:.
Streams - size	This is the total number of bytes allocated by DS for stream files.
Tree Depth	This is the total number of levels in the tree. SCANTREE counts each level beginning at the root of the tree. The root is the first level. All subordinates of the root are at the second level, and so on.

Example SCANTREE Summary Statistics

The SCANTREE results in Figure 1 are from a newly installed NetWare 4.1 server. The summary statistics are the last lines printed by SCANTREE. Notice that only 8 of the possible 17 summary statistics were reported by SCANTREE. This is because the summary statistics aren't displayed unless their counters are greater than zero. With the exception being "{ Stream - size }" which often has a mind of its own.

Each of the five object class reported in Figure 2 were automatically created by the NetWare 4.1 install program. All of the summary statistics reflect the newly-installed state of the tree.

Figure 2: SCANTREE summary statistics from a new server installation.

Scantree v1.07a
Scanning [Root], Wed Jan 17 14:32:41 1996
========= SUMMARY STATISTICS =========
Test duration was 00:00:01
NCP Server                      1
Organization                    1
Top                             1
User                            1
Volume                          1
{ Attributes - # }              40
{ Attributes - size }           2956
{ Largest container }           3
{ Objects }                     5
{ Partitions }                  1
{ Replicas - Master }           1
{ Streams - size }              0
{ Tree depth }                  2

Figure 3 includes a recent set of SCANTREE results from Novell's production tree.

The first thing you'll notice is the block of errors we received on this run, including errors -601 and -649. The -601 errors occurred because version of the SDK I used didn't handle the double-byte characters correctly. The -649 error occurred because SCANTREE didn't allocate a large enough buffer for the object.

Novell's tree contains 39 object classes. The results also include all but one of the summary statistics (NWDSList errors) because no NWDSLIST errors were encountered by SCANTREE.

Figure 3: Scantree summary statistics from Novell's production tree.

Scantree v1.07a
Scanning [Root], Fri Jan 19 17:34:32 1996
*** NWDSRead(CN=Chey Vsvr Prv-security.OU=PRV.OU=SECURITY.O=Novell) error -649 ***
*** NWDSRead(CN=.OU=Core OS.OU=NPD.O=Novell) error -601 ***
*** NWDSRead(CN=1_7.OU=Core OS.OU=NPD.O=Novell) error -601 ***
*** NWDSRead(OU=NYC.OU=AMERICAS.O=Novell) error -649 ***
*** NWDSRead(CN=SLC.OU=SLC.OU=AMERICAS.O=Novell) error -601 ***
*** NWDSRead(CN=NWAR.OU=Applications.OU=SLC.OU=AMERICAS.O=Novell) error -601 ***
========= SUMMARY STATISTICS =========
Test duration was 06:16:37
AFP Server                      1
Alias                           404
Application (DOS)               7
Application (Windows 3.x)       153
Application (Windows 95)        23
Application (Windows NT)        20
Bindery Object                  540
Bindery Queue                   137
Cheyenne Fax Queue              4
Cheyenne Fax Server             2
Computer                        65
Connect Server                  35
Directory Map                   198
External Entity                 1
Group                           2973
Job Queue                       1
Job Server                      1
Message Routing Group           1
Messaging Server                2
NCP Server                      555
NWMOBILE: Mobile File Group     1
Organization                    1
Organizational Role             254
Organizational Unit             1201
PBX Device                      147
PBX Device Group                11
Palindro:JobQueue               19
Print Server                    672
Printer                         601
Profile                         243
Queue                           1012
Top                             1
Tserver                         3
Tserver PBX                     2
Tserver Service                 11
Unknown                         58
User                            11397
Volume                          1495
Worktop                         99
{ Attributes - # }              407101
{ Attributes - size }           90593457
{ Collision renames }           113
{ Errors - NWDSRead }           6
{ Errors - Subordinate }        11
{ Group memberships }           73341
{ Largest container }           969
{ Objects }                     22351
{ Partitions }                  303
{ Replica Subreferences }       521
{ Replicas - Master }           307
{ Replicas - R/O }              8
{ Replicas - R/W }              854
{ Streams - # }                 8899
{ Streams - size }              9703148
{ Tree depth }                  12

Calculating Additional Statistics

Using these statistics, you can derive additional statistics that further describe the characteristics of your tree. The following examples use Novell's SCANTREE statistics shown above.

Average Number of Attributes Per Object

Divide your tree's total attributes value by the total number of objects to calculate the average number of attributes per object in the tree.

{ Attributes - # } / { Objects } = the tree's average
number of attributes per object

In Novell's tree: 407101 / 22351 = 18 attributes per object

Average Attribute Size

Divide your tree's total storage allocation for attributes by "Attributes - #" to calculate the average size of your tree's attributes.

{ Attributes - size } / { Attributes - # } = the
tree's average attribute size

In Novell's tree: 90593457 / 407101 = 223 bytes per attribute

Average Number of Group Memberships per User Object

Divide your tree's total group membership value by the number of user objects to calculate the average number of memberships per user.

{ Group memberships } / User = average number of
group memberships per user object

In Novell's tree: 73341 / 22351 = 3 memberships per user

User Objects as a Percentage of the Overall Tree

Divide your tree's user value by the total objects value to calculate the percentage of the tree made up of user objects.

User / { Objects } = percentage of the tree made up of
user objects

In Novell's tree: 11397 / 22351 = .51 or 51%. About one-half of the tree is user objects.

Average Number of Replicas per Partition

Divide your tree's total replica value by the number of partitions to calculate the average number of replicas per partition.

( {Replicas - Master } + { Replicas - R/O }
   + { Replicas - R/W } ) / { Partitions } = average
replicas per partition

In Novell's tree: (307+8+854) / 303 = 3.8 replicas per partition

Average Stream File Size

Divide your tree's total stream file allocation value by the total number of streams to calculate the average stream size.

{ Streams - size } / { Streams - # } = average
stream file size

In Novell's tree: 9703148 / 8899 = 1090 bytes per stream file

As another example of this calculation, we gathered the following statistics from a small tree that included images (GIF files) of each of the users:

{ Streams - # }            43
{ Streams - size }       431789

In this case, the total stream allocation (size) divided by the total number of streams provided us with the average size of the users' images: 431789 / 43 = 10,042 bytes per stream file.

Conclusion

Novell's Systems Research group performed a study of customer's NDS trees during January and February, 1996. The single purpose for SCANTREE's distribution was to collect its results from as many customers as possible.

We're using the results of this study to:

Design future versions of NDS that are better suited to real-world requirements
Produce an NDS memory and disk space calculator based on real-world object characteristics.
Produce design and optimization tips for real-world NDS configurations.
Help Novell's DeveloperNet members tune new NDS applications to real-world tree characteristics.

Look for coverage of these topics and the study's results in upcoming AppNotes and Developer Notes.

Where to Find SCANTREE.EXE

SCANTREE.EXE can be found in a self-extracting file (SCANDS.EXE) in the following locations:

Via CompuServe in NetWire's NOVUSER forum, search for SCANDS.EXE.
Via ftp://ftp.novell.com/pub/updates/nwos/nw410.
Via www.novell.com | NetWire Technical Support Services | File Updates | Use Our FTP Server | /pub/updates/nwos/nw410.

Documentation for SCANTREE is bundled with the application in SCANDS.EXE. See the README.TXT file.

Documentation can also be found in "Using SCANTREE.EXE to Gather NDS Tree Statistics," Novell Application Notes, January 1996, p.107.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.