Novell is now a part of Micro Focus

Enhancements to Novell Directory Services in NetWare 5

Articles and Tips: article

JEFFREY F. HUGHES
Senior Consultant
Novell Consulting Services

BLAIR W. THOMAS
Senior Consultant
Novell Consulting Services

01 Oct 1998


Find out about all the new features and enhancements Novell has made to NDS in NetWare 5, including transitive synchronization, new utilities, and more.

Introduction

Along with all the other exciting features of NetWare 5 comes increased functionality for Novell Directory Services (NDS). Several enhancements have been incorporated into NDS to make it more efficient and scalable, to reduce traffic across your WAN infrastructure, and to enable you to more easily manage both the NDS tree contents and your overall network system.

This AppNote briefly introduces these enhancements to NDS, organizing them into five main areas:

  • Improvements to the NDS replica synchronization process

  • Enhancements to reduce NDS-related network traffic

  • New features for easier access to information stored in the NDS tree

  • New features for easier management of objects within the NDS tree

  • New features for easier management of the overall NDS system

This information is part of the NetWare 5 series of articles begun in the August 1998 issue (see "Quoi de Neuf: What's New in NetWare 5?") and continuing in the September 1998 special issue. Refer to these articles for background information on NetWare 5. You should also visit the NetWare 5 section of Novell's Web site at

http://www.novell.com/products/netware5/

New Features Added to the Replica Synchronization Process

The following changes to NDS in NetWare 5 enhance the replica synchronization process, helping to make NDS more scalable across your network:

  • Transitive synchronization

  • Randomized replica rings

  • Caching of replica changes

  • Multiple objects per synchronization packet

  • Distributed reference links

Replica Synchronization Overview

NDS is a loosely consistent database, which means that changes or updates can occur at various replicas, but it takes time for changes made at one replica to propagate to other replicas of the partition. Thus while not all replicas may have the exact information at the same time, they are continually converging to a synchronized state. The process by which changes are automatically sent to the other replicas of that partition is a background process called replica synchronization. The purpose of replica synchronization is to guarantee consistency of the information across all the replicas of a partition over time.

Updates or modifications that affect the data in NDS include adding, deleting, moving or renaming an object, as well as changing the properties or attributes of an object. The replica synchronization process passes to other replicas only the data that has changed for each object or property. By passing only the information that is updated, the total amount of information sent is reduced and network traffic for synchronization is kept to a minimum.

In NetWare 4, the replica synchronization process involves updating all the replicas for a specific partition with all the changes made to the partition since the last synchronization cycle. The process takes the entries in the replica pointer table (also called the replica ring) and synchronizes each of the replicas one at a time with the most recent changes. Each server in the replica ring must be contacted one at a time to complete a synchronization cycle for the partition. This method of forcing the synchronization process to contact every server consumed valuable time and network resources.

In NetWare 5, significant changes have been made to the replica synchronization process to make it even more streamlined and scalable. These changes are described in the sections below.

Transitive Synchronization

To make replica synchronization more efficient and manageable, NetWare 5 features a process known as transitive synchronization. This new process uses the transitive vector to synchronize between two NetWare 5 servers instead of the Synchronized Up To property that was used in NetWare 4.

The transitive vector is a new structure that has been added to the Partition Root object for each NDS partition in NetWare 5. Unlike the Synchronized Up To property which holds only one value for each replica in a partition, the transitive vector is essentially a group of modification time stamps that represent the values held by each replica in a given replica ring. There is one transitive vector for each replica. Another major difference between the transitive vector and the Synchronized Up To property is that the transitive vector is synchronized between servers, whereas the Synchronization Up To property is not.

The transitive vector structure is kept as a vector with an attribute for each replica. Using the transitive vector, the synchronization process can achieve convergence among the replicas of a partition without having all replicas talk to every other replica. The structure of the transitive vector is described as follows:


Server ID

ID of the server that holds the replica

Time Stamp Vectors

List of time stamp vectors for each replica in replica ring

Whenever a change occurs in an NDS object or property, the replica synchronization is scheduled to run. Each time the replica synchronization process is scheduled to run on a NetWare 5 server, transitive synchronization reads the transitive vector to determine which other servers hold replicas that it needs to synchronize with. The synchronization process determines which servers need to communicate by comparing the time stamps of the source and target servers in the transitive vector. The source server is the local server that received the update, while the target server is the destination server of the replica synchronization. If the time is greater for the local server, the replica updates are sent. The local or source server does not request the target server's time stamps, as they are already local in the transitive vector. After all updates are sent, the source server sends its transitive vector values, which are merged into those of the target server.

You can easily view and track the transitive synchronization status using NDS manager (see Figure 1).

Figure 1: Example of the transitive synchronization table in the NDS Manager utility.

The advantages of transitive synchronization are many. First, since each NetWare 5 server has a transitive vector, the process can immediately determine how up-to-date the other replicas are. Better yet, a specific server does not have to contact every other server in the partition to complete the replica synchronization process. After a successful synchronization, the target or destination server merges the transitive vector with that of the source server. In addition, the changes are guaranteed to converge across all replicas of a partition over time.

You may be wondering how the transitive vector works if the synchronization occurs on a replica ring with a mix of NetWare 5 servers and NetWare 4 servers. If a replica ring includes a mixed server environment, it will synchronize using the NetWare 4 method of passing the Synchronized Up To vectors. After the synchronization has completed, either the transitive vector from the NetWare 5 server or the Synchronized Up To vector from the NetWare 4 server is merged.

Randomized Replica Rings

In NetWare 4, when the replica synchronization process starts, it contacts and updates each of the replicas in the replica ring in sequential order: the first replica in the replica ring is updated first, followed by the second one, and so on. Since the replica ring is identical on all the servers holding copies of a partition, the synchronization process contacts all the replicas for each partition in the same order. This may have an undesirable effect if multiple servers are trying to synchronize with the same replica on the same server simultaneously. Since only one inbound synchronization process is supported per partition (not per server), one of the synchronization processes will have to back off and try again at a later time. This situation arises only if the synchronization process tends to execute at approximately the same time on multiple servers.

To help alleviate this situation, NetWare 5 changes the sequence in which replicas are processed by randomizing the order of the replica ring list before attempting to contact the first server. This greatly reduces the possibility that one of the servers will have to back off and attempt to resynchronize with a replica. By randomizing which replica receives the first update, transitive synchronization will more quickly converge the data among all the replicas for a partition.

Caching of NDS Object Changes

Another way in which NetWare 5 enhances the performance of the replica synchronization process is by using a new feature that caches the changes to the NDS objects. Remember, the changes or updates that affect the NDS objects include adding, deleting, moving or renaming an object, as well as changing the properties or attributes of an object.

In NetWare 4, NDS sequentially searches the entire object file comparing time stamps to find the objects that have been updated since the last successful synchronization. This sequential searching effectively limits the size of NDS partitions, resulting in a practical recommendation to have no more than 1000-1500 objects per partition when implementing an NDS tree in NetWare 4.

To overcome this limitation, NetWare 5 eliminates the sequential search time by caching all changes to objects. The ability to cache the objects in the NDS database greatly increases the speed of searching for and updating each replica and partition. As there is no practical experience to base any numbers on, we do not yet have any new numbers for recommended partition sizes.

The object cache maintains a set of object IDs that have been modified since the last synchronization. The cache is populated by placing object IDs in the cache that have been modified as a result of client requests as well as by inbound synchronization. Each cache entry contains the modification time relative to the last synchronization. The cache that NDS maintains is for each partition that is stored on a server. The cache has a start time associated with it that identifies when the cache was built. Changes only need to be sent out if the transitive vector for the remote server is less than or equal to the transitive vector for the local server. The Janitor process cleans up the cache periodically by removing the object IDs that have already been sent to all the replicas.

Multiple Objects per Synchronization Packet

To help synchronize individual replicas faster, NetWare 5 supports more than one object in each communication packet. NetWare 4 only allows changes for one object in a single communication packet. By allowing multiple changes to different objects in a single packet, NetWare 5 reduces the total number of packets needed to synchronize with other replicas. If the number of changes between two replicas is small, it is feasible that all the updates between to two replicas can be accomplished in a single packet. Basically, this means that a partition can be synchronized in a minimum of one packet.

Distributed Reference Links

NetWare 5 features a new method for maintaining external references, called distributed reference links (DRL). These links are more efficient than the backlink method used exclusively in NetWare 4 when renaming, moving, and deleting external references.

When a backlink is created, it stores the NetWare server name and object ID for each external reference object. Thus backlinks are tied to specific NetWare servers, and as a result, each NetWare server tends to have a connection to every other server in the NDS tree.

DRLs, on the other hand, reference the partition that stores the external reference object. DRLs are implemented using a new "Used By" property of the object. This property contains the distinguished name of the partition that holds the external reference and the local ID that is assigned to the object. By storing the partition name instead of the server name, individual NetWare servers can resolve the DRLs by using the NDS tree structure.

Like backlinks, DRLs have a periodic process that checks to verify whether the associated external reference still exists. This process provides data integrity for the NDS database.

Although DRLs are fully implemented in NetWare 5, backlinks will not go away. During certain situations, backlinks will continue to be used to help maintain the external references. NetWare 5 supports both methods.

Enhancements to Reduce Network Traffic

NetWare 5's NDS reduces network traffic by:

  • Restricting outbound synchronization from all Read Only replicas

  • Eliminating subordinate references from the replica synchronization process

NetWare 5 also includes a new utility called the WAN Traffic Manager.

Restricting Synchronization from Read Only Replicas

NetWare 5 no longer assigns a replica number to Read Only replicas and does not create an entry in the transitive vector. This means that Read Only replicas will never start a replica synchronization process. This change makes the replication of the NDS tree more scalable, as the number of Read Only replicas can be increased without increasing the number of entries in the transitive vector.

Since Read Only replicas cannot perform outbound synchronization, the source server merges the transitive vector for the Read Only replicas. Since the sender has to merge the transitive vector for Read Only replicas, for the sake of simplicity and for optimization senders always merge transitive vectors after a successful outbound synchronization for both Read Only and writable replicas.

Eliminating Synchronization with Subordinate Reference Replicas

As mentioned previously, the transitive vector is stored in the Partition Root Object of each partition. There is one entry for each replica in the ring. A major difference between the transitive vector and Synchronized Up To vector is that the subordinate references are not included in the transitive vector structure. This means that the subordinate references will not participate with the normal replica synchronization. This eliminates all traffic to and from subordinate reference replicas.

Current NDS design guidelines account for the subordinate reference replica being included in the replica synchronization process. The current guideline is that each parent partition should have no more than 10-15 subordinate partitions. In NetWare 4, this recommendation greatly reduces the number of total replicas each server has to keep track of.

Now that subordinate reference replicas are no longer involved in the synchronization process, these guidelines may change to increase the scalability of NDS across your network. However, we have not yet completed enough testing to determine any new guidelines. We therefore recommend that, for now, you continue to follow the recommendation of 10-15 subordinate partitions.

Although subordinate reference replicas are no longer included in the NetWare 5 synchronization process, they are included in NetWare 4. Thus, if you have a mixed NetWare 4 and NetWare 5 environment, remember that subordinate references will be involved in the synchronization process.

WAN Traffic Manager

WAN Traffic Manager allows you to manage NDS traffic across WAN links, reducing network costs and increasing performance benefits. Networks that have significant cost or congestion on their WAN links due to NDS (and in the future, other products) will want to use WAN Traffic Manager to manage the cost and congestion.

WAN Traffic Manager controls server-to-server traffic generated by NDS. It can restrict traffic based on cost of traffic, time of day, type of traffic, or any combination of these. Although WAN Traffic Manager was designed to control traffic across WAN links, it can control NDS traffic between any servers in an NDS tree. WAN Traffic Manager controls only periodic events initiated by NDS, such as replica synchronization, the Janitor process, and the Limber process. It does not control events initiated by administrators or users, nor does it control non-NDS server-to-server traffic such as time synchronization

WAN Traffic Manager consists of three elements:

  • WTM.NLM. This NetWare Loadable Module resides on each server in the tree. Before NDS sends server-to-server traffic, WTM will read a WAN Traffic Policy to determine whether or not that traffic will be sent.

  • WAN Traffic Policies. These are rules that control the generation of NDS traffic. WAN Traffic Policies are text stored as an NDS property value on the NetWare Server object, the LAN Area object, or both.

  • NWAdmin Snap-in. This snap-in is the interface to WAN Traffic Manager. It allows you to create or modify policies, to create LAN Area Objects, and to apply policies to LAN Areas or to servers.

When WAN Traffic Manager is installed, the NDS schema is extended to include a LAN Area Object and three new detail pages for the NetWare Server object: the LAN Area Membership page, the WAN Policies page, and the Cost page.

WAN Traffic Manager selects and runs administrator supplied policies to manage WAN costs and congestion. NDS, WAN Traffic Manager's first client, uses it to control synchronization and other background tasks based on time of day, network addresses, network costs, and so on. WAN Traffic Manager is extensible so that clients can supply extra information to the policies. WAN Policies are stored in NDS to enable global administration. You can use constraints such as time of day, connection cost, source and destination addresses, other connection and address specifics, and information supplied by the WTM client.

Note: WAN Traffic Manager is not to be confused with the older Novell product known as PingFilt/DSFilter. PingFilt used a dirty hook into NDS and inspected every packet that NDS attempted to send. On ping packets, it recorded the values returned and used them to spoof ping packets when filtering was active. Otherwise it either allowed packets to proceed or returned the error -625 ERR_TRANSPORT_FAILURE to NDS.

WAN Traffic Manager provides a Policy Service by utilizing WAN Policy and Cost attributes that are stored in NDS. When it receives a request from a client application (via the GetWanPolicy API), it selects the most appropriate policy and evaluates it. It then returns the results (SEND_NOW or DONT_SEND) to the client application.

The client application must supply WAN Traffic Manager with a traffic type, optional source and destination network addresses, and optional application-specific information in the form of named variable and value pairs. The executed policy can modify these variables, and they are returned to the client application as part of the results. This enables a policy to tell the application more than just SEND_NOW or DONT_SEND. For example, the policy might indicate to send the data at a later time or when certain conditions (such as queue sizes) are met.

WAN Traffic Manager provides its services in two ways: directly and indirectly via the RollCall NLM. In the direct approach, the client application calls WAN Traffic Manager's exported APIs. This introduces a load order dependency--WTM must be loaded before the client application is loaded. Because this is most often undesirable, WAN Traffic Manager also makes its APIs available via RollCall. Both WAN Traffic Manager and the client application will have a load order dependency on RollCall, but not on each other. They can be loaded an unloaded independently. Novell anticipates that most applications will use the indirect approach.

Easier Access to NDS Information

NetWare 5's NDS also enables you to more easily access the information that is stored in the tree. Using the new Catalog Services, users now have different ways to view the NDS tree structure.

Catalog Services

Catalog Services provides a method to store NDS data in a non-partitioned format, indexed for rapid access. Catalog Services components include a dredger which reads the Directory to build catalogs, a NetWare Administrator (NWAdmin) snap-in which controls the scope, filtering, time interval and other aspects of catalog creation, and a set of query APIs which support development of applications which can find catalogs and read data from them.

Among other things, this enables the development of applications which need rapid access to Directory data in a centralized repository. Such applications might include Contextless Login, NDS White Pages, fast lookup for LDAP gateways, and many others. Catalogs are stored as high-speed databases within stream attributes in the NDS directory. The scope of catalog databases can be modified to include data from the entire NDS tree or from one or more subtrees. The attributes stored in the catalog can also be filtered according to user requirements.

The Dredger. The dredger is a server-based program that creates catalogs by periodically scanning the NDS tree for the data that was previously requested by the administrator of the catalog. The dredger runs on the following platforms:

  • NetWare 4.10 (DS.NLM version 5.01 and above)

  • NetWare 4.11 (DS.NLM version 5.73 and above)

  • NetWare 5 (all NDS versions)

You can completely control the scheduling of the dredger operations. You select how and when the dredger refreshes the catalogs on a periodic interval, or start a refresh manually at any time. Although the catalog is being updated during the refresh operation, it remains available to users. In addition, the dredging for multiple catalogs can occur simultaneously.

The access rights used by the dredger while creating a catalog are the rights assigned to the Catalog object in NDS. After you define the Catalog object in NDS, it is extremely important that you grant it the proper access rights. The access rights are controlled through Access Control Lists (ACLs), in the same manner that all other NDS access rights are controlled.

In addition to controlling the rights, you can control the scope of the Directory data included in a catalog by specifying a base object where dredging will begin and a tree depth of single object, subordinate objects, or full subtree. Multiple base objects can be specified, each with a corresponding tree depth. You can also create filters to control the object classes, attribute classes and values to be included in the catalog.

Management Utility. Included with Catalog Services is a management utility that runs as a snap-in to the NWAdmin console on Windows NT and Windows 95/98 workstations. In addition to providing a simple browser for viewing catalog contents, this management utility gives you the ability to create and edit filter expressions that determine what data is included in the catalog based on object classes, attribute classes and attribute values (see Figure 2).

Figure 2: The Catalog Services management utility that runs as a snap-in to NWAdmin.

The management utility provides a dredger server page, which allows the administrator to control certain aspects of the operation of the server on which the dredger is running. These include the number of threads allocated for dredger operation and the sync interval length at which the dredger checks its catalog list against the list on the server object in NDS. The page also displays a list of the catalogs dredged by the server. The management utility provides a summary page that lists general information about the dredger, including the time of the last dredge and log file statistics.

Installing Catalog Services. Catalog Services is easily installed using the install wizard program provided. This program runs on Windows 95 and Windows NT workstations. Since it does not run on a Windows 3.1 workstation, you must copy the Win16 SDK components to a Windows 3.1 workstation manually if application development will be performed in the Windows 3.1 environment.

When Catalog Services is installed, the program extends the NDS schema and places the dredger and management utilities on the appropriate servers. The install wizard provides the capability to remove SDK components and registry entries installed by Catalog Services on the local machine. It does not remove components that are installed on a file server. Components not removed include NDS schema extensions, the dredger, and the management utility.

Easier Management of NDS Objects

There are several new NDS features in NetWare 5 that allow you to more easily manage the objects and properties in the NDS tree. These include:

  • Inheritable ACLs

  • Password Management Property

Inheritable ACLs

The inheritable ACLs allow you to assign supervisor rights to operators or help desk staff to manage specific types of objects or resources without granting supervisor rights over the entire tree. For example, you can assign rights to a group to manage only the users' passwords, telephone numbers, and addresses. This feat is possible thanks to the new inheritable ACLs that provide the ability to inherit specific property rights down the entire NDS tree.

To give inheritable rights to a specific property and have them flow down the NDS tree, you use the NWAdmin utility. On the property rights screen, you'll see a new "Inherit" button at the bottom of the rights list. You can give specific inheritable property rights to any object by selecting the rights and then clicking the Inherit button.

Note: An enhancement to the NWAdmin utility in NetWare 5 provides the ability to see all the schema-defined properties anywhere in the NDS tree in a single list. The properties that are shown are for all the objects in the schema. This gives you the ability to select any property at any level in the NDS tree. For example, you can give a group the rights to manage all users' telephone numbers at the Organization (O) level.

Password Management Property

In addition to the NDS Inheritable ACLs feature, NetWare 5 has added a new user property called Password Management. This new property gives you the ability to assign rights to manage just the users' passwords. For example, using the new Password Management property, you can grant a specific group, PASSWORD-GROUP, the Supervisor rights to reset just the users' passwords and accounts. To accomplish the same thing using NetWare 4, you had to give all rights or supervisor rights to the individual user objects or to the entire container or tree.

Using the Password Management property in combination with the new Inherit button, you can grant rights that will flow down the entire tree. Figure 3 illustrates the example of granting the group, PASSWORD-GROUP, the rights to manage all users' passwords and accounts for the entire tree.

Figure 3: Using the new Password Management property in NWADMIN, you can grant a specific group the rights to manage the users' passwords and accounts. By clicking the Inherit button, these rights will flow down the entire NDS tree.

Easier Management of Overall NDS System

The new NDS features that enable you to more easily manage the overall NDS system include:

  • The DSDIAG utility

  • The Schema Manager utility

The DSDIAG Utility

The DSDIAG utility is a new utility included with NetWare 5 that provides administrators with diagnostic information about NDS, background processes, and replica status. Using DSDIAG you can quickly and easily determine how the partitions and replicas are synchronizing and detect errors, if any, between NetWare servers. DSDIAG also compiles this information in report files for easy viewing.

The DSDIAG utility has two main options: Generate Report Preferences and Generate Report. The Generate Report Preferences option allows you to set default preferences in DSDIAG. You first select Generate Report to see a list of report generation options (see Figure 4). Each of these options generates a separate report file and thus a separate report. Be sure to select each option to define how you want the report to be created.

Figure 4: The DSDIAG utility with the list of report generation options.

"Generate Report" Options. These are the report options you will see when you select Generate Report in DSDIAG:

  • Check NDS Versions - Returns the NDS version for each server in your NDS tree.

  • Check NDS Background Process Status - Lists the status of the NDS background processes for each server found.

  • List Server's Partition Table - Lists the partition table for each server found in the NDS tree.

  • List Replica Rings - Searches for all partitions in the tree and reports each partition's replica ring.

  • Check Partition Status - Checks the synchronization status of each partition root found in the tree.

  • Compare Replica Rings - For each partition root found in the NDS tree, compares the replica ring with other replicas in the ring.

"Generate Report Preferences" Options. The second option allows you to set up default configurations for all reports that you generate, giving you the following options:

  • Manage Naming Conventions - Used to set the format of input and display of NDS names in DSDIAG and the report files.

  • Manage Identities - Used to log in and out of identities and change the default identity.

  • Preferences - Configures the display or report preferences to be used.

Schema Manager

The NDS schema is the internal set of rules of how NDS objects and properties are created and used. Schema Manager is an enhancement to the NDS Manager utility and can be accessed from the Object pull-down menu. Schema Manager allows those with Supervisor rights to the [Root] of a tree to view and customize all aspects of the schema of that tree.

As mentioned, the Schema Manager utility can be accessed from the NDS Manager utility (see Figure 5).

Figure 5: An example screen from the Schema Manager utility.

You can use Schema Manager to do the following:

  • View a list of all classes and attributes in the schema.

  • View information on an attribute such as its syntax and flags.

  • Extend the schema by adding a class or an attribute to the existing schema.

  • Create a new class by naming it and specifying applicable attributes, flags, containers to which it can be added, and parent classes from which it can inherit attributes.

  • Create an attribute by naming it and specifying its syntax and flags.

  • Add an attribute to an existing class.

  • Compare the schemas of two trees and print the results.

  • View or print a report on a selected class or an attribute, or on the entire schema.

  • View or print the extensions to the schema.

  • Delete a class that is not in use or that has become obsolete.

  • Delete an attribute that is not in use or that has become obsolete.

  • Identify and resolve potential problems.

With Schema Manager, you also have the ability to add a completely new object class and/or attributes to your schema. You would want to add a new class or attributes if you currently don't have such a class in your schema and want to represent that particular resource or attribute within NDS. The Schema Manager provides two wizards that walk you through the creation of a new object class and the creation of a new attribute.

The article entitled "Using NDS Manager's Graphical Schema Manager Tool in NetWare 4.11" in the July 1998 issue of Novell AppNotes provides a good general overview of how to use the utility. Most of this material applies to the NetWare 5 version of Schema Manager as well.

Summary

This AppNote has introduced the key enhancements made to NDS with the release of NetWare 5. These new features and enhancements make the replica synchronization process more efficient and scalable, reduce the NDS-related traffic across your network, and enable you to more easily manage the NDS tree and its contents.

For more information about designing, implementing, and maintaining NetWare 5 networks, look for Novell's Guide to NetWare 5 by Jeffrey F. Hughes and Blair W. Thomas, available from Novell Press in late Fall 1998.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates