Environment
NetWare 4.11
NetWare 4.2
NetWare 5.0
NetWare 4.2
NetWare 5.0
Situation
Various NDS Guidelines
Resolution
This Document serves as an overview of various NDS components
Topics covered in this document are :
A) NDS Health Checks
B) NDS Background Processes
C) Replica Guidelines
D) External References and Backlinks
E) Replica Sync in NW 4 v NW 5
F) Replica Transition States
G) Steps to NDS Troubleshooting
H) Tools used to Investigate and Resolve NDS Issues
I) Performing Replica Operations
J) Dsrepair
K) Server Maintenance Tasks
L) Disaster Recovery Strategies
M) Identify and Resolve Inconsistencies Between Containers
N) Time Synchronization in the NetWare Environment
O) Server-to-Server Communication
P) Identify and Repair Corrupted Objects
Q) Troubleshooting Obituaries
R) Upgrading to NDS8
A) NDS Health Checks
For complete DS health check information and procedures, please see TID 3564075
====================
B) NDS Background Processes
==========================
* Replica synchronization:
* Guarantees consistency of NDS information across all replicas of a given partition
* Only changes are sent
* Some changes are sent immediately, for example, a user路s password change (within 10 sec.); this is known as an immediate sync cycle
* Some less critical changes are collected locally before being sent (within 30 min.); this is known as a slow sync cycle
* The master replica is a participant in the process, but is not the controlling entity
* Synchronization is done periodically and is referred to as the NDS heartbeat or skulking process; this takes place every 30 min. for NetWare 4.x and every 60 min. for NetWare 5 (default values)
* Janitor process:
* When an object is deleted or moved, the janitor process is responsible for cleaning up NDS information
* Looks after a number of tasks:
* Verifies the status attribute of the server路s NDS object
* Executes the flat cleaner process
* Manages the obituary state of moved, deleted or renamed objects
* Performs periodic optimizations of the NDS database to speed up DS lookups
* Verifies the identity of the [ROOT] most replica
* Notifies the operating system of synthetic time
* Flat cleaner:
* Removes obituaries in the purgeable state
* Ensures that each object has a public key
* Purges unused objects and attributes in the external reference and bindery partitions
* Replica purger:
* Processes obituaries
* Deletes unused objects and attributes in user-defined partitions
* Obituary:
* This is a special attribute which is used to track deleted, moved or renamed objects
* When an object is flagged for deletion it moves through the following four stages:
* flag=0 issued
* flag=1 notified
* flag=2 ok_to_purge
* flag=4 purgeable
* The obituary stages are to ensure that all servers holding a replica retain the old information while all other servers are receiving the updates
* After the obituary is flagged purgeable, it is up to the local server路s DS to actually purge the entry
* The background processes involved in processing obituaries are the janitor, flat cleaner, purger and replica synchronization processes
* Limber:
* Verifies the name of the NDS tree
* Verifies the serv.er路s name, internal network address and credentials
* Schema synchronization:
* Makes sure that the same schema structure is present across all servers
* Backlinker:
* Ensures that external references held on a server are required and properly backlinked to the real object
* Removes external references that are no longer required
* Database initialization:
* Initializes NDS and verifies the usability of the database files
* Schedules various background processes to run
* Opens the database files for use by NDS
* Is triggered by mounting the SYS: volume, or unloading and reloading DS.NLM
.
C) Replica Guidelines
===================
* NDS partitioning and replication is done to provide scalability, fault tolerance, and performance
* There are four replica types:
* Master: only one per partition; created when a partition is defined; controlling entity in partition and replica operations
* Read/write: created automatically on the second and third servers in a partition; additional read/write replicas can be created manually on servers that you specify
* Read-only: created on servers that you specify; no real purpose at this time
* Subordinate reference: created automatically on servers that hold a copy of a parent partition but not the child; does not contain object data and, therefore, does not provide fault tolerance!; used to provide tree connectivity
* Novell recommends that each partition have at least two to three replicas (typically, a master and two read/write replicas)
* The replica ring is a list of all servers that have information about a particular partition, the type of replica and the replica state
D) External References and Backlinks
===============================
* Used to help manage the database and provide tree connectivity
* External references refer to objects not physically located on the local server
* External references keep track of non-local objects such as:
* An object that authenticates or attaches to the server
* An object that is added as a file system or local object trustee
* An object that is added as a member of a group
* When NDS creates an external reference, a backlink attribute is added to the referenced object to keep track of the server on which the external reference was created
E) Replica synchronization in NetWare 4 vs NetWare 5
==========================================
* There are some important differences affecting scalability and efficiency:
* Replica sync process
* NetWare 4 Uses sync-up-to vector; each server must contact every other server in the replica ring; the number of server-to-server communications will be n*(n-1), where n=the number of servers in the replica ring
* NetWare 5 Uses transitive vector; replica convergence is quicker and takes fewer server-to-server communications
* Replica ring updates
* NetWare 4 Updates replicas in a sequential order, the master always first
* NetWare 5 Updates in a randomized order, therefore less likely to bottleneck on the inbound sync process
* Read-only replicas
* NetWare 4 Can start the synchronization process
* NetWare 5 Cannot start the sync process
* Subordinate-ref. replicas
* NetWare 4 Participate in normal replica synchronization
* NetWare 5 Not involved in replica synchronization
* Object changes
* NetWare 4 Sequentially searches the entire NDS database to detect changes
* NetWare 5 Caches object changes, therefore does not have to scan the entire database
* Objects per sync packet
* NetWare 4 Only one object per sync packet
* NetWare 5 Multiple objects per packet
F) Replica Transition States
=======================
* DSTRACE is useful to view replica transition states and to identify replica operation processes and errors
* The state of the replica is server-centric, and will be one of the following:
* [0] On: all entries have been copied for the partition
* [1] New Replica: begins the operation of adding to the replica list; sub.ref. replicas are added to appropriate servers
* [2] Dying Replica: when a replica is removed; ext. references are created and backlinked to the real objects
* [3] Locked: certain operations require that only one change on that partition can occur at one time
* [4] Change Replica Type - State 0: designate a new master
* [5] Change Replica Type - State 1: the new master places new timestamps on all replicas and changes from State 0 to State 1; the old master appoints itself as a R/W
* [6] Transition State On: occurs after [1] New Replica; after the replica has communicated with every server in the ring, the master must send the On state to all replicas once they are updated and synchronized
* [48] Split State 0: new partition boundary created
* [49] Split State 1: master makes sure all replicas are updated and synchronized and sends the On state
* [64] Join State 0: when two partitions are combined; servers that do not have a copy of both the child and parent get one; move to [65] Join State 1
* [65] Join State 1: erase the boundary between the 2 partitions; move to [66] Join State 2
* [66] Join State 2: propagate the changes of the combined partition to all servers in the replica list
* [80] Move State 0: partition (sub-tree) move; must not have any child partitions and must have consistent schema rules between source and destination servers
G) Steps to NDS Troubleshooting:
===========================
1. Identify the Scope of the Problem:
* Consider error codes and console messages; identify the server(s) involved, affected partition(s), and symptoms
2. Determine the Cause:
* Which NDS process is experiencing a problem, and why
3. List Possible Solutions:
* Understand the repercussions of each possible solution
4. Assess Possible Solutions:
* Consider the level of difficulty, the effect on users, and the effect on NDS (consider the least intrusive solutions first)
5. Implement Solution:
* Create a backup of NDS first; allow sufficient time for NDS to stabilize and correct itself
6. Verify the Problem is Resolved:
* Perform health check; confirm that NDS operations can be completed
7. Document the Resolution:
* Retain in server maintenance logs for future reference
8. Avoid Repetition of the Problem:
* Establish appropriate policies and procedures (cutting the hands off your co-workers is not considered to be an acceptable solution)
H) Tools Used to Investigate and Resolve NDS Issues:
==========================================
* DSREPAIR (NetWare 4 and 5) Server-centric; 路C-worthy路 interface on server console; checks for and corrects problems in the Directory database on an individual server basis; reports information about Directory Services, time synchronization, and server status
* DSDIAG (NetWare 4 and 5) Menu-based utility that allows you to generate diagnostic reports on NDS; generate reports that provide baseline information; provide the basis for documenting partition information; diagnose problems and areas of concern
* DSVIEW (NetWare 4) Must download; server-centric; read-only; provides a very granular view of the NDS database without locking it; text-based; somewhat cryptic to navigate
* DSBROWSE (NetWare 5) Must download; server-centric; menu-driven; primarily a read-only tool, but can force a 路send路 or 路receive路 of a selected object or attribute; provides a granular view of NDS without locking the database (but does not show all attributes)
* NDS Manager (NetWare 4 and 5) GUI, workstation-based; centralized administration tool; most of the functions of DSREPAIR; provides error codes and help; not server-centric, therefore can be difficult to tell exactly where the NDS information presented is coming from
* DSTRACE (NetWare 4 and 5) One of your best friends for advanced troubleshooting; text-based console commands; syntax fussy; no help or error messages if syntax incorrect (originally designed for Novell路s internal support engineers - it was not envisioned to be an end-user diagnostic tool)
* LogicSource Must purchase; workstation based; comprehensive library of NDS processes and error codes
* Knowledgebase https://support.novell.com ; ability to search Novell documentation, as well as TIDs (Technical Information Documents)
I) Performing Replica Operations
==========================
* Use NDS Manager, DSREPAIR, or Partition Manager (NetWare 4.x DOS-based tool)
* Replicating a partition: a considerable amount of traffic can be generated, depending on the number of objects in the partition; all object information is sent from the master
* Deleting a replica: check the health of the NDS tree first
* Change replica type: perform health check before proceeding
* Receive updates from the master: this is a destructive operation!; the replicas current data will be overwritten with the data from the master replica; NDS Manager option 路Receive updates路; DSREPAIR option 路Receive all objects from the master to this replica路
* Send updates from a replica: in this case the other replicas of the partition will combine any new objects with NDS data that they already have; NDS Manager option 路Send updates路; DSREPAIR option 路Send all objects to every replica in the ring路
J) DSREPAIR NLM
=================
-A Advanced mode - several additional options under 路Advanced options menu路 including 路declare new epoch路, 路promote subordinate reference to master路, 路destroy replica,路 and so on
-L Specify an alternate location for a log file
-U Unattended mode - useful to run scheduled repairs through CRON
-RC Create a database backup file (SYS:SYSTEM\DSREPAIR.DIB)
-P Flags all unknown objects as 路reference路, so that they get updated on the next heartbeat
-RD Launches unattended repair of local database
-XK2 Highly destructive - last resort - removes all replicas, partitions and objects - leaves behind external references to all objects. (See TID #7001592 for more information)
-XK3 Clears the backlink status of all external references - need to SET DSTRACE=*B in order to force the backlink process to rebuild the backlinks
K) Server Maintenance Tasks
========================
1. Issues to Consider for Removing a Server from the Tree:
* Time synchronization: make sure server is not a time provider
* Check to ensure that server does not hold the master replica of any partition
* NDS is healthy
* No users are connected and no files are open
* No mission critical applications or services are hosted on the server
* If removal is temporary, a placeholder object can and should be used to store any references to the server so that objects referencing that server do not become unknown
* To remove the server, use INSTALL or NWCONFIG, choose 路Directory Options路 and 路Remove Directory Services from this server路
* If NDS fails to remove gracefully, load INSTALL or NWCONFIG with the 路-dsremove路 switch to bypass the normal processes that protect the tree from damage and forcefully remove NDS (this option should only be used as a last resort!)
2. Make Changes to a Server路s Internal IPX Number
* Change the value in AUTOEXEC.NCF
* Restart the server
* Force a limber process (this ensures that the new number is reflected in all NDS entries)
* SET DSTRACE=ON
* SET DSTRACE=+LIMBER
* SET DSTRACE=*L
* SET DSTRACE=*H
* Verify that the limber process worked (DSREPAIR - Servers known to this database)
3. Make Changes to a Server路s Name
* Change the value in AUTOEXEC.NCF
* Restart the server
* Force a limber process (this ensures that the new name is reflected in all NDS entries)
* SET DSTRACE=ON
* SET DSTRACE=+LIMBER
* SET DSTRACE=*L
* SET DSTRACE=*H
* Verify that the limber process worked (DSREPAIR - Servers known to this database)
* Rename the volume objects (using NetWare Administrator)
* For NetWare 5, also change license object assignments to reflect the new name, and delete and recreate the DHCP server object
* Any references to the old server name in login scripts or NAL objects must be changed to reflect the new name and appropriate paths
4. Performing a Hardware Upgrade
* Make sure you have a good backup of the entire server
* Using INSTALL or NWCONFIG, select 路Directory Options路 and then 路.Directory backup and restore options路 (this provides the same functionality as DSMAINT.NLM)
* Choose 路Save local DS information prior to hardware upgrade路 in order to create a backup of the local NDS database in a file called SYS:SYSTEM\BACKUP.NDS (if using DSMAINT the file will be saved as BACKUP.DS)
* Copy the file to another system that will be accessible after the hardware upgrade
* Upgrade the server hardware and reinstall the operating system
* Create a temporary tree
* Login to the new temporary tree and copy BACKUP.NDS to the SYS:SYSTEM directory
* Remove Directory Services (the temporary tree) from the upgraded server and check AUTOEXEC.NCF for correct entries (server name, IPX internal address, bindery context and so on)
* Load INSTALL, NWCONFIG, or DSMAINT, and select 路Restore NDS following hardware upgrade路
* Change the time server type back to the appropriate value and check TIMESYNC.CFG for correct entries
* From DSREPAIR, and 路Advanced Options路, select 路Check volume objects and trustees路
NOTE: During this process, the NDS database is locked on the selected server. The server should be brought back up quickly and no changes made to the NDS tree while the server is down.
5. Recovering from a Failed Server
* Create a 路placeholder路 object to be used with DSMAINT.NLM (any object class is fine except the placeholder object cannot be a server object)
* Load DSMAINT -PSE on another server in the tree and select 路Replace server references路
* Verify that time is synchronized on the network
* Verify that a master replica exists for each partition that was located on the crashed server
* Delete the failed server路s object from the tree with NDS Manager
* Verify the consistency and validity of each replica ring (using DSREPAIR)
* Do NOT Remove the failed server路s volume objects from the tree - the volume objects need to remain there for the placeholder object you referenced with DSMAINT -PSE.
* Reinstall the repaired server into the same context in which it was originally located
* Load DSREPAIR and Check Volume Objects and Trustee's
* Use DSMAINT on the newly installed server to restore server references
* Verify time synchronization.
L) Disaster Recovery Strategies:
==========================
* Partition replication:
* First line of defense in a multi-server tree
* Recommended to have one master and two read/write replicas of every partition
* SMS backup of NDS and the file system:
* Regular backups should be performed
* Partition boundary information is not saved
* When restoring, NDS should be restored first, then the file system; this allows file system rights to be properly assigned to the correct objects
* An NDS database dump:
* DSREPAIR; menu option 路Create a database dump file路
* A snapshot of the NDS database as it exists on a particular server is saved in a compressed format in SYS:SYSTEM\DSREPAIR.DIB
* This is not useful as a routine backup of NDS, but could be used as a last resort by Novell technical support personnel to recover NDS if they are permitted remote access to your servers
* Can also create by executing DSREPAIR -RC at the server console; consider setting up this command in CRON in order to create the database dump file on a regular basis
M) Identify and Resolve Inconsistencies Between Containers
===============================================
* Inconsistencies include problems such as different number of objects within replicas on different servers, and mismatched attributes on the same object on different replicas
Identifying Inconsistencies:
---------------------------------------
* To view and compare the number of objects and attribute values use DSVIEW (NW4.x only), DSBROWSE (NW5 only), or NetWare Administrator
* DSVIEW allows you to safely compare replica contents on NetWare 4.x servers at a very granular level
* DSBROWSE allows you to safely view the replica contents on NetWare 5 servers
* NetWare Administrator can be used, but to ensure that you are reading the NDS information from a particular server, you should lock the NDS database on all other servers in order to query the replica of interest
Resolving Inconsistencies:
--------------------------------------
* The two most helpful menu items in DSREPAIR (available once you select the partition of interest and choose 路View replica ring路) are:
* 路Send all objects to every replica in the ring路
* A nondestructive option that will force synchronization of all replicas in the ring and add any new information from the selected replica to the other servers in the ring
* 路Receive all objects from the master to this replica路
* This is a destructive option that will force a R/W or R/O replica to be completely overwritten by the contents of the master replica (of course this is only applicable if the master replica is not corrupted)
* This option will not work on the server holding the master replica
N) Time Synchronization in the NetWare Environment
=========================================
* Time synchronization is a service running on all NetWare 4.x and 5.x servers to provide consistent time across the network
* Consistent server time is particularly important to make sure that changes to NDS are applied in the correct order. Consider the scenario where the same object is modified in two different replicas - the order in which the modifications were made must be preserved in order to maintain database integrity.
* NDS is able to collate and track changes by applying a time stamp to every NDS action. Servers read the time stamps and are able to process changes in the correct order.
* Time stamps consist of three parts:
* The UTC time (Universal Time Coordinated) - this references Greenwich Mean Time so that all servers will report the same time regardless of their physical time zone
* The replica number which indicates the replica where the NDS action took place
* An event ID to take care of multiple NDS changes which take place within the same second
Time Synchronization with NetWare 4.x (IPX)
-----------------------------------------------------------------
* TIMESYNC only uses IPX as its communication protocol
* SAP (Service Advertising Protocol) is used to advertise the presence of time providers on the network
* There are four types of time servers:
* Single Reference: the default configuration for the first server in the tree; all other servers will be automatically configured as Secondary time servers; does not adjust it路s internal clock and cannot coexist with any other time provider
* Reference: provides a high priority, central point of time control; does not adjust it路s internal clock; must arbitrate time with at least one Primary time server (configuring two to seven Primary time servers is recommended if a Reference server is used)
* Primary: polls Reference and other Primary servers to calculate a weighted average value for time and adjusts its clock accordingly (each Primary has a weight of 1; the Reference has a weight of 16)
* Secondary: the default time server type for the second and subsequent servers installed into the tree; adjusts it路s internal clock to be consistent with the chosen time provider; can coexist with all other time servers
* There are two methods in which time servers learn of each other:
* SAP: the default; no configuration required
* Configured lists: maintained in SYS:SYSTEM\TIMESYNC.CFG; provides the ability to specify which time providers will arbitrate time and which time providers will service a particular Secondary server
Time Synchronization with NetWare 5 or a Mixed Environment (IP only or IP/IPX)
-------------------------------------------------------------------------------------------------------------------
* Time stamps are provided by Internet time sources using NTP (Network Time Protocol) - a standard Internet Protocol specification (RFC 1305)
* Novell路s TIMESYNC.NLM is able to function over IP and provides interoperability with NTP (Note: this eliminates the need to use NTP.NLM and the NTP.CFG configuration file)
* To point to an external NTP time source, set 路Timesync configured sources路 to ON, and in the 路Timesync time sources路 field enter the IP address of the source time server (Eg. SET TIMESYNC TIME SOURCES=192.168.1.1:123; (where 123 is the is the UDP port that NTP uses))
Troubleshooting Time Synchronization Issues
------------------.------------------------------------------------
* Synthetic Time:
* A condition where the time provided by the local operating system is older than the timestamps already recorded in the NDS database - this would be the case if server time was set forward, and then turned back (as, for example, in Y2K testing)
* Two ways to resolve:
* Do nothing: let server time catch up to the future timestamps; appropriate if the timestamps are not too far out in the future
* DSREPAIR -A (must use -A switch to enable advanced options): choose partition issuing synthetic time and select 路Repair time stamps and declare a new epoch路; this operation is performed on the master replica; all replicas will receive a copy of all objects from the master; do this only when absolutely necessary; reduce the number of replicas to as few as possible and make sure all servers communicating properly
* Time Packet Filtering:
* Make sure that SAP filtering is not preventing servers relying on that advertisement to keep their time in sync
* When working with IP and TIMESYNC, be aware that TIMESYNC uses UDP port 524, and NTP will most often require UDP port 123
.
O) Server-to-Server Communication
============================
* If NDS is distributed across multiple servers, then good communication between servers is vital
* A common DSTRACE or DSREPAIR error that indicates a communication problem is a 625 error
* Two useful server NLMs exist to check to see if servers are communicating properly:
* IPXPING: must also have IPXRTR.NLM loaded; LOAD IPXPING and provide the IPX internal network and node address of the server you wish to communicate with; can also specify the IPX external network address and the actual MAC address of the network card
* PING: checks the status of IP communications between hosts; LOAD PING and provide the IP address of the server you wish to communicate with
* Possible causes of communication problems:
* Routers: dropped packets; insufficient buffer capability
* WAN links: numerous slow links; partition boundaries spanning a slow link
* Locked NDS database: DS.NLM not loaded
* High server CPU utilization: may prevent or hinder NDS communication
* Packet filtering: SAP filters and IP ports that are blocked may prevent NDS communication
* Authentication errors: if the public key for a server becomes corrupt the server with the corrupt key will issue a 632 error when other servers try to establish a connection
Troubleshooting Authentication Failures (632 Error)
-------------------------------------------------------------------------
* Often caused by a corrupt public key for a server object
* Two possible solutions:
* If the server object is contained in a replica on the server, delete the replica and place a new one back on the server
If the server object is contained in an external reference, there are 2 options:
* Place a replica on the server: this will overwrite the external reference attributes with the actual server object information
* Load DSREPAIR -XK3 and choose 路Repair local DS database路 to clear the backlink status of all external references; then SET DSTRACE=*B to force the backlink process to rebuild the links (Note: XK3 is one of the so-called 路killer switches路 which should be used only as a last resort because of its destructive nature)
P) Identify and Repair Corrupted Objects
================================
* It is possible that objects in NDS become corrupted and require administrator intervention to manually clean them up
Unknown objects which require intervention from the network administrator are represented in NetWare Administrator as yellow circles with a question mark inside
* Another type of unknown object is represented with a white box encompassing a question mark, which indicates a valid object type, but one which * NetWare Administrator cannot manage (this usually occurs because the appropriate snap-in to manage that object class has not been installed, or does not exist)
Strategies for Cleaning Up 路Unknown路 Objects:
-------------------------------------------------------------------
* Delete the unknown object(s)
* Replace or repair the replica which has the unknown object(s)
* Use NDS Manager to delete the corrupted replica, and place a replica back on the server
* Use DSREPAIR to have the corrupted replica 路Receive all objects from the master to this replica路
* Using DSREPAIR from a server holding a good replica select 路Send all objects to every replica in the ring路
* On the server with the corrupted replica, load DSREPAIR with the -P switch to flag all unknown objects as 路reference路 objects; on the next heartbeat interval the master will recognize that it must send all attributes to these objects (which appear to be newly created)
Q) Troubleshooting Obituaries
=========================
* NDS creates an obituary attribute when any of the following actions occur:
* Renaming an object
* Deleting an object
* Moving a sub-tree (container) or an object
* There are two types of obituaries:
* Primary: indicates an action on an object (numbered 1 through 5, and 7 through 10)
* Secondary: indicates which servers must be contacted (number 6)
NOTE: Obituary numbers are referenced on the DSTRACE debug screen.
* The obituary process usually becomes stalled because servers are not able to communicate reliably, or a server object has been improperly removed from the tree
* Problems can usually be resolved in one of three ways:
* Make sure all servers are up and communicating with other servers: the obituary process will continue through its four stages and the janitor process will do the final cleanup
* If the server is permanently gone, take the appropriate steps to delete the server from the tree using NDS Manager
* If the server is communicating, use DSREPAIR -XK3 to clear the backlinks and then SET DSTRACE=*B to force the backlink process to rebuild the links
R) Upgrading to NDS 8
====================
Minimum Requirements:
-----------------------------------
* NetWare 5 with Support Pack 2 or later
* NICI 1.2 or later (Novell cryptography support modules)
* Administrative rights to [Root] (in order to modify the schema)
* DSREPAIR download if the first installation of NDS 8 will be to a server that does not hold a replica of [Root]
* The NDS tree may not contain servers running NetWare 4.10 or earlier
Preparation:
-----------------
* If the first server to be upgraded contains a replica of [Root] no special preparation steps need to be undertaken
* If the first server to be upgraded does not contain a replica of [Root], the schema must be extended by copying the most current version of DSREPAIR to every server in the tree and choosing 路Advanced options,路 then 路Global schema operations路 and 路Post NetWare 5 Schema Update路
Steps to Deploy NDS 8:
----------------------------------
* Download source files and review any included readme files
* Prepare the tree as per above (extend the schema if the first server to be upgraded does not contain a replica of [Root])
* Upgrade to the newest version of DSREPAIR on all servers in the tree
* Pick a target server: NDS 8 can be installed on any NetWare 5 server in your network; Novell suggests picking a server other than one of your servers holding the [Root] partition (i.e. a less critical server), but to first extend the schema on one of the servers holding a copy of [Root]
* Make sure you have a complete and trusted backup
* Install at least Support Pack 2
* Choose a method of installation (across the wire, from CD, etc.)
* Make sure all volumes are mounted to ensure that trustee assignments are properly updated
* Re-mark out third-party NLMs from AUTOEXEC.NCF
* Make sure AUTOEXEC.NCF file is closed
* If running in a pure IP environment, load IPXSPX.NLM (only required temporarily during the upgrade process)
* Run the installation program
* Test and re-enable third-party NLMs
* Update backlinked objects with SET DSTRACE=*B
* Extend the LDAP schema (load DSREPAIR, choose 路Advanced options路, 路Global schema operations路, and 路Optional schema enhancements路)
* Load the new version of DSREPAIR and do an 路Unattended full repair路 to apply the new containment rule to all objects
Topics covered in this document are :
A) NDS Health Checks
B) NDS Background Processes
C) Replica Guidelines
D) External References and Backlinks
E) Replica Sync in NW 4 v NW 5
F) Replica Transition States
G) Steps to NDS Troubleshooting
H) Tools used to Investigate and Resolve NDS Issues
I) Performing Replica Operations
J) Dsrepair
K) Server Maintenance Tasks
L) Disaster Recovery Strategies
M) Identify and Resolve Inconsistencies Between Containers
N) Time Synchronization in the NetWare Environment
O) Server-to-Server Communication
P) Identify and Repair Corrupted Objects
Q) Troubleshooting Obituaries
R) Upgrading to NDS8
A) NDS Health Checks
For complete DS health check information and procedures, please see TID 3564075
====================
DSTRACE Commands for NDS Health Checks: (NOTE: DSTRACE commands are not case-sensitive)
---------------------------------------------------------------------------------------------------------------------------------------------------
* SET DSTRACE=+variable -enables messages pertaining to an NDS process
* SET DSTRACE=-variable -disables messages pertaining to an NDS process
* SET DSTRACE=*variable -initiates an NDS process
* SET DSTRACE=!variable -tunes an NDS parameter
Health Check Item DSTRACE Command Action
========================================================================================
General SET DSTRACE=ON Enables dstrace screen
SET TTF=ON Creates DSTRACE.DBG (if it doesn路t exist)
SET DSTRACE=*R Clears the log file if it already exists
SET DSTRACE=*. Initializes the NDS database
Partition continuity SET DSTRACE=+S Displays messages during replica synchronization
SET DSTRACE=*H Initiates a 路heartbeat路 (replica synch.)
External references SET DSTRACE=+BLINK Displays messages during backlink process
SET DSTRACE=*B Initiates the backlink process
Obituaries SET DSTRACE=+J Displays messages during janitor process
SET DSTRACE=*F Initiates the flat-cleaner process (incl. janitor)
NDS schema SET DSTRACE=+SCHEMA &.nbsp; Displays schema messages
SET DSTRACE=*SS Initiates the schema synchronization process
All background processes SET DSTRACE=+S Displays synchronization messages
SET DSTRACE=+SCHEMA Displays schema messages
SET DSTRACE=+IN Displays incoming synch. Information
SET DSTRACE=+LIMBER Displays messages for the limber process
SET DSTRACE=+MISC Bagging and other background processes
SET DSTRACE=+AGENT Janitor and additional background processes
SET DSTRACE=*H Triggers the replica synchronization process
NDS SET parameters SET DSTRACE=*P Displays the current value of tunable NDS parameters
Use of CRON.NLM to Automate Health Check Procedures
-------------------------------------------------------------------------------------
* Schedule commands to be executed at the server console for specific dates and times
* Must first create the configuration file, SYS:ETC\CRONTAB
Document Server Configuration Information
--------------------------------------------------------------
* CONFIG.NLM creates a text file detailing server configuration information and startup files
* TECHWALK.NLM generates a report on system configuration by walking through INETCFG (the report file is created as SYS:ETC\TECHWALK.OUT)
.
B) NDS Background Processes
==========================
* Replica synchronization:
* Guarantees consistency of NDS information across all replicas of a given partition
* Only changes are sent
* Some changes are sent immediately, for example, a user路s password change (within 10 sec.); this is known as an immediate sync cycle
* Some less critical changes are collected locally before being sent (within 30 min.); this is known as a slow sync cycle
* The master replica is a participant in the process, but is not the controlling entity
* Synchronization is done periodically and is referred to as the NDS heartbeat or skulking process; this takes place every 30 min. for NetWare 4.x and every 60 min. for NetWare 5 (default values)
* Janitor process:
* When an object is deleted or moved, the janitor process is responsible for cleaning up NDS information
* Looks after a number of tasks:
* Verifies the status attribute of the server路s NDS object
* Executes the flat cleaner process
* Manages the obituary state of moved, deleted or renamed objects
* Performs periodic optimizations of the NDS database to speed up DS lookups
* Verifies the identity of the [ROOT] most replica
* Notifies the operating system of synthetic time
* Flat cleaner:
* Removes obituaries in the purgeable state
* Ensures that each object has a public key
* Purges unused objects and attributes in the external reference and bindery partitions
* Replica purger:
* Processes obituaries
* Deletes unused objects and attributes in user-defined partitions
* Obituary:
* This is a special attribute which is used to track deleted, moved or renamed objects
* When an object is flagged for deletion it moves through the following four stages:
* flag=0 issued
* flag=1 notified
* flag=2 ok_to_purge
* flag=4 purgeable
* The obituary stages are to ensure that all servers holding a replica retain the old information while all other servers are receiving the updates
* After the obituary is flagged purgeable, it is up to the local server路s DS to actually purge the entry
* The background processes involved in processing obituaries are the janitor, flat cleaner, purger and replica synchronization processes
* Limber:
* Verifies the name of the NDS tree
* Verifies the serv.er路s name, internal network address and credentials
* Schema synchronization:
* Makes sure that the same schema structure is present across all servers
* Backlinker:
* Ensures that external references held on a server are required and properly backlinked to the real object
* Removes external references that are no longer required
* Database initialization:
* Initializes NDS and verifies the usability of the database files
* Schedules various background processes to run
* Opens the database files for use by NDS
* Is triggered by mounting the SYS: volume, or unloading and reloading DS.NLM
.
C) Replica Guidelines
===================
* NDS partitioning and replication is done to provide scalability, fault tolerance, and performance
* There are four replica types:
* Master: only one per partition; created when a partition is defined; controlling entity in partition and replica operations
* Read/write: created automatically on the second and third servers in a partition; additional read/write replicas can be created manually on servers that you specify
* Read-only: created on servers that you specify; no real purpose at this time
* Subordinate reference: created automatically on servers that hold a copy of a parent partition but not the child; does not contain object data and, therefore, does not provide fault tolerance!; used to provide tree connectivity
* Novell recommends that each partition have at least two to three replicas (typically, a master and two read/write replicas)
* The replica ring is a list of all servers that have information about a particular partition, the type of replica and the replica state
D) External References and Backlinks
===============================
* Used to help manage the database and provide tree connectivity
* External references refer to objects not physically located on the local server
* External references keep track of non-local objects such as:
* An object that authenticates or attaches to the server
* An object that is added as a file system or local object trustee
* An object that is added as a member of a group
* When NDS creates an external reference, a backlink attribute is added to the referenced object to keep track of the server on which the external reference was created
E) Replica synchronization in NetWare 4 vs NetWare 5
==========================================
* There are some important differences affecting scalability and efficiency:
* Replica sync process
* NetWare 4 Uses sync-up-to vector; each server must contact every other server in the replica ring; the number of server-to-server communications will be n*(n-1), where n=the number of servers in the replica ring
* NetWare 5 Uses transitive vector; replica convergence is quicker and takes fewer server-to-server communications
* Replica ring updates
* NetWare 4 Updates replicas in a sequential order, the master always first
* NetWare 5 Updates in a randomized order, therefore less likely to bottleneck on the inbound sync process
* Read-only replicas
* NetWare 4 Can start the synchronization process
* NetWare 5 Cannot start the sync process
* Subordinate-ref. replicas
* NetWare 4 Participate in normal replica synchronization
* NetWare 5 Not involved in replica synchronization
* Object changes
* NetWare 4 Sequentially searches the entire NDS database to detect changes
* NetWare 5 Caches object changes, therefore does not have to scan the entire database
* Objects per sync packet
* NetWare 4 Only one object per sync packet
* NetWare 5 Multiple objects per packet
F) Replica Transition States
=======================
* DSTRACE is useful to view replica transition states and to identify replica operation processes and errors
* The state of the replica is server-centric, and will be one of the following:
* [0] On: all entries have been copied for the partition
* [1] New Replica: begins the operation of adding to the replica list; sub.ref. replicas are added to appropriate servers
* [2] Dying Replica: when a replica is removed; ext. references are created and backlinked to the real objects
* [3] Locked: certain operations require that only one change on that partition can occur at one time
* [4] Change Replica Type - State 0: designate a new master
* [5] Change Replica Type - State 1: the new master places new timestamps on all replicas and changes from State 0 to State 1; the old master appoints itself as a R/W
* [6] Transition State On: occurs after [1] New Replica; after the replica has communicated with every server in the ring, the master must send the On state to all replicas once they are updated and synchronized
* [48] Split State 0: new partition boundary created
* [49] Split State 1: master makes sure all replicas are updated and synchronized and sends the On state
* [64] Join State 0: when two partitions are combined; servers that do not have a copy of both the child and parent get one; move to [65] Join State 1
* [65] Join State 1: erase the boundary between the 2 partitions; move to [66] Join State 2
* [66] Join State 2: propagate the changes of the combined partition to all servers in the replica list
* [80] Move State 0: partition (sub-tree) move; must not have any child partitions and must have consistent schema rules between source and destination servers
G) Steps to NDS Troubleshooting:
===========================
1. Identify the Scope of the Problem:
* Consider error codes and console messages; identify the server(s) involved, affected partition(s), and symptoms
2. Determine the Cause:
* Which NDS process is experiencing a problem, and why
3. List Possible Solutions:
* Understand the repercussions of each possible solution
4. Assess Possible Solutions:
* Consider the level of difficulty, the effect on users, and the effect on NDS (consider the least intrusive solutions first)
5. Implement Solution:
* Create a backup of NDS first; allow sufficient time for NDS to stabilize and correct itself
6. Verify the Problem is Resolved:
* Perform health check; confirm that NDS operations can be completed
7. Document the Resolution:
* Retain in server maintenance logs for future reference
8. Avoid Repetition of the Problem:
* Establish appropriate policies and procedures (cutting the hands off your co-workers is not considered to be an acceptable solution)
H) Tools Used to Investigate and Resolve NDS Issues:
==========================================
* DSREPAIR (NetWare 4 and 5) Server-centric; 路C-worthy路 interface on server console; checks for and corrects problems in the Directory database on an individual server basis; reports information about Directory Services, time synchronization, and server status
* DSDIAG (NetWare 4 and 5) Menu-based utility that allows you to generate diagnostic reports on NDS; generate reports that provide baseline information; provide the basis for documenting partition information; diagnose problems and areas of concern
* DSVIEW (NetWare 4) Must download; server-centric; read-only; provides a very granular view of the NDS database without locking it; text-based; somewhat cryptic to navigate
* DSBROWSE (NetWare 5) Must download; server-centric; menu-driven; primarily a read-only tool, but can force a 路send路 or 路receive路 of a selected object or attribute; provides a granular view of NDS without locking the database (but does not show all attributes)
* NDS Manager (NetWare 4 and 5) GUI, workstation-based; centralized administration tool; most of the functions of DSREPAIR; provides error codes and help; not server-centric, therefore can be difficult to tell exactly where the NDS information presented is coming from
* DSTRACE (NetWare 4 and 5) One of your best friends for advanced troubleshooting; text-based console commands; syntax fussy; no help or error messages if syntax incorrect (originally designed for Novell路s internal support engineers - it was not envisioned to be an end-user diagnostic tool)
* LogicSource Must purchase; workstation based; comprehensive library of NDS processes and error codes
* Knowledgebase https://support.novell.com ; ability to search Novell documentation, as well as TIDs (Technical Information Documents)
I) Performing Replica Operations
==========================
* Use NDS Manager, DSREPAIR, or Partition Manager (NetWare 4.x DOS-based tool)
* Replicating a partition: a considerable amount of traffic can be generated, depending on the number of objects in the partition; all object information is sent from the master
* Deleting a replica: check the health of the NDS tree first
* Change replica type: perform health check before proceeding
* Receive updates from the master: this is a destructive operation!; the replicas current data will be overwritten with the data from the master replica; NDS Manager option 路Receive updates路; DSREPAIR option 路Receive all objects from the master to this replica路
* Send updates from a replica: in this case the other replicas of the partition will combine any new objects with NDS data that they already have; NDS Manager option 路Send updates路; DSREPAIR option 路Send all objects to every replica in the ring路
J) DSREPAIR NLM
=================
-A Advanced mode - several additional options under 路Advanced options menu路 including 路declare new epoch路, 路promote subordinate reference to master路, 路destroy replica,路 and so on
-L Specify an alternate location for a log file
-U Unattended mode - useful to run scheduled repairs through CRON
-RC Create a database backup file (SYS:SYSTEM\DSREPAIR.DIB)
-P Flags all unknown objects as 路reference路, so that they get updated on the next heartbeat
-RD Launches unattended repair of local database
-XK2 Highly destructive - last resort - removes all replicas, partitions and objects - leaves behind external references to all objects. (See TID #7001592 for more information)
-XK3 Clears the backlink status of all external references - need to SET DSTRACE=*B in order to force the backlink process to rebuild the backlinks
K) Server Maintenance Tasks
========================
1. Issues to Consider for Removing a Server from the Tree:
* Time synchronization: make sure server is not a time provider
* Check to ensure that server does not hold the master replica of any partition
* NDS is healthy
* No users are connected and no files are open
* No mission critical applications or services are hosted on the server
* If removal is temporary, a placeholder object can and should be used to store any references to the server so that objects referencing that server do not become unknown
* To remove the server, use INSTALL or NWCONFIG, choose 路Directory Options路 and 路Remove Directory Services from this server路
* If NDS fails to remove gracefully, load INSTALL or NWCONFIG with the 路-dsremove路 switch to bypass the normal processes that protect the tree from damage and forcefully remove NDS (this option should only be used as a last resort!)
2. Make Changes to a Server路s Internal IPX Number
* Change the value in AUTOEXEC.NCF
* Restart the server
* Force a limber process (this ensures that the new number is reflected in all NDS entries)
* SET DSTRACE=ON
* SET DSTRACE=+LIMBER
* SET DSTRACE=*L
* SET DSTRACE=*H
* Verify that the limber process worked (DSREPAIR - Servers known to this database)
3. Make Changes to a Server路s Name
* Change the value in AUTOEXEC.NCF
* Restart the server
* Force a limber process (this ensures that the new name is reflected in all NDS entries)
* SET DSTRACE=ON
* SET DSTRACE=+LIMBER
* SET DSTRACE=*L
* SET DSTRACE=*H
* Verify that the limber process worked (DSREPAIR - Servers known to this database)
* Rename the volume objects (using NetWare Administrator)
* For NetWare 5, also change license object assignments to reflect the new name, and delete and recreate the DHCP server object
* Any references to the old server name in login scripts or NAL objects must be changed to reflect the new name and appropriate paths
4. Performing a Hardware Upgrade
* Make sure you have a good backup of the entire server
* Using INSTALL or NWCONFIG, select 路Directory Options路 and then 路.Directory backup and restore options路 (this provides the same functionality as DSMAINT.NLM)
* Choose 路Save local DS information prior to hardware upgrade路 in order to create a backup of the local NDS database in a file called SYS:SYSTEM\BACKUP.NDS (if using DSMAINT the file will be saved as BACKUP.DS)
* Copy the file to another system that will be accessible after the hardware upgrade
* Upgrade the server hardware and reinstall the operating system
* Create a temporary tree
* Login to the new temporary tree and copy BACKUP.NDS to the SYS:SYSTEM directory
* Remove Directory Services (the temporary tree) from the upgraded server and check AUTOEXEC.NCF for correct entries (server name, IPX internal address, bindery context and so on)
* Load INSTALL, NWCONFIG, or DSMAINT, and select 路Restore NDS following hardware upgrade路
* Change the time server type back to the appropriate value and check TIMESYNC.CFG for correct entries
* From DSREPAIR, and 路Advanced Options路, select 路Check volume objects and trustees路
NOTE: During this process, the NDS database is locked on the selected server. The server should be brought back up quickly and no changes made to the NDS tree while the server is down.
5. Recovering from a Failed Server
* Create a 路placeholder路 object to be used with DSMAINT.NLM (any object class is fine except the placeholder object cannot be a server object)
* Load DSMAINT -PSE on another server in the tree and select 路Replace server references路
* Verify that time is synchronized on the network
* Verify that a master replica exists for each partition that was located on the crashed server
* Delete the failed server路s object from the tree with NDS Manager
* Verify the consistency and validity of each replica ring (using DSREPAIR)
* Do NOT Remove the failed server路s volume objects from the tree - the volume objects need to remain there for the placeholder object you referenced with DSMAINT -PSE.
* Reinstall the repaired server into the same context in which it was originally located
* Load DSREPAIR and Check Volume Objects and Trustee's
* Use DSMAINT on the newly installed server to restore server references
* Verify time synchronization.
L) Disaster Recovery Strategies:
==========================
* Partition replication:
* First line of defense in a multi-server tree
* Recommended to have one master and two read/write replicas of every partition
* SMS backup of NDS and the file system:
* Regular backups should be performed
* Partition boundary information is not saved
* When restoring, NDS should be restored first, then the file system; this allows file system rights to be properly assigned to the correct objects
* An NDS database dump:
* DSREPAIR; menu option 路Create a database dump file路
* A snapshot of the NDS database as it exists on a particular server is saved in a compressed format in SYS:SYSTEM\DSREPAIR.DIB
* This is not useful as a routine backup of NDS, but could be used as a last resort by Novell technical support personnel to recover NDS if they are permitted remote access to your servers
* Can also create by executing DSREPAIR -RC at the server console; consider setting up this command in CRON in order to create the database dump file on a regular basis
M) Identify and Resolve Inconsistencies Between Containers
===============================================
* Inconsistencies include problems such as different number of objects within replicas on different servers, and mismatched attributes on the same object on different replicas
Identifying Inconsistencies:
---------------------------------------
* To view and compare the number of objects and attribute values use DSVIEW (NW4.x only), DSBROWSE (NW5 only), or NetWare Administrator
* DSVIEW allows you to safely compare replica contents on NetWare 4.x servers at a very granular level
* DSBROWSE allows you to safely view the replica contents on NetWare 5 servers
* NetWare Administrator can be used, but to ensure that you are reading the NDS information from a particular server, you should lock the NDS database on all other servers in order to query the replica of interest
Resolving Inconsistencies:
--------------------------------------
* The two most helpful menu items in DSREPAIR (available once you select the partition of interest and choose 路View replica ring路) are:
* 路Send all objects to every replica in the ring路
* A nondestructive option that will force synchronization of all replicas in the ring and add any new information from the selected replica to the other servers in the ring
* 路Receive all objects from the master to this replica路
* This is a destructive option that will force a R/W or R/O replica to be completely overwritten by the contents of the master replica (of course this is only applicable if the master replica is not corrupted)
* This option will not work on the server holding the master replica
N) Time Synchronization in the NetWare Environment
=========================================
* Time synchronization is a service running on all NetWare 4.x and 5.x servers to provide consistent time across the network
* Consistent server time is particularly important to make sure that changes to NDS are applied in the correct order. Consider the scenario where the same object is modified in two different replicas - the order in which the modifications were made must be preserved in order to maintain database integrity.
* NDS is able to collate and track changes by applying a time stamp to every NDS action. Servers read the time stamps and are able to process changes in the correct order.
* Time stamps consist of three parts:
* The UTC time (Universal Time Coordinated) - this references Greenwich Mean Time so that all servers will report the same time regardless of their physical time zone
* The replica number which indicates the replica where the NDS action took place
* An event ID to take care of multiple NDS changes which take place within the same second
Time Synchronization with NetWare 4.x (IPX)
-----------------------------------------------------------------
* TIMESYNC only uses IPX as its communication protocol
* SAP (Service Advertising Protocol) is used to advertise the presence of time providers on the network
* There are four types of time servers:
* Single Reference: the default configuration for the first server in the tree; all other servers will be automatically configured as Secondary time servers; does not adjust it路s internal clock and cannot coexist with any other time provider
* Reference: provides a high priority, central point of time control; does not adjust it路s internal clock; must arbitrate time with at least one Primary time server (configuring two to seven Primary time servers is recommended if a Reference server is used)
* Primary: polls Reference and other Primary servers to calculate a weighted average value for time and adjusts its clock accordingly (each Primary has a weight of 1; the Reference has a weight of 16)
* Secondary: the default time server type for the second and subsequent servers installed into the tree; adjusts it路s internal clock to be consistent with the chosen time provider; can coexist with all other time servers
* There are two methods in which time servers learn of each other:
* SAP: the default; no configuration required
* Configured lists: maintained in SYS:SYSTEM\TIMESYNC.CFG; provides the ability to specify which time providers will arbitrate time and which time providers will service a particular Secondary server
Time Synchronization with NetWare 5 or a Mixed Environment (IP only or IP/IPX)
-------------------------------------------------------------------------------------------------------------------
* Time stamps are provided by Internet time sources using NTP (Network Time Protocol) - a standard Internet Protocol specification (RFC 1305)
* Novell路s TIMESYNC.NLM is able to function over IP and provides interoperability with NTP (Note: this eliminates the need to use NTP.NLM and the NTP.CFG configuration file)
* To point to an external NTP time source, set 路Timesync configured sources路 to ON, and in the 路Timesync time sources路 field enter the IP address of the source time server (Eg. SET TIMESYNC TIME SOURCES=192.168.1.1:123; (where 123 is the is the UDP port that NTP uses))
Troubleshooting Time Synchronization Issues
------------------.------------------------------------------------
* Synthetic Time:
* A condition where the time provided by the local operating system is older than the timestamps already recorded in the NDS database - this would be the case if server time was set forward, and then turned back (as, for example, in Y2K testing)
* Two ways to resolve:
* Do nothing: let server time catch up to the future timestamps; appropriate if the timestamps are not too far out in the future
* DSREPAIR -A (must use -A switch to enable advanced options): choose partition issuing synthetic time and select 路Repair time stamps and declare a new epoch路; this operation is performed on the master replica; all replicas will receive a copy of all objects from the master; do this only when absolutely necessary; reduce the number of replicas to as few as possible and make sure all servers communicating properly
* Time Packet Filtering:
* Make sure that SAP filtering is not preventing servers relying on that advertisement to keep their time in sync
* When working with IP and TIMESYNC, be aware that TIMESYNC uses UDP port 524, and NTP will most often require UDP port 123
.
O) Server-to-Server Communication
============================
* If NDS is distributed across multiple servers, then good communication between servers is vital
* A common DSTRACE or DSREPAIR error that indicates a communication problem is a 625 error
* Two useful server NLMs exist to check to see if servers are communicating properly:
* IPXPING: must also have IPXRTR.NLM loaded; LOAD IPXPING and provide the IPX internal network and node address of the server you wish to communicate with; can also specify the IPX external network address and the actual MAC address of the network card
* PING: checks the status of IP communications between hosts; LOAD PING and provide the IP address of the server you wish to communicate with
* Possible causes of communication problems:
* Routers: dropped packets; insufficient buffer capability
* WAN links: numerous slow links; partition boundaries spanning a slow link
* Locked NDS database: DS.NLM not loaded
* High server CPU utilization: may prevent or hinder NDS communication
* Packet filtering: SAP filters and IP ports that are blocked may prevent NDS communication
* Authentication errors: if the public key for a server becomes corrupt the server with the corrupt key will issue a 632 error when other servers try to establish a connection
Troubleshooting Authentication Failures (632 Error)
-------------------------------------------------------------------------
* Often caused by a corrupt public key for a server object
* Two possible solutions:
* If the server object is contained in a replica on the server, delete the replica and place a new one back on the server
If the server object is contained in an external reference, there are 2 options:
* Place a replica on the server: this will overwrite the external reference attributes with the actual server object information
* Load DSREPAIR -XK3 and choose 路Repair local DS database路 to clear the backlink status of all external references; then SET DSTRACE=*B to force the backlink process to rebuild the links (Note: XK3 is one of the so-called 路killer switches路 which should be used only as a last resort because of its destructive nature)
P) Identify and Repair Corrupted Objects
================================
* It is possible that objects in NDS become corrupted and require administrator intervention to manually clean them up
Unknown objects which require intervention from the network administrator are represented in NetWare Administrator as yellow circles with a question mark inside
* Another type of unknown object is represented with a white box encompassing a question mark, which indicates a valid object type, but one which * NetWare Administrator cannot manage (this usually occurs because the appropriate snap-in to manage that object class has not been installed, or does not exist)
Strategies for Cleaning Up 路Unknown路 Objects:
-------------------------------------------------------------------
* Delete the unknown object(s)
* Replace or repair the replica which has the unknown object(s)
* Use NDS Manager to delete the corrupted replica, and place a replica back on the server
* Use DSREPAIR to have the corrupted replica 路Receive all objects from the master to this replica路
* Using DSREPAIR from a server holding a good replica select 路Send all objects to every replica in the ring路
* On the server with the corrupted replica, load DSREPAIR with the -P switch to flag all unknown objects as 路reference路 objects; on the next heartbeat interval the master will recognize that it must send all attributes to these objects (which appear to be newly created)
Q) Troubleshooting Obituaries
=========================
* NDS creates an obituary attribute when any of the following actions occur:
* Renaming an object
* Deleting an object
* Moving a sub-tree (container) or an object
* There are two types of obituaries:
* Primary: indicates an action on an object (numbered 1 through 5, and 7 through 10)
* Secondary: indicates which servers must be contacted (number 6)
NOTE: Obituary numbers are referenced on the DSTRACE debug screen.
* The obituary process usually becomes stalled because servers are not able to communicate reliably, or a server object has been improperly removed from the tree
* Problems can usually be resolved in one of three ways:
* Make sure all servers are up and communicating with other servers: the obituary process will continue through its four stages and the janitor process will do the final cleanup
* If the server is permanently gone, take the appropriate steps to delete the server from the tree using NDS Manager
* If the server is communicating, use DSREPAIR -XK3 to clear the backlinks and then SET DSTRACE=*B to force the backlink process to rebuild the links
R) Upgrading to NDS 8
====================
Minimum Requirements:
-----------------------------------
* NetWare 5 with Support Pack 2 or later
* NICI 1.2 or later (Novell cryptography support modules)
* Administrative rights to [Root] (in order to modify the schema)
* DSREPAIR download if the first installation of NDS 8 will be to a server that does not hold a replica of [Root]
* The NDS tree may not contain servers running NetWare 4.10 or earlier
Preparation:
-----------------
* If the first server to be upgraded contains a replica of [Root] no special preparation steps need to be undertaken
* If the first server to be upgraded does not contain a replica of [Root], the schema must be extended by copying the most current version of DSREPAIR to every server in the tree and choosing 路Advanced options,路 then 路Global schema operations路 and 路Post NetWare 5 Schema Update路
Steps to Deploy NDS 8:
----------------------------------
* Download source files and review any included readme files
* Prepare the tree as per above (extend the schema if the first server to be upgraded does not contain a replica of [Root])
* Upgrade to the newest version of DSREPAIR on all servers in the tree
* Pick a target server: NDS 8 can be installed on any NetWare 5 server in your network; Novell suggests picking a server other than one of your servers holding the [Root] partition (i.e. a less critical server), but to first extend the schema on one of the servers holding a copy of [Root]
* Make sure you have a complete and trusted backup
* Install at least Support Pack 2
* Choose a method of installation (across the wire, from CD, etc.)
* Make sure all volumes are mounted to ensure that trustee assignments are properly updated
* Re-mark out third-party NLMs from AUTOEXEC.NCF
* Make sure AUTOEXEC.NCF file is closed
* If running in a pure IP environment, load IPXSPX.NLM (only required temporarily during the upgrade process)
* Run the installation program
* Test and re-enable third-party NLMs
* Update backlinked objects with SET DSTRACE=*B
* Extend the LDAP schema (load DSREPAIR, choose 路Advanced options路, 路Global schema operations路, and 路Optional schema enhancements路)
* Load the new version of DSREPAIR and do an 路Unattended full repair路 to apply the new containment rule to all objects
Additional Information
Formerly known as TID# 10064019