Novell is now a part of Micro Focus

An Introduction to Novell Replication Services

Articles and Tips: article

TOM MATTINGLY
Product Manager
Novell Technical Services

01 Jun 1997


Another exciting product is Novell Replication Services, an IntranetWare application that provides an effective way to distribute information to network users. This AppNote brings you up to speed on how it all works.

Introduction

Today's fast-moving, global business environment often requires widely distributed groups of people to have immediate access to information­market studies, sales presentations, financial reports, address books, and other productivity documents­normally found only on corporate servers. Remote locations often have only a slow or intermittent WAN connection, making quick access to this information difficult. Although faster WAN technologies are being created and used by companies, end users trying to access information are still faced with the problem of how to quickly access data from main corporate servers. Additionally, keeping remote machines in synchronization with the main servers presents a difficult challenge. Network administrators have often turned to replicating file systems on servers throughout a network to reduce delays and provide remote users with the latest copies of software and documents.

Replication improves WAN performance by moving information closer to the end user. However, manually replicating and synchronizing data across an entire enterprise requires daily analysis of file system changes, and frequently involves error-prone manual file transfers or writing batch files or scripts. To maximize any benefits of replication, Novell has developed a new replication management tool that is easy to use and administer, and that optimizes WAN usage and performance. This tool is called Novell Replication Services.

This AppNote provides a detailed introduction to Novell Replication Services, including the administration benefits incurred by using NRS and the importance of synchronization in replication. It includes an explanation of how NRS works and an example of how NRS might be used in a global company. An appendix lists some of the things you should be aware of when planning for replication.

Overview of Novell Replication Services

Novell Replication Services (NRS) is an IntranetWare application that provides an effective way of distributing information across a network of geographically- separated IntranetWare servers. It is specifically intended for, but not limited to, use over slow, high-latency, or intermittent WAN links. It takes advantage of Novell Directory Services (NDS), which provides the required administrative and security infrastructure for the replication service.

The information that you, as a network administrator, can manage using NRS includes documents and other files within a Windows environment that are normally updated by only one user or group at a time, but which are required simultaneously in several different locations. Typical examples include:

  • Application documents, including word processing and desktop publishing documents, spreadsheets and presentations

  • HTML files, including both text and images

  • Software updates

NRS does not support replication of concurrently updated files, such as databases, which would require selective replication of individual modified data fields within the entire database document.

Benefits of Using Novell Replication Services

Before we get into a technical discussion of how NRS works, it is helpful to review some of the benefits that such a service can offer.

Improved Data Availability. NRS offers easy dissemination of information across the WAN. Because information can be replicated to remote locations, users at those sites can have access to WAN information­even graphics, audio, and video files­at LAN speeds. Access to critical corporate resources will no longer be subject to outage and disruption due to WAN faults or performance constraints.

With important information replicated across an enterprise, users can more effectively share and collaborate on their work. Sales figures, engineering documents, software, and code changes can be accessed and acted upon more easily by key personnel.

In addition, web sites can be replicated for content consistency while providing improved remote access. Replicating web pages/sites across multiple servers helps in network and server load balancing.

Automatic Synchronization. Powerful synchronization algorithms provide real-time data consistency across multiple servers. There is no longer any need to spend time searching and tracking changes for replication or writing batch files or scripts. NRS detects changes made in any of the replicated directories and automatically replicates only those files that have changed, creating a fully synchronized file and data image at each server. This creates a highly scalable replication architecture. NRS supports two-way synchronization between any pair of servers and supports simultaneous one-to-many data flows for replicating files to all of the critical servers on your network.

Ease of Administration. NRS is administered by means of a "snap-in" to the standard NetWare Administrator (NWAdmin utility). The snap-in adds pages to the NDS Server object dialog. This provides a consistent interface with other NWAdmin tools, shortens the learning curve for administrators, and allows you to control NRS from any network client.

Once NRS is installed, replication is defined, and synchronization schedules have been set up, no further action is required by the user. The network administrator's subsequent involvement is generally confined to routine activities such as periodic inspection of logs. Any modifications made by the users to the data in the replication set are automatically synchronized to the defined servers.

Help with Managing Bandwidth. NRS has a powerful scheduling facility that can help smooth out WAN traffic. You can schedule replica synchronization to occur as often as every minute or during off-peak hours. Replication also puts information closer to the end-users, which reduces the need to constantly access the WAN and frees your WAN bandwidth for other critical applications.

Reduction in WAN Costs. Because NRS automates the replication process, companies can make more productive use of existing staff that are involved with distribution of information. The NRS WAN-optimized design minimizes WAN traffic. Better utilization of existing equipment, lines and connections means that purchases of expensive additional capacity can be postponed.

How Does Replication Work?

There are two main processes involved in providing replication services: replication and synchronization.

Replication is a process by which a file or group of files is copied from its original source to other servers within an organization's NDS tree. A file or document that has been distributed in this way using NRS is called a replica (not to be confused with an NDS replica, which is a copy of an NDS database partition stored on a NetWare 4 server). The original file is called a master. The server holding the original file is the master server, while the servers the file is copied to are known as replica servers. The process of replication allows users local to a replica server to obtain the documents more quickly, and optimizes the use of long-distance connections between servers.

Synchronization is the process by which the distributed files and documents are kept up-to-date, so that users on each server can be sure they are viewing the most recent version. When a master server and its replicas are synchronized, the contents of the transaction log for each replicated volume are used to identify those files that have changed, those that have been added, and those that have been removed. The most recent copies of each are then replicated to all other servers that have requested replication of those files.

What Happens During Replication? Novell Replication Services is implemented as a single NetWare Loadable Module (NLM) file named NRS.NLM. This module is loaded on each server to or from which you want to distribute information. During replication, NRS.NLM intercepts file system hierarchy updates on the servers and, using advanced replication technology, maintains a database on each master and replica server. The database records the state of the server's file system hierarchy and logs changes made to the file system. The log entries include:

  • Changes to file contents

  • Movement of files and directories from one directory or location to another

  • Addition of files and directories

  • Deletion of files and directories

These database entries identify the system hierarchy and files modified since the last synchronization, and therefore define those required for the next synchronization process.

The Reconciliation Process. The database is updated only while NRS.NLM is running. If the NLM is not running, the database entries are not being maintained. When the NLM does start, one of its initial tasks is to reconcile the contents of the database with the server's file system hierarchy. The task is performed by a tree-walk on the server and may carry an operational overhead depending on the volume of data so processed.

What Is Applied During Synchronization? The following changes to the replication set on master and replica server will be recognized and applied during synchronization:

  • Modification of file contents

  • Addition, deletion, renaming, and moving of files and directories

  • Modification of file and directory attributes

  • Modification of file rights

  • Name space entry changes (for DOS and long name space entries only)

During synchronization, the modification times of the master and replica versions of a file are set to the same value. The modification time of the parent directory, however, is set to the time at which synchronization of the file system object took place and will therefore be different on different servers.

Configuring a Replication Network

NRS allows network administrators to selectively replicate NetWare file system objects (volumes, directories, files, trustees, inherited rights filters, and name space entries) from one server to any number of other servers in the same NDS tree. You configure the server from which file system objects are to be replicated as a master server, and each server to which file system objects are to be replicated as a replica server. The group of servers comprising a single master server and the replica servers containing objects replicated from that master server is called a replication region.

A replica server can also be configured as a master server to its own set of replica servers, acting as a link server between two replication regions. This eliminates the need for direct replication from an existing source of information via the same WAN link to numerous other locations requiring the information. Instead, a single replication can occur via the WAN to a defined link server and then onward to the other replica servers.

The resulting hierarchy of replication servers is called a replication tree. A replication tree can range from a simple configuration with a single master server and one or more replica servers, to a more complex tree with link servers and multiple replication regions (see Figure 1).

Figure 1: A simple replication tree and a more complex tree consisting of multiple replication regions.

A replication tree includes all servers that are connected by replication­independently of what they replicate. In other words, if server B replicates files from server A and also replicates files to server C, servers A, B, and C are all part of the same replication tree even if C replicates an entirely different set of files than those replicated by B from A. You can have as many replication trees within a single NDS tree as are required.

Rules for Replication

There are a few simple rules to keep in mind when designing your replication hierarchy:

  1. All servers connected by replication must existwithin the same NDS tree.

    This ensures that file system trustee rights can be synchronized, since trustee rights reference User and Group objects in the NDS tree. For example, if only a single user has rights to change a replicated file, those rights are synchronized and other users on any server are there by prevented from updating the file.

  2. A single master server can support up to 50 direct replica servers.

    This does not constrain scalability because the replication set can be extended without limit through the use of link servers.

  3. No one server may replicate from more than one master.

    By this rule, the following replication tree scenarios would be invalid.(1) Master servers in New York and Tokyo both replicate to a server in London.(2) A master server in Paris replicates to link servers in Frankfurt and Zurich; the Frankfurt server in turn replicates to Rome, while the Zurich server replicates to Budapest. Files on the Zurich server which were not replicated from Paris are being replicated to Rome. (In this case, the files at Zurich which do not originate from Paris effectively assign the Zurich server to a master role. This could not be supported because Rome would then be replicating from both Frankfurt and Zurich.)

Replication Options

NRS provides a number of different replication options for selecting the data you want to replicate. These include:

  • Volume-to-volume or volume-to-directory replication

  • Excluding or masking items from replication

  • Maintaining consistent views of a hierarchy

  • One-way or two-way synchronization

Volume-to-Volume vs. Volume-to-Directory Replication NRS supports two methods of replication: volume-to-volume and volume-to-directory. Volume-to-volume replication allows you to replicate selected contents of a single disk volume to an identically named disk volume on another server. This is the default replication option, and allows the master and replica data to reside at the same hierarchical level within the volumes (for example, directory \WORKon volume APPS replicates to directory \WORK on the replica server's APPS volume). Users of the replicated file system objects will see the same directory structure from the root of the NetWare volume regardless of their location in the network or whether they map to the master server or one of its replica servers.

Volume-to-directory replication allows you to replicate selected contents of one or more disk volumes to a specific directory within a disk volume on another server. Because you are replicating from a volume to a directory, the master and replica data sets are at different hierarchical levels (for example, directory \WORKon volume APPS replicates to directory \REPLICA\APPS\WORK on the selected replica volume).


Note: Volume-to-volume replicationdoes not support replication of the SYS volume,but volume-to-directoryreplication does.

Figure 2 shows an example of volume-to-directory replication in graphical form.

Figure 2: In volume-to-directory replication, the master and replica data sets are at different levels in the file system hierarchy.


Caution: There are implications of each type of replication; you need to plan carefullyto identify the appropriate method for your data. See the Appendix, "Planning for Replication,"at the end of this AppNote for information onwhat you need to consider.

Including, Excluding, and Masking Items By default, when you define a volume or directory as an item for replication, all its subsidiary items (such as subdirectories, files, and attributes) are automatically assumed to be a part of that replication. If you do not want to include all subdirectories, you can specify which you do require and which you do not. Individual files cannot be replicated (unless only a single file exists in a subdirectory); you must replicate all of the items within a directory level, or none of those items.

Each directory can have one of the following replication attributes. If no attribute is defined for a particular directory, it inherits the attribute of its parent directory.


Included Directory

The directory and its contents are replicated from master to replica server. Changes to the contents of the directory on the replica server aresynchronized with the master, and vice versa.

Excluded Directory

The directory and its contents are not replicatedfrom master to replica server. An empty directory of the same name is createdon the replica server, but no modifications can be madeto this directory. Changes to the contents of the directory on the master server arenot synchronized.

Any files put into an excluded directory are treated as "clashes" andare moved to the recovery directory.

Masked Directory

The directory and its contents are not replicated frommaster to replica server. An empty directory of the same name is created on thereplica server, but, unlike an excluded directory, additions and modifications can be made to a masked directory. Changes to the contentsof the directory on the master server or replica server are not synchronized.

Files may be put into a masked directory, and will notbe treated as clashes.

WhatIs a Clash?

A clash is the result of more than one copy of replicated data beingupdated between synchronizations,or replicated/master data being moved, added to, or deleted from servers in an incompatible way. For example,if synchronization between server A (master) and server B (replica) occurs every day at 6:00 p.m., and the copies of a document on both A and Bare updated independently before this time, a clash will occur when synchronizationis attempted at 6:00. Similarly, a clash will occur if the document is relocated on A or deleted from B,after having been modified on the other server.

Some files and directories are never replicated.These "ignored"items are shown in the following table.


Items Being Replicated
IgnoredItems

Filecontents

NRSdatabase directory (NRSDB)

Fileand directory attributes

Fromthe volume root directory:NRSLOCAL directory_NETWARE directoryVOL@LOG.ERRTTS@LOG.ERRDELETED.SAV

Fileand directory trustee rights and inheritedrights filters

Iconsand the Network Trash Folder from the rootdirectory of volumes with the Macintosh namespace enabled

Note: Name space attributes are replicated for the LONG name space only. OS/2 extended attributes, Macintosh Resource Forks, and Macintosh and NFS name space attributes are not fully replicated. You should therefore avoid replicating OS/2, Macintosh,and NFS files.

Maintaining a Consistent View of a Replicated Hierarchy. You may be replicating only a subset of a file hierarchy­for example, a single subdirectory or a selection of top-level directories. Users on the replica server will be able to access only those directories that you have specifically included for replication. But, through the use of "name-only" directories, users are able to view a representation of the original directory structure.

Name-only directories on the replica server maintain a semblance of the master server's file hierarchy, only without replicating unwanted files. Each name-only directory represents the name and attributes of one directory that exists on the master server, but is excluded from replication. The users of the replica server cannot manipulate these directories by adding items to them or deleting them, but they can view without confusion the overall hierarchy as it would have appeared on the master server.

There are two types of name-only directories:

  • A stem name-only directory appears between the root of the volume and the included replicated directory. These directories preserve naming and inherited access rights, and are therefore important to ensure the same security rights are maintained on replicated files and directories.

  • A leaf name-only directory appears at the same level as an included replicated directory. Files and directories cannot be added to leaf name-only directories.

As an example of name-only directories, consider a master server volume which has the structure shown in Figure 3. In this example, only the OFFICE subdirectory is replicated.

Figure 3: Example directory structure on a master server.

The replica server will contain a stem name-only directory called APPS, with subdirectories OFFICE and LAPTOP. The OFFICE directory will contain replicated data, but LAPTOP will be a leaf name-only directory. The GAMES, TEST and WORK directories will also be name-only directories, but will not contain subdirectories because nothing at that level is replicated from those directories.

One-way or Two-way Synchronization. By default, changes to data on either the master server or its replica server are counted as valid changes and will be synchronized to the other server when synchronization occurs. In some cases, this may not be what you require. To allow only one-way synchronization (from master to replica), NRS has an option to protect the master server from updates by synchronization. If document updates are only done on the master server, you can set the protected master server attribute. This allows only one-way synchronization, so updates to the replica server are never synchronized on the master, but updates to the master server are always synchronized on the replicas.


Note: To configure a master server as a protected master server (thereby disablingtwo-way synchronization),select the "Prevent updatesfrom the replica servers"check box on the Replication panel when configuring the server for replication. Users of the replica serverswithout Supervisor rights are then not able to update. Users with Supervisor rights will be allowed to update the replicated files, although these updates will not be copied to the master server to overwrite its existingcontent. They will be treated as clashes on the replica server when synchronizationnext occurs. This setting can be changed at any time, if required, to allow the updatesto overwrite the master again. (The default option for master server protectionis off.)

Synchronization Tips

Scheduling Synchronization of Data Changes. Synchronization is the process by which changes made to replicated data are applied to the master and replica versions of that data, wherever the changes originate. NRS is designed to minimize use of long-distance WAN links by supporting scheduled synchronization of replicated file system objects. As a network administrator, you can initiate immediate manual synchronization between servers, or specify times at which synchronization should occur automatically, allowing you to take advantage of low-cost connection periods and adjust for international time zones.

Ensuring That All Files Are Synchronized. If a file is opened with Deny Read on a source server, or opened with Deny Write on a target server, the file cannot be synchronized until it is closed. If the file is one of a set of files (for example, part of an application) in which all files need to be synchronized at once, it is important that the file is synchronized even if the file is in use.

To work around this problem, you can set a per-server synchronization option to force users to close the relevant file(s) or risk being logged out of the server containing those files. You can thus ensure that all the relevant files are available to be synchronized. This option is intended for use when synchronizing changes at off-peak times (for example, when synchronization occurs at night) and thus should rarely affect users.

If a user has a file open that is required to be updated for synchronization, that user will receive the following NetWare broadcast message:

NRS forced logout in 45 seconds. Open file:filename

On receiving this message, the user can close and save the file within the indicated time period, and thus avoid being logged out of the server. Synchronization then continues as usual. If the file is not saved and closed within the specified time, the user will be logged out of the server containing the specified file and any unsaved changes to that file (and any other files from that server the user was working with) are lost. After the user is logged out, synchronization continues as usual.

The Force User Logout attribute is set for each server when you define the synchronization schedules. Setting this attribute from a master server means that only users of the master server will receive the broadcast if they have a file open. You may need to set the attribute for each replica server as well. If users only have access to the replica servers, you may need to set this attribute for the replica servers only. This would ensure that updates from the master server are always synchronized with the replica servers, even if the users have files open on the replicas.

Synchronization will be retried once per periodic synchronization if the following situations prevail:

  • The Force User Logout option is not used and the file fails to synchronize because it is in use; and

  • The file was open on the source server.

If the file was open on the target server, the file will be copied to a temporary file on the target server and will replace the target file when the target file is closed. If the target file has itself been modified, a clash will occur, resulting in no loss of data.

A Simple Example of Data Replication

As an example of data replication, consider this scenario: Company ABC has its main office in New York City and sales offices in various locations elsewhere in the United States. It also has sales offices in London, Paris, and Frankfurt, which often require documents originating in New York. Some of these documents are updated by the European offices. The head office in New York needs to distribute the required documents to London, Paris, and Frankfurt, and the European sites need to send New York their updates to those documents. However, ABC's transatlantic WAN link is slow and expensive, and the transfer of updates is complicated by the different time zones. The result is that the European sales staff are often left waiting for the information they need, while the head office in New York is often using information that is out-of-date.

Using NRS, ABC's network administrator can configure New York as a master server, London as a link server, and Paris and Frankfurt as replica servers. The sales offices in Los Angeles and Dallas can be configured as replica servers of the New York master server. Documents are selectively replicated from New York to all the offices that require them; but those destined for Paris and Frankfurt are sent via London to minimize use of the transatlantic WAN link.

Synchronization also takes place via London and can be scheduled to overcome the problems presented by international time zones; document updates made in Frankfurt, for example, can be applied to the London server at 19:00 GMT every evening, while synchronization can be scheduled between London and New York for 02:00 GMT (21:00 in New York). Synchronization between New York and the other US offices can be scheduled to take place after this time, so that they also receive the latest documents from Europe.

Figure 4 shows a diagrammatic representation of the replication tree for this solution, called a replication map.

Figure 4: Sample replication map for company ABC.


Note: It is strongly recommended that you create a map of your proposed replicationtree before youstart to install and use NRS. Doing so will enable you to identify the most appropriate hierarchy of master, replica and link servers.

For Additional Information

To provide detailed information about the software and how to use it, NRS includes the following sources of product information:

  • Novell Replication Services Administrator's Guide(part no. 100-003976-001 A)

  • Online help

  • Cue cards

General information about NRS and other Novell products can be found on Novell's web site at http://www.novell.com.

The Administrator's Guide

The Administrator's Guide should provide all the information you need to:

  • Understand what replication is, and how you can use it

  • Plan your replication requirements

  • Configure your environment for replication

  • Enable and run NRS

  • Identify when a problem has occurred, and what to do to resolve it

Online Help

NRS has a fully-contextual online help facility, which is linked to its NWAdmin snap-in interface. This help contains:

  • Explanations of NRS concepts and features

  • Step-by-step guidance on using NRS

  • Specific help for using dialogs, and items within dialogs

  • Explanations of error messages and a description of how to interpret the diagnostics log

  • Definitions of terms used in the help and interface

You can display help from a NRS dialog, message, or snap-in panel by pressing F1 or selecting the Help button when one is shown. The Help topic displayed describes the dialog, message, or panel that you are viewing.

Buttons on the Help Toolbar Use the button bar at the top of the Help window to display other Help information if required. The standard Help buttons perform routine Help tasks but there are the following additional ones present also. The Messages button takes you to the topics that help you interpret NRS messages appearing during execution. The Glossary button opens a separate Help window providing an online glossary of terms similar to that at the end of the Administrator's Guide.

Cue Cards

Cue cards are provided throughout the NRS interface. They provide assistance with specific dialogs, or specific tasks related to the dialog or item you have just displayed. You can choose to display cue cards as you work, or not to display them unless you specifically request them. To switch the display of cue cards on or off, select NRS from the NWAdmin Tools menu. From a displayed cue card, you can select to disable this specific cue card or all cue cards.

Appendix - Planning for Replication

Before you start setting up replication between servers, you should plan carefully what you need to replicate and how your replication map will look. Bear in mind the following points:

  • All servers in the proposed replication scheme must exist within the same NDS tree.

  • Each replica server can only have one master server (which may be a link server).

  • Up to 50 direct replica servers are supported from a master server. This does not constrain scalability since the replication set can be extended without limit through the use of link servers.

  • Use link servers to minimize the requirement for replicating the same files over the same WAN link. For volume-to-directory replication only, the path names are extended each time the replicated data moves from one server to another; long chains of link servers may result in excessively long path names.

  • Ensure that you have Supervisor rights to all replica servers and Browse, Read and Write rights to the NDS tree to administer replication and synchronization using NWAdmin.

  • Files on the replica server will have the same access rights as on the master server. However, for secure data, it is important to ensure that the replica servers are as physically secure as the master server. If adequate physical security cannot be guaranteed, secure data should not be replicated.

  • Data on a master server may be protected from accidental update on a replica if required.

  • When protected masters are not used, use access rights in replicated directories to minimize the chance of multiple versions of a file being updated at one time. If this happens, a synchronization clash will occur.

  • Avoid replicating files which are updated concurrently at multiple sites.

  • Read-only media are not supported as either replication sources or targets.

  • Do not replicate files used by Macintosh, OS/2, and NFS users if the Macintosh or NFS name space is enabled on the server they are accessing.

  • If non-default configuration of the following server parameters is chosen for the master server, the same options should be configured on all replica servers: enable long name space support, and choose the same code page.

  • Maximum subdirectory tree depth should be the same on all volumes supporting volume-to-volume replication.

  • If volume-to-directory replication is used, maximum subdirectory tree depth should be increased by the depth of the replica directory plus one.

  • NRS requires 20MB of disk space and 2MB of RAM on each master and replica server that has NRS.NLM loaded.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates