Global Backup with Storage Management Services
Articles and Tips: article
Software Engineer
Developer Support
01 Feb 1997
Provides an overview of Storage Management Services (SMS) from a development perspective. Discusses the overall architecture of SMS, the components shipped with the SDK, and how to write a backup engine.
Introduction
Management has asked you to create a new backup strategy for the corporate network. The requirements include backing up data on several heterogeneous nodes, including NetWare 4.1 and 3.12 servers, UNIX servers, and MS-DOS, OS/2, and Macintosh clients. Also, you need to provide a way to back up and restore all systems from one central point, to minimize the need for additional backup devices. Finally, you need to provide the capability to selectively backup data, such as backing up only those files which have been modified in the last week.
Novell Storage Management Services (SMS) provides a complete set of software modules, API sets, and associated documentation that will allow you to create a complete backup solution that addresses each of these requirements. This article provides an overview of SMS from a development perspective-we'll look at the overall architecture of SMS, the components shipped with the SDK, and how to write a backup engine.
SMS Architecture
SMS was designed to provide reliable backup and restoration across networks, while being independent of any particular data format. This is accomplished via a standard API set which is cross platform: the set of function calls needed to back up data residing on a DOS disk are the same as those required to back up an OS/2 system. To understand how this is accomplished, let's look at SMS from an architectural standpoint.
Figure 1 shows the software components of SMS. The top module is the Storage ManagementEngine, or SME. The SME component interfaces directly with the backup administrator, presenting the various backup/restore options, then acting upon the administrator's selections.
Figure 1:The software components of SMS.
The SME interfaces indirectly with the backup target(s) as well as the backup storage mechanism, through the SMDR. It is important to note that most third-party development under SMS revolves around developing SMEs. For example, a developer may want to develop a backup engine which uses high speed WORM (Write Once, Read Many) drives as the repository.
Or, an engine could be created which would oversee all aspects of disaster recovery, such as monitoring backup frequency, media verification, etc.
The Storage Management Data Requester, or SMDR, handles communications between the SME and the backup targets. Additionally it communicates with the Storage Device Interface (SMSDI) to store and retrieve backup data.
The Target Service Agent, or TSA, is a software module that interacts with the SME to back up or restore a platform's data set. There are specific TSAs for NetWare 3.x, NetWare 4.x, MS-DOS and Windows, Macintosh, OS/2, and UNIX. Each of these TSAs knows how to scan, read, and write the data on the respective platform. Note that the target may be a NetWare volume, a file system, or even a database. The TSA prepares the backup target for backup or restoration, according to the directives of the SME.
Since knowledge of the specific target file system is confined to the TSA, the TSA can package data from the target and present it to the backup engine in a generic format. Thus, several different types of TSAs can interact with one SME. Also, support for a new release of an operating system can be added via creating a new TSA-no other changes to SMS are needed.
Novell provides SMS TSAs for NetWare 3.x and 4.x servers, Novell Directory Services, NetWare SQL, DOS and MS Windows, OS/2, Macintosh, and UnixWare clients.
Storage Management Services Device Interface, or SMSDI, provides a common view of all media devices, such as tapes, WORM drives, etc., to SMS. It handles all device-specific requirements for SMS, as well as enforcing SIDF rules that apply to the media.
Not shown in figure 1 is System Independent Data Format, or SIDF. SIDF is a data tagging and logical media format which is used throughout SMS. It provides a common media format that can be used on any type of media (disks, tapes, WORM drives, etc.). Any backup application which is SIDF compliant can share data with other SIDF-compliant products.
Together, the components of this architecture provide a flexible framework for network storage management that is independent of any particular hardware or software platform, and can be tailored to meet the needs of a particular environment.
Components of the SDK
The SMS development environment is shipped with Novell's SDK CD-ROM. By default, the SMS directory is installed under SDKCDX/SMS, where 'X' is a digit which reflects the release version of the SDK.
Under the SMS directory are a number of subdirectories which contain platform-specific executables. For example, the WIN95 directory contains W95TSA.EXE, the TSA specifically written for Windows 95. These executables are provided so that a developer can test his or her SMS application against several platforms.
The other directories under SMS contain files, libraries, and sample code for developing SMS applications. Additionally, several online documents are provided which describe the SMS architecture and API sets.
The sample directory contains source code for a simple backup engine, as well as a "get started" document which describes the steps needed to build and execute the sample application. Under the sample directory are two subdirectories:
NLM-contains sample code and configuration files for building a backup engine as a NetWare Loadable Module
DOS-contains the same sample code, but different configuration files, for creating a DOS-based backup engine.
DE1.C and DE2.C
The source files DE1.C and DE2.C contain sample C code which can be compiled and linked to produce a simple backup engine-this code is a good starting point for anyone looking to develop to SMS because it illustrates the various steps needed to back up or restore data on a NetWare network. DE2.C defines several internal routines referenced by DE1.C. The general flow of the code is as follows: list the SMDR connections, list the TSA connection(s), connect to a specific SMDR and TSA, create a selection list (that is, a list of items to be backed up, such as files, directories, or NDS objects), then perform the backup. The restore menu allows a user to restore all or selected portions of the backed-up data. For an in-depth discussion of the various menu selections within the DE code, refer to the Envoy document GETSTART.EVY under SDKCD9/SMS/SAMPLE.
Let's examine the minimum functionality the engine must provide. It needs to connect to both an SMDR and one or more TSAs to be backed up or restored. Next, the engine needs to discover what backup device will be used. Then it needs to determine if the user wishes to perform a backup or restore. For a backup, the engine must determine what files and directories should be transferred to media. Once the target set has been created, the engine may label the backup media with information specific to this backup session for future reference, then begin transferring data.
If the user selects restore, the engine needs to know what to restore from the backup media-either all of the set or specific files and directories within the set.
These are the basic tasks which an engine must support. A developer can provide additional functionality in several ways. For example, an intuitive user interface should be written to allow the user to navigate quickly through the process, or to set up unattended backup for off hours. Or the engine can be tailored to a specific environment, such as one containing a large number of UNIX systems. SMS provides the requisite plumbing to connect, back up, and restore local and remote systems, so the developer can focus on adding value to the engine.
Process Flow Within the Engine
The first set of tasks the engine handles are selecting devices and media. The engine calls NWSMList SMDRs() to list SMDRs running on the network-the administrator then selects one to use during the session. On a large network it is normal to have a large number of SMDRs running on different servers. Next, the engine invokes NWSMListTSAs()-this API displays the TSAs that are connected to the selected SMDR (note that when a TSA is invoked, it must specify a server running an SMDR to connect to-this forms the basis for communication between the engine, the SMDR, and the locally running or remote TSA). The engine calls NWSMConnectToTSA() to complete the selection of devices.
After connecting to a TSA, the engine calls NWSMTSListTargetServices() to discover what services it provides. For example, a TSA may support multiple target services, such as backing up different types of file systems. An engine may also call NWSMTSGetTargetServiceType() to determine what service type the TSA provides ("NetWare," "DOS," etc.), as well as version number information. This means an engine can determine unique characteristics of the TSA and tailor the backup and restore process accordingly, if desired. Stated another way, since a UNIX file system has a different format than a NetWare volume, the engine may choose to exploit this via presenting backup information specific to the particular format, thus adding value to the engine. Or, the engine can treat each TSA generically, making no distinction between different platforms. This flexibility allows SMS developers to tailor their engines to specific environments, as well as being able to address different levels of backup.
After the selection process is complete, the next task is to gather information specific to the backup session. To begin this process, the engine may choose to call NWSMTSGetUnsupportedOptions() to determine what backup-specific options the TSA does not support. For example, a TSA may not keep track of files skipped during a backup session, which means that the engine must not assume that it does. Next, the engine prepares to do a scan-the scan is a process in which several related factors are examined to create a data set, which represents the information the user wants to back up.
Once the engine determines what options the TSA doesn't support, it can call NWSMTSGetOpenModeOptionString() to find out what TSA-specific open mode options are available, or more specifically, what parts of the data set can be opened or closed. For example, an administrator can back up all parts of a data set with the exception of trustee information, if the TSA is of service type "NetWare." The option(s) returned by this API can then be passed to NWSMTSOpenDataSetForBackup() to control which parts of the data set will be opened.
After the engine has collected the open mode options, the next task is to build the scan control list, which consists of at least two sublists: a date and time list and scan type list. The date and time list tracks when a data set has been accessed, archived, created, etc. The scan type is an action which essentially filters in or out parts of the data set. For example, a scan type may be specified to exclude or include archived parents or children during the scan. The scan control information is kept in a structure of type NWSM_SCAN_CONTROL.
A further refinement of the data set can be specified by using selection options, which name the data sets to backup. That is, for each selected type there is an associated path identifying the location of the data set. A parent and a child are two examples of selected types. The path to the data set is combined with the selected type to form a linked pair entry within a selection list. This information is stored in structure type NWSM_SELECTION_LIST.
Thus, the engine can:
determine what capabilities the TSA supports
specify what parts of the data set to open
control the search through the data set
include/exclude specific parts of the data set
when preparing to backup.
After the engine has gathered the prerequisite backup specification, it then needs to account for unique characteristics of the entity to backup, such as upper or lowercase naming and path separator conventions (forward slash or back slash).
Once all of the requisite selection and scan lists are complete, the backup can begin. The actual backup process consists of an outer loop which retrieves a data set and an inner loop which reads all of the data from the data set. Processing is complete when the outer loop has no more data sets to fetch. The engine then releases any connections it has opened to Target Services and TSAs, via NWSMTSReleaseTargetService() and NWSMTSReleaseTSA()respectively, and frees memory.
The steps necessary for restoring data are similar to backup: find the TSA's unsupported options, determine open-mode options, build the selection list, and gather whatever other information is needed to support the restore, such as unique characteristics of the volume or file system. The differences are that with a restoration the scan control options are not used, and restore-specific open mode options are available. For example, a data set can be opened with NWSM_UPDATE_DATA_SET specified, which tells the restore to update the data on the Target Service only if the copy on the media is newer than the Target Service's copy. Having backup and restore-specific open modes means the engine has more control and granularity in handling the data sets.
Other Types of TSAs
Thus far a Target Service has been depicted as a volume or a file system. However, it can also be a database, such as Novell Directory Services (NDS). NDS has its own specific TSA which handles all interaction between the engine and the data sets comprising NDS. This TSA accounts for the unique requirements in backing up and restoring a dynamic distributed database while at the same time providing a uniform interface for the engine.
Conclusion
SMS provides a scalable architecture uniquely suited for data migration over large networks. It is extensible in that new TSAs can be created to support new target services, such as a database or a network file system. It preserves data independence by utilizing platform-neutral formats, allowing heterogeneous platforms to exchange data without additional overhead.
The API set provides a broad range of data set operations, allowing a consistent and familiar view of dissimilar entities. The developer can focus on adding value to the engine, rather than fighting details related to specific platforms.
SMS enjoys significant name recognition in the storage management industry today. Several vendors provide backup solutions based on SMS, such as IBM, Seagate, and Hewlett-Packard. It is also used for development of custom in-house backup and data-migration tasks.
As networks become more complex, the uses for this flexible technology will increase. Large-scale product installations and upgrades, data retrieval and distribution, and media management are some of the areas which can benefit from SMS. Combined with its scalability, transport and device independence, and granularity at the data-set level, SMS is unsurpassed in its ability to provide storage-management solutions.
Glossary of SMS Terms
|
Term
|
Definition
|
|
Child |
An object that cannot have subordinates. A file is a child object. |
|
Data Set |
A logical collection of data. The backup administrator defines a particular data set for backupor restoration. |
|
Parent |
An object that has (or can have) subordinate data sets. For example, directories and subdirectoriesare parents. |
|
System Independent Data Format (SIDF) |
An international standard defining a common format for sharing data on storage devices such as tape. SIDF allows SMS to deal with the media and Target Services' data sets generically. |
|
Storage Management Data Requester (SMDR) |
The SMDR directs communication between the SME and the TSA(s). Additionally, the SMDR interfaceswith the storage device interface on behalf of the SME. |
|
Storage Management Engine (SME) |
The engine is the central component within an SMS backup system. The SME interfaces with the backup administrator, and forwards requests to both the TSAs and to the device driver controlling the backup media. The SME is the value-added component of SMS, that is, the part created by the developer. |
|
Target Service Agent (TSA) |
A specific software module designed to facilitate backup and restoration of a particular target,such as a NetWare volume, a UNIX file sytem, etc. The TSA communicates with the SME, and understands specific details of the target's data, such as file layout and directory structure. Thus, even though there are different typesof TSAs for different targets, each TSA packages its data and sends it to the SME in a standard format. |
Bibliography
Fairbanks, Michael; Neff, Ken. "Backing Up and Restoring Novell Directory Services in NetWare 4.11" (Novell Application Notes, October 1996).
Mimmack, Peter. "Novell's Storage Management Services" (BrainShare Presentation, 1995).
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.