Novell is now a part of Micro Focus

Enhancements to Storage Management Services (SMS) in NetWare 5

Articles and Tips: article

SUDHIR SUBBARAO
Software Engineer
India Development Centre

01 Nov 1998


Lists enhancements to SMS for NetWare 5 and presents an overview of what has changed in the new release of the SMS modules.

Introduction

Storage Management Services (SMS) is a set of modules and APIs which provide a platform-independent strategy for data management and migration across networks. With SMS, developers create backup engines which are used to back up and restore data, not only on NetWare platforms, but all Microsoft platforms, Macintosh, OS/2, and Unix.

There are actually two parts to SMS: The set of modules and libraries shipped with Novell's SDK and the Enhanced SBACKUP backup utility which provides basic backup services for NetWare users.

This AppNote gives an update of all the enhancements that were made for NetWare 5 networks and presents an overview of what has changed in the new release of the SMS modules. A follow-up article will detail the features and procedures of the Enhanced SBACKUP Utility.

For specific details, consult the documentation and readme files that come with your NetWare operating system software and with your particular backup application.

This AppNote assumes the reader is familiar with the basics of SMS and NDS. A good reference to SMS can be found in an article "Storage Management Services", NetWare Connection, September 1996 (visit http://www.nwconnection.com).

The Storage Management Data Requester

SMS architecture is open to allow for storage management services (through Target Service Agents and other means) to be built on heterogeneous platforms. More importantly, SMS allows for these services to operate in a heterogeneous environment. In a heterogeneous environment, one may have a variety of networks, namely IPX-only or TCP/IP-only or even a mixed IPX-IP environment where some services would be available from an IP segment while others can be accessed only through an IPX segment. In the earlier versions of NetWare (4.x and below), Novell focused only on IPX/SPX networks. But now, Novell has adopted a two-step approach to allow existing LANs and WANs to participate in the TCP/IP-based Internet environment:

  1. NetWare/IP, which allows IPX services to work in a TCP/IP environment while maintaining full backwards compatibility with IPX-based applications and networks.

  2. NetWare in a Pure-IP environment, which makes use of TCP and UDP at both NCP and NDS application-layer without the need for IPX encapsulation.

The Storage Management Data Requester (SMDR) is a communication module in the SMS architecture. It provides transparent access to SMS services in an intranet environment, as shown in Figures 1 and 2. It exports a set of SMS APIs which can be used to access local or remote SMS services transparently. These APIs are used by backup/restore software from Novell (SBACKUP) and many other third-party vendors.

Figure 1: SMDR provides transparent access to SMS services.

Figure 2: SMDR provides transparent access to SMS Services

In order to propagate SMS onto heterogeneous environments, SMS now comes with what is called the Portable SMDR. As the name suggests, SMDR is now portable, in that it provides an interface whereby the functionality of SMDR can be extended to incorporate more and more services across heterogeneous platforms. Also, the Portable SMDR is protocol independent and uses NDS for name resolution.

In this section, we address the name resolution mechanism with SMDR in an IPX/SPX environment, a NetWare/IP environment, and a Pure-IP environment. We also examine the backward compatibility of Portable SMDR.

SMDR in an IPX/SPX Environment

In the earlier versions of NetWare (4.x and below), Novell focused only on IPX/SPX networks. In the IPX architecture, Service Advertising Protocol (SAP) was available to take care of initial service discovery. Services could advertise their presence on the network and populate the bindery tables on every server. Clients could either interrogate the tables or listen to the wire for SAP traffic.

The initial SMDR implementation was designed specifically for the NetWare server-to-server communication. It has a built-in dependency on SAP to locate remote services and assumes that the only available communication protocol is SPX (via TLI). In such a scenario, the NetWare SMDR poses some overhead.

The NetWare SMDR makes extensive use of the unprotected-process nature of the NLM environment to exchange information between processes (or threads). These dependencies are moot as long as all of the important data resides on a NetWare server, and the network is small enough that SAP traffic is not an issue. However, they do make it costly to incorporate client workstations into the NetWare backup and restore model (especially when the associated SAP overhead is considered).

SMDR in a NetWare/IP Environment

To allow for a gradual deployment of Internet technology into existing IPX-based networks, Novell introduced the NetWare/IP technology in 1993. NetWare/IP takes care of mixed IPX and IP networks. Its architecture allows for existing IPX applications to exist in a TCP/IP environment through IPX encapsulation, as shown in Figure 3.

Figure 3: TCP/IP communication through IPX encapsulation.

This means that the IPX information is captured and placed inside an IP packet and sent through the TCP/IP stack and onto the wire. From an SMS point of view, the existing SMDR which was written for IPX/SPX networks would continue to work in a TCP/IP environment without any modifications.

For name resolution, SMDR used SAP in order to advertise and locate storage management services in IPX-only environments. But in a TCP/IP environment, the DNS needs to be used for this purpose. To enable this, Novell created DSS (Domain SAP/RIP Services) to provide service addresses for NetWare/IP. From an SMS point of view, when the SMDR would SAP in order to broadcast its services, the DSS would capture the SAP and place all the SMDR service addresses in the DSS database. For locating services, NetWare/IP would then query the local DSS centralized server which provides the necessary information on available services. This is illustrated in Figure 4.

Figure 4: DSS provides service address information in a NetWare/IP environment.

SMDR in a Pure IP Environment

Novell's goal in moving towards Pure IP is to provide NetWare services over the Internet. The NetWare 5 operating system is designed to remove all dependencies on IPX, while the NetWare Core Protocols (NCP) are modified to use the TCP and UPD transports.

In a Pure IP environment, TCP/IP protocol stack can be concurrently loaded along with the IPX/SPX stack. This allows services to exist in either IP-only, IPX-only, or both IPX and IP environments.

Since NetWare 5 must be able to co-exist with IPX/SPX-based versions of NetWare and existing IPX/SPX applications, NetWare 5 on Pure IP implements Service Location Protocol (SLP). SLP is an Internet Draft Standard for TCP/IP. Novell has implemented a scheme similar to NetWare/IP which it calls the Compatibility Mode Driver (CMD) in NetWare 5 to allow for support of IPX/SPX applications in a Pure IP environment. Unlike NetWare/IP, however, CMD stores IPX SAP information in SLP. There is no NetWare/IP DSS in the Pure IP environment.

In a Pure IP environment, the best way for applications and services to obtain naming services is through NDS. The clients discover NDS either by SLP requests or by DHCP (Dynamic Host Configuration Protocol). Figure 5 illustrates how this is accomplished via DHCP.

Figure 5: Services exchange information in a Pure IP environment.

Also, since the NCP verb engine has been separated from the IPX transport, applications making an NCP request need not know what protocol is servicing the request (see Figure 6).

Figure 6: The co-existance of both legacy IPX and newly written applications in Pure IP environment.

In order to leverage this new Pure IP technology, Novell has enhanced the SMDR to provide support for IP. The SMDR is now called Portable SMDR (PSMDR).

Portable SMDR and How It Works

Portable SMDR is an effort to propagate the SMS architecture across an intranet environment. The SMDR has been made generic so that developers can build SMDRs on various platforms, yet provide the same functionality.

The primary purpose of Portable SMDR is to provide a simple development system that minimizes development time, allows developers to focus on the work at hand, and addresses only platform-specific issues. This has been accomplished by re-structuring and designing the SMDR using Object Oriented Programming principles and encapsulating these issues into classes. The goals of Portable SMDR are to:

  • Propagate SMS on heterogeneous networks.

  • Locate other SMDRs through name brokering service (like NDS) so that Portable SMDR for NetWare will not use SAP by default.

  • Provide extensibility or ease of porting.

  • Provide protocol independence so that TCP/IP, IPX/SPX, and any other protocol can be easily enabled.

  • Provide fully functional backward compatibility.

Figure 7 depicts the results of the above goals being achieved.

Figure 7: Illustration fo Portable SMDR and NDS providing a complete view of all network services.

As mentioned earlier, NDS will provide a single, logical network-wide view of all services. Each of the NetWare servers, as well as the client workstations, has the Portable SMDR component. The Portable SMDR then registers all its known services with NDS. Figure 8 shows the services and protocols as plug-ins which can be dynamically bound without having to recompile the Portable SMDR.

Figure 8: Servies and protocols as plug-ins without recompiling the Portable SMDR.

The Portable SMDR's object-oriented design consists of platform-independent concrete classes and platform-dependent abstract classes. Porting efforts involve concrete representation of platform-dependent abstract classes. Therefore, building an SMDR requires the developer to implement the applicable abstract classes and the concrete classes together with the glue to build an SMDR module (for example, executable, library, and so on).

Novell plans to provide a code that is common to all SMDRs called "common code" SMDR. Some of the advantages of Portable SMDR include:

  • The design of Portable SMDR is extremely portable. Porting SMDR on a new platform requires very little effort.

  • When SMDR is ported to other platforms, an engine can be developed on that platform. Thus, the storage solution will not be NetWare-centric but will address the needs of enterprise-wide backup.

  • TSAs can be loaded whenever required, optimizing memory usage.

  • SMS is IP-enabled, and allows any protocol to be added without much effort.

  • Portable SMDR registers itself with NDS. It doesn't need to use SAP, reducing network traffic.

  • If SMDRs are available on different platforms, existing proxy services such as TSAPROXY are not required.

In order to achieve these goals, Portable SMDR is organized so that:

  • Code that is common (portable) to all SMDRs is organized into classes. The common code classes are provided by Novell and generally will not need to be modified, forming the base for SMDR development.

  • Code with platform dependencies are organized into classes. The platform-specific code is called by the common code to perform OS and protocol-specific tasks.

  • Code with service dependencies are organized into classes. Services are modeled as dynamic libraries that are loaded on demand at run time. The client accesses a service by dynamically binding the service's constructor to itself (the client) and invoking it.

Backward Compatibility of PSMDR

The Portable SMDR (PSMDR) needs to be backward compatible with old SMDR and other SMS modules. Backward compatibility is provided by the wrapper functions. The SMDR interface is classified into following: PSMDR-SMDR, SMDR-PSMDR, SMDR-TSA, SMDR-SMSDI, SME-PSMDR. The PSMDR is modified to accommodate some changes in the semantics of connection and authentication.

The current implementation of PSMDR does not include Service, Service Requester, and Service Responder. These classes are implemented to provide interface with services like TSA and SMSDI. These classes are specific to service type; however, virtual methods can be overridden for specific service.

Novell Storage Services and SMS

Novell Storage Services (NSS) changes the way we look at backup. It provides two basic distributed services: the distributed naming service and the distributed storage service. NSS helps build an intelligent data storage and protection service that controls the actual placement of data on storage media. The distributed naming system provides a single, logical network-wide view of all names, file, and directories. The naming service looks at the entire network and creates a "snapshot" physical view of the network"a symbolic view that is equivalent to what is physically there.

In Novell's strategy to provide smart network services, NetWare backup and storage services have very important roles to play. Just as storage (file services) is a fundamental part of the network fabric, the management of storage in backup, archival, disaster recovery, and online access is just as important. As more and more information is generated, it becomes critical for users and network managers to retain and store information over an extended amount of time.

As a first step towards a smart global network, SMS has introduced the TSA for the NetWare 5 File Systems. NetWare 5 has two file systems: the legacy "NetWare Classic" file system and the Novell Storage Services. Both of these file systems can co-exist and act to provide a migration path for users to move from the legacy file system to NSS. In this section, we see how the NetWare Server TSA has been enhanced in order to support some of the features that NSS provides.

Backing Up Files Larger Than 4GB

With NetWare 5, a single Server TSA now caters to both the Legacy and the NSS volumes. The name of Server TSA module that ships with NetWare 5 is TSA500.NLM (the "500" reflects the version of NetWare, 5.00). This new Server TSA is a derivation of the existing Server TSA TSA4xx.NLM.

In earlier versions of Server TSA, with the existing interfaces that are available to the Legacy Netware File System, the maximum size of files that could be backed up was about 4GB. Since NSS has overcome this limitation because of the 64-bit interfaces, it is capable of storing files greater than 4GB. The new NetWare Server TSA makes use of these interfaces and thus can back up files larger than 4GB from NSS volumes.

The API interface for the Server TSA has remained unaffected except for one small change that will affect the Backup Engines. Engines make use of the NWSM_SCAN_INFORMATION structure in order to obtain the size of the files being backed up. The size field in the structure is a 32-bit value. In order to represent the 64-bit value, we make use of the "OtherInformation" field to hold the other 32-bit value. This is a temporary solution until a new API for the NWSMTSScanDataSetBegin and NWSMTSScanNextDataSet is introduced where the ScanInformation structure would be extended to accommodate the 64-bit values.

Efficient Handling of Sparse Files

The current method of storing large sparse files up to 2^64 bytes (supported by file systems like NSS, NTFS, and others) in SIDF is inefficient, as it adopted a simple strategy for representing sparse files that worked well when they were limited to 32-bit file size. The existing SIDF implementation (a bitmap with each bit representing a block) does not scale well when a large file is very sparse.

The new addition to SIDF is an Extended Block Map and an Extended Sparse Stream format to handle massively sparse files. Existing SIDF implementations can continue to use the existing stream formats and are not required to understand the Extended Sparse format, as long as the TSAs correctly refuse to restore extended sparse data stream format.

Current Representation of Streams

In SIDF, a stream representation consists of the Stream Header Field Table that contains the metadata of the stream, the Stream Data that contains the actual data, and the Stream Trailer Field Table (see Figure 9).

Figure 9: Illustration fo the extended stream image.

One overhead of this method is that if the file is sparse and very large, the representation of these files in SIDF results in very large block maps containing very little information. As an example, if we consider a 4TB (tera bytes) sparse file containing only 1KB of relevant information, this method would require to store 256 megabytes as Block_map. For this reason, a new addition in SIDF has been made that represents massively sparse files more efficiently.

For Sparse files, the Stream Header Field Table now contains a new Extended sparse stream format (stream format 5) and the following additional field: Extended Block Map. It is optional but contains iterations of the following two fields:

  • Start Block: The offset from the first block.

  • Block Extent: The span of the information in blocks (of Block Size) from Start Block.

The extended sparse format specifies its Expanded Stream Size, Block Size, and Extended Block Map in its stream header. The term expanded stream image and Expanded Stream Size are the same as in the Block-mapped Stream formats specification. Extended Block Map is a map containing pairs of Start Block and Block Extent to represent the actual data only.

Start Block specifies the starting block from the beginning of stream.

Block Extent indicates the number of contiguous blocks from Start Block.

Start Block and Block Extent is 8 bytes long.

Considering the 4TB example discussed earlier and using the new representation, the Extended Block Map representation would be:

DATA LENGTH = 2 START BLOCK = 10 BLOCK EXTENT = 1

This requires 16 bytes for Start Block and Block Extent and 8 bytes for Data Length and 1 byte for FID. It requires only 25 bytes to represent sparse stream information (assuming the Extended Block Map is 1 byte FID).

New Target Service Agents

The following sections outline the new Target Service Agents (TSAs) included in NetWare 5 SMS.

DOS Partition TSA

Novell SMS introduces a TSA that facilitates backup and restore of the DOS Partition on a NetWare server. The name of the module is TSADOSP.NLM. The TSADOSP ships with NetWare 5 and also as part of the SDK CD release 13. Figure 10 shows where the DOS Partition TSA fits into the SMS architecture.

Figure 10: Illustration of where the DOS Partition TSA fits into SMS architecture.

TSADOSP.NLM needs to run on the NetWare server whose DOS partition needs to be backed up or restored. The single TSADOSP.NLM runs on all 5.0, 4.x, and 3.x versions of NetWare. For authentication purposes, on 4.x and 5.0 servers, it dynamically loads the DSAPI.NLM and performs a DS Login, whereas on 3.x servers it does a bindery login. The TSADOSP.NLM functions similar to the NetWare File System TSA (TSA4xx.NLM) and can handle multiple connections concurrently.

The DOS Partition TSA makes use of the NetWare DOS Partition APIs in order to make DOS calls. The DOS calls are implemented as Real Mode calls. In effect, when the DOS calls are made, the NetWare server temporarily gets suspended in order to service those calls. Hence, when many concurrent connections are made to access the DOS Partition, performance of the NetWare server may be slowed. If there are only a few connections, such a performance hit would hardly be felt and could be safely used.

One of the areas where the DOS Partition TSA could be used is in the process of disaster recovery. The DOS Partition TSA must be used with utmost care especially during the restore where there is a possibility of overwriting the files that pertain to the NetWare server. Keeping this in mind, Novell has restricted the usage of the DOS Partition TSA in the sense that only administrators are authorized to perform backup and restore operations on the DOS partition.

The behavior of the DOS Partition TSA on SFT III machines is an interesting situation. SFT III machines only have the NetWare partitions mirrored and have two independent DOS partitions. In the case when both the SFT III machines are up and running, the TSADOSP of the master server is active and thus accesses the DOS partition of that local server. In the case of a failure of one of the servers, the TSADOSP of the mirrored NetWare server takes over. Since the DOS partition of this server is entirely different from that of the failed server, the ability of backup and restore to continue is invalid.

By default, the DOS partition TSA excludes a set of files that resides on the DOS partition. These are IO.SYS and MSDOS.SYS. These files are either system files or read-only and do not need to be backed up.

With NetWare 5, NSS users also have the option of mounting DOS (FAT) hard disk partitions using the FAT.LSS. The Server TSA (TSA500.NLM) could then be used for backing up and restoring the contents of that volume. However, usage of this option for backup and restore has not been tested and users use it at their own risk.

Workstation TSAs

In a network backup, the SMS architecture allows for backup and restore of heterogeneous workstation platforms. In accordance with this, Novell SMS has developed various workstation TSAs. Among them are the Win95 TSA, OS2 TSA, MAC TSA, and DOS TSA (Windows 3.1 TSA). The new workstation TSA being introduced with NetWare 5 is the NT TSA (see Figure 11).

Figure 11: Illustration of how workstations fit into the SMS architecture.

In this section, we discuss the new NT TSA and also give an update to the other TSAs where enhancements have been made.

NT TSA. There has been a high demand for a good backup solution for the Windows NT workstation. Figure 12 shows the architecture of the NT File System together with the SMS components for the NT workstation.

Figure 12: Architecture of the NT File System together with the SMS components for the NT workstation.

Windows NT TSA is a Target Service Agent for the Windows NT workstation. The main task of the WinNT TSA is to perform backup and restore by reading data from and writing to the WinNT File System (NTFS) and File Allocation Table (FAT). The WinNT TSA behaves like any other SMS-compatible TSA in that it will convert the data into the SIDF before passing it to the engine and will reconvert it from the SIDF after receiving it from the engine. For the benefit of customers, it is the mechanism for accessing the Windows NT File System data. It is therefore very important that the features present in Windows NT are also available in the WinNT TSA as a product.

The following features are provided in the Windows NT workstation TSA:

  • WinNT TSA backs up and restores both FAT and NTFS data. This includes backing up and restoring all the associated normal and extended attributes of the file.

  • WinNT TSA supports the multiple streams and user-defined streams supported by NTFS. The Target Service Agent can back up and restore all the streams for a file in a consistent manner.

  • WinNT TSA provides support for Unicode. WinNT TSA uses the set of guidelines that are defined by Novell SMS in backing up and restore file/directory names in Unicode.

  • WinNT TSA makes use of the 64-bit interfaces of the NTFS. Hence the TSA can backup and restore large files.

  • WinNT TSA supports TCP/IP and SPX/IPX protocols.

  • The authentication will be provided as per the username and password of the Windows NT workstation. The traditional practice (as practiced with other Workstation TSAs) is that the user specifies some arbitrary username and password at the time when he starts the TSA, and the engine authenticates itself by supplying these username and password. This was so since the operating systems like Windows 95 and OS/2 do not have any built-in security methods and authentication schemes. However, Windows NT provides these features. Hence, the engines requesting for TSA services can access the files if they have the required access rights.

  • Windows NT provides a facility of a Backup User who will have all the access permissions on the Windows NT workstation for backing it up. This allows the NetWare administrator to back up a Windows NT workstation without having an account and administrative rights on that workstation.

  • The primary resources available to the Windows NT workstation are shown to the engines when they want to connect to the workstation. These are provided to let them know about the drives and volumes available with the workstation and they can browse down from these root points in the volumes and select the directories and files that they want to backup.

  • WinNT TSA provides a good run time status reporting to the user. The "Event Log" tab form allows the user to be able to track the status of the WinNT TSA. This includes the current state of the TSA (whether registered, listening, connecting to the engine, and so on) and the data transfers taking place between the engine and the TSA.

  • WinNT TSA also allows the user to have control over the behavior of the NT TSA. The user is able to dynamically configure the parameters of the TSA like drives, server name, etc. The "Preferences" tab form provides this ability to control the working of the TSA.

  • WinNT TSA can detect the presence of TSAPROXY on the servers from where it gets SAP from SMDR.

  • WinNT TSA polls at regular intervals for the presence of TSAPROXY on the server where it has registered itself. In case the TSAPROXY is unloaded from the server, it pops up a dialog box indicating the same. The user may then register the TSA to another TSAPROXY.

  • The TSA can register itself to more than one TSAPROXY. This will help the engine to reach the TSA in more ways than one.

  • WinNT TSA supports all the APIs that are specified by the TSA API reference manual and is fully SMS compliant.

  • WinNT TSA also provides a good on-line help in order to guide users to use the NT TSA in a more effective manner.

  • WinNT TSA is the mechanism employed by Novell and its partners to provide target services for Windows NT workstation. This is also to encourage all of the third-party engine developers to use the TSA for Windows NT to enable backup and restore support for Windows NT workstations. This will help them to integrate the Windows NT workstation within a Novell network.

The Windows NT TSA can be downloaded from the Novell SMS support site at: http://supported.sms.novell.com.

Windows 95 TSA Update. Novell SMS recently updated Win95 TSA to now allow users from Pure IP machines to back up and restore the target service's data which it is serving, as opposed to the earlier scenario where it only allows the IPX machines to backup and restore.

The Win95 TSA GUI has been updated for user input in terms of which protocol the user wants to use, such as TCP/IP or SPX/IPX. These options are available to users only if these stacks are present (if TCP/IP stack is not present the user cannot select TCP/IP protocol as his choice). This is also needed in the terms that the same proxy name might be different for TCP/IP and SPX/IPX protocols respectively. While registering is also required to know which protocol to use to register into a TSAProxy which might have TCP or SPX or both TCP and SPX listeners running on the server. By default the preferred protocol is IPX/SPX.

A new registry entry called "PrefProtocol" under the existing "NetWare TSA" key was created for keeping the selected preferred protocol by the user. Currently, this can be "TCPIP" or "SPXIPX". Two new radio-box controls, which are selectable mutually exclusively to each other, have been added with IPX/SPX and TCP/IP labels so that the user can select the protocol on which he wants Win95 TSA to register and listen to the SMDR requests. If he selects TCP/IP, the server name is interpreted as the DNS name (complete DNS name is required in case the server is not in the current domain).

DNS is used for resolving the name of the server to address. If users know the IP address of the server where TSAProxy is running, they can give the IP address in the a.b.c.d format directly in the server name edit field. If SPX/IPX is selected the server name is interpreted as NetWare Server name whose details are got from reading the bindery from the nearest default NetWare server. In both the cases the list of servers shown to the users contains the names of the NetWare servers where the SMDR is running. This is with the assumption that most NetWare servers with Native IP release carry both SPX and TCP protocols. It is assumed that IP servers having SMDR and TSAProxy need to be specified by the user directly in terms of DNS name or IP address.

Summary

SMS is a set of modules and APIs which provide a platform-independent strategy for data management and migration across networks. This AppNote provided a description of all of the enhancements that were made to the NetWare 5 operating system and presented an overview of what has changed in the new release of the SMS modules.

For more information on SMS visit Novell's Web site at

http://www.novell.com.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates