Novell is now a part of Micro Focus

Developing Target Service Agents Using CCSCL

Articles and Tips: article

SUDHIR SUBBARAO
Integration Manager
Systems Engineering Division

VIJAYACHANDRAN JAYASEELAN
Software Engineer
Systems Engineering Division

01 Dec 1998


Describes how with Storage Management Services (SMS) developers can create backup engines to back up and restore data not only on NetWare platforms, but on Microsoft, Macintosh, OS/2, and UNIX platforms.

Introduction

Storage Management Services (SMS) is a set of modules and APIs which provide a platform-independent strategy for data management and migration across networks. With SMS, developers create backup engines which are used to backup and restore data not only on NetWare platforms, but all Microsoft, Macintosh, OS/2, and Unix platforms.

There are two parts to SMS: 1) the enhanced SBACKUP utility (which ships with NetWare 5) provides basic backup services for NetWare users, and 2) the set of modules and libraries shipped with Novell's SDK.

This Developer Note focuses on the modules and libraries in the SDK.

Common Code Service Class Library & TSA

Common Code Service Class Library (CCSCL), as the name indicates, is a class library for writing new services compliant to Novell Storage Management Services (SMS) standard. Untill now the most used and written service was Target Service Agent (TSA), but with Portable Storage Management Data Requestor (PSMDR) it is possible to write many kinds of services. The CCSCL object model is extensible to accommodate new types of services.

Currently, Common Code Target Service Agent (CCTSA) class library is extended from CCSCL object model for writing new TSAs. It is expected that writing new TSA using CCTSA will require significantly less development time. The resulting TSA is more modular and better maintainable. CCTSA abstracts most of the control flow and generic TSA functionality in the base classes so that the developer only needs to focus on the data or resource abstraction which needs to be backed up or restored. Although, most of the TSA functionality is provided in the base classes of CCTSA, the object model is extensible enough to accommodate any particular features or to replace an existing feature as required by the developer.

Clearly CCSCL and CCTSA would save developers a lot in TSA development time by eliminating the need for developers to know a lot of details and System Independent Data Format (SIDF). CCSCL and CCTSA also make use of the common control flow that most TSAs have and concentrate on only the data or resource which they are operating on.

This section provides an overview of how you can use CCTSA to write new TSAs. It first explains the CCSCL object model. This is followed by an explanation of the CCTSA object model for writing TSA type services. It also provides a high-level explanation of design issues associated with writing new TSAs. The issues addressed by CCSCL & CCTSA are:

  • Broker (or SMDR) Interface

  • Connection Management

  • Sequence Management

  • SIDF Formatting

  • SIDF Deformatting

  • Resource (or DataSet) Abstraction

Using CCSCL and CCTSA to write Novell SMS compliant services offers a variety of benefits to the development and user community. Some of these benefits are:

  • Independence from file system revisions: When users upgrade to the newest version of NetWare, their storage management applications continue to operate reliably.

  • Flexibility and interchangeability of modules and products: Users can customize storage solutions to their specific requirements.

  • An open architecture to support current and future technology: Storage management application development is facilitated, resulting in increased third-party participation, full-featured application extensions and differentiation.

  • Backward compatibility: As new storage management applications are delivered, users maintain compatibility with existing storage solutions.

  • Eliminates hardware and software interdependence: Users can upgrade to higher performance hardware without replacing storage management software. Likewise, users can move to more sophisticated SMS applications, while maintaining their investments in installed storage hardware.

CCSCL Domain

Figure 1 illustrates the domain of CCSCL w.r.t to other domains. As illustrated, the two other domains CCSCL interacts with are Broker and TargetService Domain. A Service Agent written on CCSCL exposes the interface, generally a set of functions, that a client can use. Service Agent needs to interact with Broker (communication channel) to offer its services to remote clients. Similarly, the Service Agent also needs to interact with TargetService to operate on its data through the interface provided by the TargetService.

Figure 1: CCSCL Domain.

For example, NetWare Target can host multiple TargetServices like Novell Directory Services (NDS), Domain Name Service (DNS) and File System Service. Each of these TargetServices are exposed for use by SMS services via Target Service Agent for backup and restore. Each of these Target Services have their own interface for reading and writing their respective objects. For example NDS has NWDSBackupObject() and NWDSRestoreObject() APIs for reading and writing NDS objects.

CCSCL Object Model

The CCSCL Object Model is explained using Object Modeling Technique (OMT) class diagrams. Through out this article only static models have been used to explain the abstraction and the relationship between them.

As illustrated in Figure 2, there are five basic classes. These are Target, TargetService, ServiceAgent, Broker and Resource. ServiceAgent abstracts the service which exposes a set of functionality to be used by engine. TargetService abstracts the service entity on the network which requires storage management services. Some of the example of TargetService are NDS, DNS, and File System service. Resource abstracts the data that need to be operated upon through TargetService interface. Broker abstracts the communication bus interface which provides name resolution and location transparency. Target abstracts the entity on which the TargetServices are executed.

Figure 2: CCSCL Base Object Model.

Broker, which is generally an executable entity, "Runs On" Target. On a Target, generally one or more Broker can exists. Target also "Hosts" multiple ServiceAgents. Different ServiceAgents "Registers Into" Broker to make them available for use by clients. TargetService which is hosted on Target "has" many Resources to be "exposed out" by ServiceAgents.

Broker abstraction takes care of the interface between ServiceAgents and communication bus like Storage Management Data Requestor (SMDR) or Common Object Request Broker Architecture's (CORBA) Object Request Broker (ORB). For an existing service to operate on new communication bus, the Broker alone needs to be specialized and implementation needs to be carried out for that communication bus.

Different types of ServiceAgents can be implemented by customizing the ServiceAgent class. ServiceAgent class exposes the interface for a particular Service to be used by the client. For example, TgtSvcAgent class which represents the TSA type of service, is derived from ServiceAgent and has member functions which map directly to all the TSA APIs exposed earlier through TSA APIs.

If the implementation is being carried out on a NetWare platform the Target needs to be extended to abstract IntranetWare Target. Once this IntranetWareTarget class is available, it is reusable for different services written on the NetWare platform. For example, NDS TSA and IntranetWare TSA running on NetWare can reuse this IntranetWareTarget class to access host specific functionality.

Similarly, once the Broker interface has been extended for a particular communication bus it should be reusable by all the services that want to use the same communication bus to communicate to a remote client. For example, if the Broker is derived as PortableSMDR class, this PortableSMDR will interact with PSMDR module to implement the Broker interface. PortableSMDR class will be reusable for all kinds of services trying to use PSMDR communication bus. Essentially, Broker is an abstract class which uses the Adapter design pattern. Adapter helps convert one interface to another as expected by the client. Adapter lets classes work together that otherwise couldn't because of incompatible interfaces.

In this scenario, PSMDR will have an interface for the clients to interact with. ORB will have another interface for the clients to interact with. But some classes which need to interact with PSMDR, ORB or other communication bus, cannot be rewritten every time a new communication bus needs to be used. Hence, Broker class abstracts the interface needed to interact with communication bus. Derivations from Broker like PortableSMDR class implements the Broker interface by delegating the functionality to PSMDR via the PSMDR client interface. This in turn indicates that the PortableSMDR class needs to be implemented only once for all the different ServiceAgents trying to use PSMDR as communication bus for remote client usage.

CCTSA Object Model

CCTSA extends CCSCL to write different kinds of TSAs like File System TSA, DataBase TSA, and so on. A brief object model for CCTSA is given in Figure 3.

Figure 3: Overall CCTSA Object Model.

As illustrated in Figure 3, TgtSvcAgent class is derived from ServiceAgent class to expose the TSA interface as member functions. The TSA interface consists of a set of functions that need to be implemented. Some of them are ScanDataSetBegin(), ScanNextDataSet(), ScanDataSetEnd(), ReadDataSet() and WriteDataSet(), and so on.

Resource is specialized to TSAResource. TSAResource abstracts the entity that needs to be backed up or restored, using TgtSvcAgent or in other words, TgtSvcAgent "Backs up and Restores" one or more TSAResource. Both TSAResource and TgtSvcAgent are abstract classes that needs to be further derived to implement specific functionality. Most of the code reuse, in terms of control flow occurs at TgtSvcAgent class implementation. While writing new TSAs you need to derive TgtSvcAgent class and implement abstract functions, and override selective functionality as required.

Resource Abstraction

Every TSA backs up or restores a TSAResource object. The TSAResource provides adequate interface so that it can be used by TgtSvcAgent class which depicts the TSA.

Figure 4, shows some of the possible ways of extending the TSAResource abstraction to suit the NetWare file system TSA.

Figure 4: TSAResource specialization for NetWare file system TSA.

Connection Management

Every TgtSvcAgent manages multiple connections from different clients. In CCTSA object model this is handled by using ConnectionMgr and Connection abstractions. Connection class abstracts a connection which the TSA maintains with a particular client. ConnectionMgr manages various connections a TSA is currently maintaining.

As illustrated in Figure 5, ConnectionMgr manages one or more Connections. ConnectionMgr is derived to form DefConnectionMgr which is the default implementation of ConnectionMgr class available in CCTSA. If you want to have a different kind of Connection and ConnectionMgr just derive and implement from these classes.

Another important abstraction in Figure 5 is TSAFactory. TSAFactory uses Abstract Factory design pattern. Abstract Factory design pattern is used to provide an interface for creating families of related or dependent objects without specifying their concrete classes. For detailed information see, "TSAFactory."

Figure 5: ConnectionMgr and Connection abstraction w.r.t TgtSvcAgent class.

The 1:N relationship between Connection and ConnectionMgr is implemented using a linked list such as, data member "m_connList" in DefConnectionMgr class. For performance reasons, if you want to have a connection manager having hash table, derive from ConnectionMgr and implement the abstract methods.

Sequence Management

On a TSA connection, multiple scan sequences can occur concurrently. CCTSA object model has ScanSequence and ScanSequenceMgr classes abstraction to take care of this functionality. The class diagram for this scenario is given in Figure 6.

Figure 6: ScanSequence and ScanSequenceMgr's relationship with TgtSvcAgent.

As illustrated in Figure 6, TgtSvcAgent has one instance of ScanSequenceMgr. ScanSequenceMgr manages zero or more ScanSequences. DefScanSequenceMgr is the default implementation of ScanSequenceMgr abstract class in CCTSA. TSAFactory binds DefScanSequenceMgr to ScanSequenceMgr instance in TgtSvcAgent.

Broker Interface

For remote usage, a service needs to talk to the communication bus. Different communication buses have different interfaces. For example, PSMDR has a different interface from CORBA's ORB bus. Broker interface takes care of different communication bus interfaces by using Adaptor design pattern.

SIDF Formatting

One of the major tasks for TSA developers is to format the TargetService's native data into SIDF format. SIDF formatting requires a thorough understanding of SIDF. Essentially, a major amount of time is spent on formatting the resources. CCTSA tries to make the task of formatting easier by using conceptual abstractions for SIDF and the help of Formatter. Figure 7 illustrates the class diagram for this scenario.

Figure 7: SIDF formatting of TSAResource via SIDFFormatter.

As seen from Figure 7, Field, FieldTable and SIDFSection classes directly map to the SIDF conceptual abstractions. Field consists of a Field Identifier (FID), optional length and the data. FIDs can be 1, 2, 3 or 4 bytes. FIDs can be of fixed data length or variable data length format. Only variable length Field has length part. This length part can be of Direct, InDirect or BitData format. A FieldTable consists of one or more Fields. For example, Path Field Table consists of Path field, NameSpace field, NamePositions field, etc. A SIDFSection is a logical collection of FieldTables and SIDFSections in predefined sequence.

For example, a File is formatted into SIDF using File Section which has:

  1. SrcFileHeader FT

  2. Path FT

  3. Characteristics FT

  4. StreamHeader FT

  5. StreamData

  6. StreamTrailer FT

  7. SrcFileTrailer FT

The stream specific information (for example, 4, 5, and 6 in the list above), can be repeated to take care of multiple streams corresponding to the file that is being backed up.

Each concrete TSAResource derivative specifies, the SIDFSection the Formatter should use to format into SIDF. The Formatter is an abstract class which defines an interface. All the derivatives of the Formatter to be used by TSA should implement this interface.

SIDFFormatter class is just a derivation from Formatter which formats the TSAResource data into SIDF. Formatter also maintains the state of formatting, which includes the current resource being formatted, the amount of formatting that has already been completed for the current TSAResource, buffer management, etc.

SIDFFormatter is generic enough to format new TSAResource types which will be derived in future. This is because, SIDFFormatter uses the abstract interfaces to implement formatting. The advantage is that a TSA developer needs to focus only on TSAResource and implement its interface by deriving it to suit their needs, rather than formatting the TSAResource.

SIDF Deformatting

Deformatter and SIDFDeformatter classes takes care of deformatting SIDF data. Abstraction wise deformatting related classes are similar to formatting classes. SIDFDeformatter is derived from Deformatter class. Figure 8 gives the object model for this scenario.

Figure 8: Deformatter and SIDFDeformatter interacting with TgtSvcAgent and TSAResource.

TransferBuffer is an abstraction which is used for maintaining SIDF formatted data, before passing it on to the engine in case of backup or writing via TSAResource in case of restore.

This buffer has the ability to maintain partial information in memory and remaining data in files. How much memory needs to be committed in RAM can be specified during the creation of the TransferBuffer object. In case the TransferBuffer object is marked as virtual, it uses file for caching the data in the buffer once it runs out of committed RAM.

TSAFactory

A lot of TSA control flow has been coded into TgtSvcAgent class implementation using abstract interfaces as evident from the previous section. Now the question is, how does the TgtSvcAgent know which concrete implementation for an abstract interface to use?

TSAFactory helps in this, by using the Abstract Factory design pattern. Abstract Factory provides an interface for creating families of related or dependent objects without specifying their concrete classes. Abstract Factory is also called a Kit.

Most of the new TSAs being written should be derived from TSAFactory to implement their own factories. Assume TgtSvcAgent gets derived as INWTgtSvcAgent class. INWTgtSvcAgent class constructor will bind the concrete factory instance to the abstract TSAFactory instance present in TgtSvcAgent class.

TSA Development using CCTSA

This section explains the steps you need to follow to develop new TSAs using CCTSA. Assume that we want to develop a new DataBase TSA called "MyDataBase TSA." You need to complete the following steps:

  1. Create a new TSAFactory derived class called MyDBTSAFactory. Defer implementation of each member function of this class untill other steps are performed.

  2. Abstract all the resources that you will be exposing through the TSA using TSAResource as your base class. For example, assume that, MyDataBase TSA will expose DataBase, Table and Record Resources. These can be abstracted as MyDataBaseResource, MyTableResource and MyRecordResource derived from TSAResource. For container resources like MyDataBaseResource and MyTableResource, implement the iterators by extending TSAResourceIterator class. Implement NewTSAResource() member function in MyDBTSAFactory class to bind the concrete resource created to abstract TSAResource.

  3. Decide how you want to layout MyDataBaseResource, MyTableResource and MyRecordResource in SIDF. Assume that we have some new Fields and FieldTables. Extend all the new field tables using FieldTable class as the base class. For example, some of the field tables can be Source DataBase Header Field Table, Source DataBase Trailer Field Table, Source Table Header Field Table, Source Table Trailer Field Table, Source Record Header Field Table and Source Record Trailer Field Table. All these field tables should be extended from FieldTable class.

After completing the field tables, the new SIDF section needs to be specified for all three resources, DataBaseSIDFSection, TableSIDFSection and RecordSIDFSection. These sections will specify how the field tables are laid out and the number of times each field table or section will repeat. All these sections should be derived from the SIDFSection base class.

Implement the NewSIDFSection() and NewFieldTable() member functions of MyDBTSAFactory. This will bind all field tables and SIDF sections specific to MyDBTgtSvcAgent to FieldTable and SIDFSection abstract interfaces. These SIDFSection classes will be reusable if you want to write TSA for some other DataBase.

  1. Abstract the TSA in a class called MyDBTgtSvcAgent derived from TgtSvcAgent class. Provide a fresh implementation of the member function to override the default functionality implemented in TgtSvcAgent class. For example, if you want to override ScanTargetService() member function then this needs to be implemented in MyDBTgtSvcAgent class. Implement all abstract methods of TgtSvcAgent class for example, BuildResourceList().

In the constructor of MyDBTgtSvcAgent class, bind MyDBTSAFactory class instance to TSAFactory class instance present in TgtSvcAgent base class.

  1. If a Broker interface implementation already exists then skip this step. Otherwise, implement the Broker interface for the required communication bus, for example, Portable SMDR. In MyDBTgtSvcAgent class constructor, bind Portable SMDR instance to Broker instance present in TgtSvcAgent base class.

  2. Compile, link and form a Dynamic Link Library / Shared Library. Register this library to the Broker (communication bus) through which you expose this service. Whenever there is a request for this service, Broker takes care of loading the library and creating an instance of the MyDBTgtSvcAgent class and invoking the appropriate methods.

You are done with your major TSA development work other than small tasks like ScanControl, TSA Generic Options, TSA Specific Options and Open mode strings implementation. There are classes to represent these specific features. Extend and implement your classes using these as base classes. Bind them to their base class instance in MyDBTgtSvcAgent class constructor.

CCTSA Features

Extensibility

CCTSA object model is very extensible. One example of this is, if you want to change the Connection Management and Connection abstraction all you need to do is derive from Connection and ConnectionMgr classes and implement your specific code. Bind your connection manager to ConnectionMgr class through TSAFactory derivative.

Modularity

CCTSA object model is modular. For example, Broker adapter class segregates communication bus interface from CCTSA. If you have your TSA running on Portable SMDR and you want to make the TSA available through ORB then, all you need to do is to provide an ORB specific implementation of the Broker Interface. The other part of CCTSA code will work as is.

Reusability

CCTSA is reusable. CCSCL defines the interfaces, CCTSA implements these interfaces and exposes some abstract interfaces on its own to be implemented by the developer. TgtSvcAgent class contains most of the TSA functionality already coded to abstract interfaces. Other parts of the CCTSA are also reusable, for example, SIDFFormatter and SIDFDeformatter will work with all future TSAResource derived classes, provided developers stick to and implement TSAResource interface.

Another instance of reusability is Broker interface and its implementation. Once Broker implementation is carried out for a communication bus like Portable SMDR, it can be reused with various other services like TSAs. This reusability factor should reduce TSA development time significantly.

Maintainability

Base classes have most of the control flow. Different types of TSAs can be written on top of these base classes. If there is a bug in the base class it needs to be fixed only once. All the other modules just need to be recompiled to fix the bug.

Since real word abstractions are used frequently, you can easily locate a functionality by looking at the class name. Following this, you can either extend the class or patch some feature in the class. Most of the abstractions in CCTSA are for data that needs to be operated upon. So you just need to focus on the data that is being backed up or restored.

Summary

CCSCL defines a generic SMS object model that can be used to implement various kinds of services. CCTSA extended from CCSCL helps developers write TSA services quickly. TSA developers don't need to worry about how a particular TSA functionality is carried out, for example, how the scan is done when you call ScanDataSetBegin(). Instead they only need to concentrate on the data that is being backed up or restored. TSAs written using CCTSA are modular, extensible, reusable and maintainable.

Support

Support for SMS is provided via this newsgroup (devsup.novell.supported.sms.). If you have an SMS-related issue, this is the place to have it addressed.

In the event that an issue requires additional work or a significant amount of communication, Novell Developer Support may elect to open a support incident, to better track activity.

Note: From http://www.developer.novell.com/support you can search for information on SMS, including problem/resolution scenarios and articles.

Glossary


Glossary of SMS and NDS Terms

Master Replica. The first replica created when the partition is defined. It is a writable replica used to provide database integrity during partition operations (splits, joins, moves, rebuilds, and type changes). The master replica "locks" all replicas of the partition so that no other partition operations can be performed while that operation is in progress. For each partition of the NDS tree, there can be only one master replica. (This is sometimes erroneously referred to as the "master partition").

NDS Tree. The hierarchical structure into which objects are organized in NDS, resembling an inverted tree wherein objects representing groups of network resources branch downward from a single Root object.

Object. A logical representation of a physical entity or network resource such as a server, volume, user, group, print queue, directory map, and so on.

Partition. A subtree, or branch, within the NDS tree identified by the name of its top-most object (the one closest to the root of the tree). A partition contains only NDS objects and related data. It does not include any information about file system data files or directories.

Read-Only Replica. A third type of replica that is not writable, so it can be used for authentication but not for login requests. It can only be updated from other replicas, not from a client application. Read-Only replicas are seldom used.

Read/Write Replica. A replica created for a partition that can be used for authentication. Since they are writable, read/write replicas can also be used for users' login requests. You can have any number of read/write replicas of a partition.

Replica. An identical copy of an NDS partition. Replicas allow multiple copies of each partition to be stored on NetWare 4 servers around the network.

[Root]. The top-most container object in an NDS tree, from which all other container objects branch downward.

Schema. The rules governing what types of objects can be stored in the NDS database and what attributes they can have. The NDS schema is extensible to allow developers to define and use objects that aren't included in the default schema that comes with the NetWare 4 software.

Storage Management Data Requester (SMDR). The communication link between the backup application (SME) and the SMS TSAs. It is the function of SMDR to locate the SMS TSAs on the network and to communicate instructions to the intended agent.

Storage Management Engine (SME). The central component in an SMS-based backup system. The SME handles all the user interaction and forwards the users' requests to the TSAs and to the device driver. SMEs communicate with network clients through a common interface that ensures cross-platform support and the ability to support new platforms and versions with minimal changes. The SME contains most of the value-added features and is the component most often provided by third-party developers.

Storage Management Services (SMS). An open architecture for interfacing storage management products with the NetWare operating system. The SMS architecture provides a standard interface for storing, accessing, and managing data on a variety of computing platforms (NetWare servers, DOS/Windows clients, Macintosh clients, and so on).

System Independent Data Format (SIDF). An international standard (ISO 14863 and ECMA 208) defining a common format for storing data on secondary storage devices such as tape. SIDF ensures compatibility between current and future SIDF-based products, and stored or archived data.

Target. Any item on the network that requires backup. Examples of targets include NetWare servers, client workstations, NDS databases,print servers, and SQL database engines.

Target Service Agent (TSA). A software module that knows how to scan, read, and write the target file system data. The primary functions of an SMS TSA are to prepare the target data for backup or restoration, and to communicate with the SME. The TSA is the only module in SMS that needs to understand the details of the target file system's structure (path specifications, directory hierarchy, access rights, and so on). Because the knowledge of the target file system structure is confined to the TSA, the TSA can package data from the target and present it to the SME in a generic format.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates