Novell is now a part of Micro Focus

Inside Novell's High Capacity Storage System (HCSS)

Articles and Tips: article

ROSS MAXFIELD
Software Engineer
NetWare Products Division

01 Feb 1995


This AppNote is an in-depth exploration of Novell's High Capacity Storage System (HCSS), one of the new features in NetWare 4 to help utilize server disk storage more efficiently. The AppNote presents a theory of operations for HCSS and describes the principal processes associated with migrating and demigrating files between a NetWare volume and a magneto-optical media jukebox. It explains the settable parameters that accompany HCSS, and ends with some answers to frequently asked questions about HCSS.

Introduction

NetWare 4 features three new server technologies to help you utilize disk storage on your NetWare server more efficiently:

  • File compression

  • Disk block suballocation

  • Data migration

The first two topics are discussed in the AppNote entitled "Compression and Suballocation in NetWare 4" published in the June 1994 issue of Novell Application Notes. This AppNote explores data migration. We won't go into the mechanics of installing and administering the High Capacity Storage System (HCSS) in this AppNote, as that information is covered in Chapter 6 of the Supervising the Network manual for NetWare 4.1. Instead, we will look at the underlying philosophy behind data migration, examine the internal workings of HCSS, explain the settable parameters for HCSS, and answer a few questions at the end.

Philosophy Behind Data Migration

The deluge of electronic data in today's information-centric society raises some serious concerns with regards to storage capacity. As the amount of data to be stored increases, it becomes more and more difficult to determine what data to keep and what data to archive or throw away.

Typically, much of the data we hold on to is rarely used and eventually becomes disposable. Consider the number of executable, data, multimedia, and help files that are seldom, if ever, accessed and yet continue to occupy precious disk space on your server. When faced with the prospect of "disk full" errors, the knee-jerk reaction is to add another hard disk. However, this may not be the wisest choice because it can become costly and it increases the probability of device failure and potential data loss. Besides, since a good portion of the data is seldom used, it should be kept somewhere where the storage costs per byte are lower, while maintaining reasonable access.

Hierarchal Storage Management (HSM), or data migration, is a better solution to the dilemma of what to do with rarely-used data. The philosophy behind HSM is to migrate aged files from more expensive, faster, relatively low-capacity storage devices onto less expensive, slower, relatively high-capacity devices. Eventually, if the files remain unused and are determined to be no longer necessary, they are discarded. This process is akin to moving old National Geographic magazines from the coffee table in the front room, to boxes in the basement, and then eventually to the recycling bin.

A Brief History of HCSS

Hierarchical storage management has been around for some time on mainframes and other specialized systems. With NetWare 4.0, Novell introduced its own hierarchal storage management system: the High Capacity Storage System (HCSS).

HCSS was originally developed by Imagery Software, Inc. of Bedford Massachusetts. Imagery Software is a subsidiary of Eastman Kodak Company. Imagery is producing a comprehensive document management system, and HCSS was the first part of this development.

Note: Hierarchical Storage Management solutions have recently become available for NetWare 3.1x through third-party developers.

Since the release of NetWare 4.0, Novell has acquired ownership of HCSS and has continued its development. With the release of NetWare 4.1 comes a more mature HCSS with added features. Notable changes to HCSS for NetWare 4.1 include:

  • Support for larger jukeboxes

  • Support for multiple jukebox magneto-optical drives

  • Support for the migration of compressed files

  • Elevator Queues for requests to reduce jukebox thrashing

  • Asynchronous deletes

  • HCSS console commands and settable parameters

  • Complete support for client migration APIs

We'll look at each of these features more closely as we discuss the operation of HCSS.

HCSS Theory of Operations

HCSS migrates less-active file data from a NetWare volume to a magneto-optical (MO) jukebox, without disrupting the directory structure on the volume. If an attempt is made to access the data after it has been migrated, the data is "demigrated" from the jukebox back onto the volume at the time the file is opened.

To the casual user, it appears that the data has always been on the volume because the original directory structure is still intact. The more perceptive user may notice a delay when opening the file. The delay occurs while the MO media is moved from a slot in the jukebox to one of the drives in the jukebox.

The extremely astute user may also notice when executing the DIR command that the aggregate number of bytes used in file data may be greater than the total capacity of the volume. Each time file data is migrated from the volume to the jukebox, the space the file data occupied on the volume becomes available again for new file data. Thus the apparent volume capacity is the total storage potential of the jukebox. So, for example, a 100 megabyte volume can appear to contain many gigabytes of data.

It is important to understand that HCSS is not a backup solution. The term "backup" implies multiple copies of sets of data. With HCSS, it is intended that there be but one copy of data - either on the volume or on a piece of storage media in the jukebox.

Architecture of HCSS

HCSS is made up of four principal modules: a Support Module, a Decider Module, an Import/Export Module, and a System Error Handling Module. These pieces of software are implemented as NetWare Loadable Modules (NLMs). As Figure 1 implies, these four modules are tightly integrated and share interfaces.

Figure 1: The four main modules in HCSS work closely together and share interfaces.

  • The SystemError Handler Module (HCSSERR.NLM) provides the mechanism to create, display, broadcast, and log error events and messages from the other HCSS modules.

  • The Support Module (HCSS_SM.NLM) is responsible for the management and movement of the media in the jukebox and for the movement of data to and from the media (that is, the file system on the media).

  • The Decider Module (KDECDAEM.NLM) scans HCSS's domain directories and builds a list of candidates for file migration.The list is sorted with least-recently-used files first to ensure that these files are migrated first. The migration event is triggered when the percent of volume capacity used crosses a definable upper threshold. Migration then continues until either (1) the list of candidates is exhausted, or (2) a definable lower volume capacity threshold is reached.

  • The Import/Export Module (IMPEXP.NLM) manages the importing and exporting of media to and from the jukebox. When media is exported from the jukebox, HCSS migrates all the files associated with that piece of media. These files are then deleted from the volume and the media is moved to the jukebox mail slot to be shelved. When media is imported into the jukebox, HCSS recreates the directory structure and directory entries as they were at the time the media was exported. Access to the filedata is thus restored. Media may also be formatted at the time of import for use as a new piece of media. Any data that was on the media before the format is lost.

  • The System Error Handler Module (HCSSERR.NLM) provides the mechanism to create, display, broadcast, and log error events and messages from the other HCSS modules.

These four modules will be discussed at greater length later in the document. In addition to these modules, HCSS includes three other modules:

  • HCSSUTIL.NLM, a collection of functions that are shared by the HCSS modules.

  • HCSS_IR.NLM, which makes function calls that report the results of the jukebox inventory public to the HCSS subsystems. These calls are used by IMPEXP.NLM, KDECDAEM.NLM and HCSS.NLM.

  • HCSS.NLM, which makes available some HCSS Enhanced NCPs. These NCPs allow HCSS to be administered from NWAdmin.

Six dynamic link libraries (DLLs) are also used to provide HCSS snap-in functions for the Windows-based NetWare administrator utility (NWADMIN.EXE, or NWAdmin for short). These DLLs will be listed later.

HCSS requires CLIB.NLM and the Real Time Data Migrator (RTDM.NLM). Loading HCSS.NLM will autoload all the dependent NLMs listed above. The drivers required for access to the jukebox must be loaded before HCSS is loaded. Once HCSS is loaded and completes an inventory of the jukebox, data migration is enabled.

The HCSS Volume

HCSS migrates file data between a single NetWare volume and a single jukebox. For best management and efficiency, this volume should be created explicitly for HCSS. The HCSS volume acts as a cache for files to be migrated and for files that have already been migrated. The volume's migration attribute must be set to ON. Other attributes (such as for compression and suballocation) are optional.

The size of the HCSS volume should be carefully planned. If the intent is to ultimately migrate millions of files, you should consider the storage overhead per file. Typically 512 bytes of volume storage space are required for each migrated file. If 1 million files are migrated, 512 million bytes of storage space will be required just for the directory entries. (This does not include the space used by unmigrated file data.)

If this seems like a lot of overhead, consider the following example. Suppose the average migrated file size is 50K. This means that 512 million bytes are used to maintain access to over 51 gigabytes of file data - a mere 1 percent overhead.

In addition to this overhead, you should consider the rate at which files will be created, migrated, and demigrated to ensure the volume will have enough free space to work efficiently.

Media Centricity

HCSS is a media-centric storage system. This means that files in specific places on the HCSS volume go to specific media in the jukebox. From the root of the HCSS volume, an administrator creates HCSS domain directories. The domain directories are the first level of management for migrated files. After a piece of MO media is imported and formatted, it is assigned to a domain directory.

Within the domain directories are media-label directories. During the format process, the administrator defines labels for each side of the media. These labels are used as names for subdirectories created in the domain directory when the media is assigned to that directory. The media-label directories are the second level of management for migrated files. Files created in the media-label directories or in subdirectories below them are migrated to the side of media corresponding to the media label.

Figure 2 illustrates the media centricity of HCSS. In this example, HCSS is the volume name and HISTORY and EVENTS are domain directories. Under HISTORY are media-label directories named MAY, JUNE, JULY, and AUGUST. Under EVENTS are the PROVO and OREM media-label directories.

Media centricity implies that no more files can be migrated from the media label directory tree than there is capacity on one side of media. The administrator should take care when selecting media to be used in the jukebox, as this will determine the amount of migrated data per media-label directory. Currently, media capacities per side range from approximately 300 megabytes up to approximately 650 megabytes, depending on the media type (high or low density) and the number of bytes per sector (512 or 1024).

Figure 2: HCSS is media centric, which means that directories in the NetWare volume correspond to specific sides of the MO media.

HCSS and Related Modules

Following is a discussion of the main HCSS modules, as well as other modules that HCSS depends upon.

Real Time Data Migrator (RTDM.NLM)

HCSS interfaces with the NetWare file system via the Real Time Data Migrator (RTDM). The RTDM is an NLM that exposes a collection of APIs used to move file data between the NetWare file system and various support modules. This functionality essentially creates transparent access to foreign file systems. Several support modules may be registered with the RTDM at any given time.

When a file is migrated, the RTDM calls the support module's Create routine. The Create routine responds by giving the RTDM a Data Migration Key (DMKey). The DMKey is stored in the migrated file's extended attributes area in the NetWare file system. After the file's data is passed to the support module, the file is truncated within the NetWare file system. Truncating a file removes the file's data from the NetWare file system; however, the file size information remains intact, as if the data had not been removed.

When a request is made to open a migrated file, the request is suspended and the RTDM passes the requested file's DMKey to the support module. The support module uses information in the DMKey to locate the file data on the media and return it to the NetWare file system. After the data is demigrated, the Open request continues.

The RTDM also exposes a set of application APIs that are used to get a variety of information out of a support module - for instance, version information, media capacity totals, migrated file information, and other assorted information. These APIs are available to server or client applications.

Jukebox Driver (HPCHGR.DSK)

The Hewlett-Packard Jukebox Driver (HPCHGR.DSK) was written for HCSS to support several of HP's magneto-optical jukeboxes or jukeboxes that emulate the HP jukeboxes. The jukeboxes used in the development of HCSS were:

  • Model 20T (16-slot, single drive)

  • Model 40T (32-slot, dual drive)

  • Model 120T (88-slot, dual drive)

  • Model 200T (144-slot, four drive)

There are third-party drivers available for other jukeboxes.

HCSS assumes media is 512 bytes per sector; however, media is also available in 1024 bytes per sector format. HPCHGR.DSK emulates 1024-sector media as 512-sector media. As a result of this emulation, writes to 1024-sector media are a bit slower. The driver must first read the entire 1024-byte sector, place 512 bytes of data in the correct half of the 1024-byte sector, and then write the 1024-byte sector. This decrease in performance is not apparent to the client because writes take place only during migrations and deletes. Neither of these events degrades the performance of activity at the client.

The HP jukebox driver is not bus specific and therefore requires the ASPITRAN layer. Also note that HPCHGR.DSK does not lock the front panel of the HP jukebox. This creates the possibility for media to be removed from or inserted into the jukebox without the driver's knowledge. Doing so will be disastrous to the operation of HCSS. All media must be moved into or out of the jukebox via HCSS's administration snap-ins to NWAdmin.

One final note: the driver for the SCSI adapter to which the jukebox is connected must be loaded first, before any other SCSI drivers.

System Error Handler Module (HCSSERR.NLM)

HCSSERR.NLM is the error message handling system for each of the HCSS NLMs. Three types of messages are received by HCSSERR: events, warnings. and errors. Some examples in each category follow.

Events

  • Upper Threshold Exceeded

  • HCSS is unable to attain Low Threshold

  • Decider suspended due to volume dismount

  • HCSS is up and running

Warnings

  • Media exceeded remaining capacity limit

  • Could not restore attributes

  • Failed to delete a file

  • Failed to migrate file

Errors

  • Write to media failed

  • Movement of media in the jukebox failed

  • HCSS failed to mount the media

Messages are handled in three ways, depending on the message:

  • Some messages are broadcast to user ADMIN.

  • Some messages are simply posted to the server console screen.

  • All messages are logged in the HCSSERR.LOG file located at the root of the SYS volume.

The HCSSERR.LOG File. SYS:\HCSSERR.LOG logs all messages from HCSS and includes a time stamp with each one. The log file is opened and closed each time a message is written to the file. This allows the administrator to examine, rename, copy, delete, and otherwise manipulate the file while HCSS is running.

The log file will grow no larger than 2 megabytes. At that point messages will be written starting at the beginning of the file again, overwriting the oldest messages. A dashed line marker follows the latest entry in the log file so that the latest messages can easily be found.

Typically there should be no need to examine the log file. In the event of ill behavior, however, the contents of the log file may be used to quickly identify the source of the problem.

Messages broadcast to user ADMIN are an attempt get attention. These messages generally indicate that HCSS is not operating properly or is not configured efficiently. Explanation of and recourse for the broadcast messages are found in the NetWare System Messages manual.

If a message cannot be posted, HCSS will display a message on the server's console indicating the failure to post the message and identifying the message number that failed to post. This will occur if the log file is currently being held open by another application (likely an editor).

Support Module (HCSS_SM.NLM)

HCSS_SM.NLM is by far the largest of the HCSS modules. This is the support module which registers with the RTDM and manages the media in the jukebox and the data on the media. It performs the Migration, Demigration, Rename, and Delete processes and provides the Asynchronous Deletes and Elevator Queue functions. These, and other, support module functions are discussed in this section.

Before media can be used by HCSS, it must be formatted by the Support Module. Media is typically purchased already low-level formatted. The Support Module simply places a high-level format on the media to set up the file system.

Format of Data on HCSS Media. To understand the overhead required for migrated data, it's important to have some knowledge of the format HCSS places on each side of a new piece of media. As shown in Figure 3, each side of media contains five types of data:

  • Stamp

  • Used Sectors Bit Table

  • Directory Segments

  • File Headers

  • File Data

The Stamp is a data block that contains the media label and all the migration statistics. Statistics such as; last access, number of files migrated, total capacity, and remaining capacity. The stamp is updated each time a file is migrated, accessed, or deleted. Some of these statistics are available via HCSS server console commands. These commands will be discussed later.

The Used Sectors Bit Table keeps track of which sectors on the media contain valid data. Each sector on the side of media is represented by a bit in this table. As data is placed in sectors, corresponding bits are set in the table. Bits are reset as sectors become available by deleting migrated files.

Figure 3: HCSS stores five types of data on the MO media.

A Directory Segment contains the directory entries for migrated files. Each directory segment occupies 16 sectors and contains no more than 319 entries. When a directory segment fills up, another is allocated. Thus, directory segments may be scattered throughout the media. The first directory segment is allocated when the media is formatted. Once allocated, the sectors in a directory segment are never freed, even if all directory entries in the segment are deleted.

Each directory entry in the directory segment points to a file header, which is the first sector of every data segment of a migrated file. Basically, the file header contains the size of the migrated file, the full path name of the migrated file, and the DMKey created for the file. If file data occupies 10 sectors on the volume, 11 sectors are reserved on the media for the migrating file's data segment: 10 for the data and 1 for the header.

Note: For each file migrated, one sector is reserved for the information file built during media export (see the discussion of IMPEXP.NLM).

The file data follows the file header. The file data is not necessarily contiguous and may be split up into multiple data segments to make best use of available media space. The file header, in this event, then contain information that points to the next data segment.

HCSS does not modify the data in any way. The file data is written to media as it appears on the NetWare volume. If the data is compressed on the volume, it is compressed on the media. If the data is not compressed on the volume, it is not compressed on the media.

Performing the Jukebox Inventory

When the HCSS Support Module is first loaded, it reconciles the media in the jukebox with the media-label directories in the domain directories. It does this by reading the media label in the stamp of each side of media and confirming the existence of the associated media-label directory.

Should a piece of media be found without associated media-label directories, the media is ejected from the jukebox. Each piece of media ejected from the jukebox must be acknowledged at the server console by typing MEDIA REMOVED after the media appears in the jukebox's mail slot.

During the inventory, all requests for migrated data are denied. Shortly after completing the inventory, HCSS posts the message "HCSS is up and running . . ." to the server console and to the log file (SYS:HCSSERR.LOG). The message is also broadcast to user(s) ADMIN. Data migration and demigration is now enabled. These processes are described in the following sections.

The Migration Process

HCSS migrates data to the MO media as follows:

  1. A file is migrated with a call to the NetWare OS public function MoveFileDataToDM(). Specified in this call is the file to be migrated and the support module to receive the file data.

  2. The support module receives the full path of the file being migrated. The media label is extracted from the path and used to identify the media to receive the file data. For example,if the full path to the file to be migrated is MIGRATE:\AUTOS\FORD\B41970\MUSTANG.PKG, the domain directory would be AUTOS and the media label would be FORD.

  3. Once the media is identified, the support module verifies that there is sufficient space on the media for the migrating file. The number of sectors required to migrate a file is the number of bytes of file data divided by 512 bytes in a sector, rounded up to the nearest sector boundary. Two more sectors are then added to this value: one sector for the file header and one sector reserved for the Info File should the media be exported in the future. (See the discussion on IMPEXP.NLM.)

  4. If sufficient space is available, the support module builds a DM Key and allocates the available space on the media for the file to be migrated. The DMKey is given to the file system and is placed in extended directory space the OS allocates on the volume for the file.

  5. The file's data is then passed to the support module and written to the allocated space on the media.

  6. After a successful migration, the file data on the NetWare volume is truncated. That is, the space occupied by the file on the volume is made available to new files.

The file's directory entry status is also changed to reflect that the file data is migrated. You can use the FLAG.EXEor NDIR.EXE commands with the "/r" option to see a file's migration status. An "M" under the status column indicates that the file is migrated.

The Demigration Process

Migrated data is demigrated when an application submits a request to open a file that is migrated file. The process unfolds as follows:

  1. The Open request is suspended and the DMKey stored in the file's extended attributes is retrieved and handed to the RTDM.

  2. The RTDM identifies the support module responsible for the migrated data and submits an Open request to the support module, passing to it the DMKey.

  3. The support module verifies the validity of the DMKey and uses information stored in it to identify the correct media and the location of the file data on the media.

  4. The RTDM reads the file data back from the support module and places it on the NetWare volume. The directory entry's migration status is cleared.

Once the data is restored, the application's Open request continues as if the file data had always been on the volume.

If HCSS is unable to demigrate file data for any reason, the Open request will fail and the client should get a message indicating "access denied." A message will be posted in the HCSS log file that may indicate the reason for the demigration failure.

The Remigration Process

A file may be remigrated after it has been demigrated. When this happens, NetWare will know that the file was once migrated. Instead of making a request to migrate the file, the OS calls the Support Module to verify whether the file needs to be remigrated.

Since a demigrated file's data is still on the media, it does not make a whole lot of sense to remigrate the data if it has not changed. If the file to be migrated has been modified, or if its compression state has changed since it was demigrated, the original migrated data is deleted from the media and the new file is migrated. If not, the file is simply truncated on the NetWare volume and the directory entry's migration status is set.

Deleting a Migrated File

Migrated files are deleted when an explicit file delete request is submitted or if a demigrated file gets modified. A file can be opened such that if it is modified, NetWare will delete the original file when the new (modified) file is closed.

When NetWare deletes a file that is, or was, migrated, the Support Module is alerted of the event and receives the file's DMKey from the RTDM. The Support Module uses the information in the DMKey to delete the directory entry in the directory segment. It then relinquishes the media space by updating the Used Sectors Bit Table. Currently, the file data on the media is not erased on a delete. The data is overwritten only when a new migrating file allocates the space of the deleted file.

The Support Module cannot prevent the deletion of the file from the NetWare volume, even if the Support Module is unable to delete the file from the media. However, if the Support Module is not loaded at the server, the request to delete a migrated file will fail. A notice indicating "access denied" is returned to the application responsible for the delete request if the support module is not loaded.

A migrated file cannot be salvaged once it is deleted. For this reason,all files in the HCSS domain directories should be immediately purged when deleted. In an attempt to ensure this, HCSS will set the Immediate Purge attribute on domain directories, media-label directories, and each subdirectory below the media-label directory in the directory structure.

Renaming a Migrated File

A migrated file's directory entry must remain in the directory the file was migrated from. However, a migrated file can be renamed. When a request is made to rename a migrated file, the file's DMKey is passed to the support module, along with other information. If the rename places the file in another directory, the rename process will fail. Otherwise, the file is renamed and the DMKey is updated.

Subdirectories under the media-label directory should never be renamed. Doing so will disrupt the import process. The import process will complete, but files under a renamed directory will lose their original attributes and will appear under the original directory name.

Elevator Queues

As requests are received for data on storage media, HCSS queues each request and then releases the requests in a more efficient order. This reordering somewhat resembles the operations of an elevator, where all who want to exit the elevator on a given floor do so regardless of the order in which the buttons in the elevator may have been pushed.

In the past, HCSS serviced requests in a first-in-first-out (FIFO) fashion. For example, suppose the media were labeled A, B, C, and D and demigration requests were received in the following order:

A B D C B A D A

With FIFO, the media would be loaded in the jukebox drive in the same order. A would be placed in the drive three times, B would be placed in twice, and so on. This approach can be rather inefficient and time consuming.

With elevator queuing, the media is placed in the drive to service a request and remains there to service all requests queued up for that side of media. The demigration request sequence in the example above (A B D C B A D A) is serviced in the following order:

A A A B B D D C

The requests for media D are serviced before C because the first request for D entered the queue before the request for C.

In conjunction with elevator queueing, HCSS also has request type precedence. Precedence is defined in the following order:

  1. Demigrations

  2. Deletions

  3. Renames

  4. Migrations

In the request sequence A B D C B A D A, suppose the first request for A and the request for C are demigrations, the subsequent requests for A and all for D are migrations, and the requests for B are deletions. The requests would then be serviced in the following order:

A A A C B B D D

The two migration requests for A are serviced while media A is in the drive, even though migrations have a lower precedence over the pending requests. In this way, the most work gets done in the shortest amount of time.

Settable Parameters for Elevator Queueing. Three settable parameters are used to control the elevator queues:

  • Minimum Time In Drive specifies the minimum amount of time a piece of media remains in the drive waiting for or servicing requests

  • Maximum Time In Drive specifies the maximum amount of time a piece of media remains in the drive waiting for or servicing requests

  • Request Idle Time specifies the maximum time to wait between requests before servicing a request for another piece of media.

These parameters are used to configure the elevator queues to get the best performance out of a wide variety of operating environments. The following paragraphs summarize the three parameters. For more details on setting these parameters, see the section "HCSS Settable Parameters" later in this AppNote.

The Minimum Time in Drive parameter defines the minimum time the media will remain in a drive waiting for requests. When media is placed in a drive, it will remain there for this amount of time,even if there are no further requests for the media. This time includes the time it takes to get the media into the drive. For example, if this parameter is set to 20 seconds and it takes 10 seconds to get the media into the drive, the media will remain in the drive for at least 10 more seconds whether or not there are more requests for the media. Should the media already be in the drive, at least 20 seconds will expire before the media may be removed from the drive to service a request for data on another piece of media.

It is suggested that this amount of time be at least twice the amount of time it takes to get the media into the drive.

The Maximum Time In Drive parameter defines the maximum time the media will remain in the drive servicing requests. This parameter takes effect in the event of continuous requests on a side of media. When this amount of time expires and requests are pending for other media, the current media will be removed and the pending requests will be serviced. If there are no pending requests, the media remains in the drive until requests are received for other media.

This value must be greater than the value for Minimum Time In Drive.

The Request Idle Time parameter defines the maximum amount of time to wait between requests before possibly removing the media from the drive to service pending request for other sides of media. This parameter takes effect only after the Minimum Time In Drive has expired. Once the Minimum Time In Drive has expired, the Request Idle Time is reset each time a request is received for the current side of media. Should the Request Idle Time expire before another request is received, the media is eligible for removal.

This value cannot be greater than the difference between the Maximum and Minimum Time In Drive.

Figure 4 illustrates the interaction between the elevator queue parameters over time with a single-drive jukebox. In this example, the values for Maximum Time In Drive, Minimum Time In Drive, and Request Idle Time are 20, 10, and 3 seconds, respectively.

Figure 4: Example of the interaction between Elevator Queue parameters.

At time t0, media A is placed in the drive to service the request Ar1. The Minimum Time In Drive count begins at this point because a drive has become available to service a media request. If the media is not already in the drive, the time to move the media to the drive will use up some of the Minimum Time In Drive. This load time is depicted by the dashed portion of the Minimum Time In Drive line.

Before the Minimum Time In Drive expires, requests Ar2 and Ar3 are received and serviced. Notice that the Request Idle Time is reset on each request for the media in the drive.

At time t7, the request Br1 for media B is received. Since the drive is not available, this request is placed in the elevator queue. The Request Idle Time has expired at this point, but the Minimum Time In Drive has not.

At time t10, the Minimum Time In Drive has expired but the Request Idle Time begun for Ar3 is still in effect. At this same time, request Ar4 is received and the Request Idle Time is reset. As a result, media A remains in the drive for at least another 3 seconds. During this time media B receives another request, which is also queued.

At time t13, the last Request Idle Time expires. Because there are queued requests for other media, media A is removed from the drive and media B is inserted in its place. The elevator queue time starts over again.

Had Ar5 been received before the last Request Idle Time expired, it too would have been serviced. The processing of requests for media A would continue until the maximum Idle Time has been exceeded. At that point, media A would be removed so that HCSS can service queued requests on other media. If there are no other pending requests, media A remains in the drive to service subsequent requests.

Note in this example that there are two queued requests for media B when it gets placed in the drive. Because of request type precedence, if Br2 is a demigration request and Br1is not, Br2 would be serviced first even though it came after Br1.

Asynchronous Deletes

The Asynchronous Deletes feature of HCSS significantly decreases the latency (delay) a user may experience when deleting migrated files. The delay comes from waiting for the media to be placed in an available drive and from the overhead of deleting each file on the media. This delay can become annoying when a large number of migrated files are being deleted. With Asynchronous Deletes, the Support Module reports success for each delete request immediately, before actually deleting the file. The delete request is then queued and the file is deleted at a later time - usually just moments later, depending on other activity within HCSS.

Delete requests are queued in a hidden file named ASYNCDEL.QUE located in the root of the SYS volume. This file should never be modified or deleted by a user. No more than 32,767 asynchronous deletes can be queued in this file at a time. Asynchronous Deletes will be turned off if this limit is exceeded. ASYNCDEL.QUE can grow to nearly 5 megabytes in size, and it does not shrink after the queued deletes are serviced.

During "dead" time when files are not being demigrated, HCSS releases the delete requests queued in ASYNCDEL.QUE. At this point, the file deletion process continues as described above.

Asynchronous Deletes can be turned off by setting the HCSS Delete Through parameter to ON. With this parameter on, the delete request will not complete until it has been deleted from the MO media.

Protecting Media

During the time an MO cartridge is in a jukebox's drive, the gate protecting the media inside the cartridge is pulled away to expose the media. If the media is left in the drive for prolonged periods of time, it may be adversely affected by dust and other elements in the environment.

For this reason, instead of leaving media in the drive indefinitely, HCSS will remove media from the drive and place it in an available slot after 15 minutes of inactivity. The inactivity time is definable through the Secure Media Time parameter.

Multiple Drive Support

HCSS for NetWare 4.1 supports multiple MO drives in a jukebox. An elevator queue is in effect for each drive in a jukebox, and each elevator queue works independently of the others. However, the elevator queue settable parameters discussed above are global and are not settable on a per-queue basis.

Also with multiple drive support comes a certain measure of fault tolerance. Should a drive go bad, HCSS will attempt to remove the media currently in that drive and place it in an available slot in another drive. HCSS will no longer use the defective drive.

Handling Device Failures

Occasionally devices fail. Should an MO device fail while HCSS is reading or writing to the media, a message indicating the event is posted to the console screen and broadcast to user(s) ADMIN. HCSS will no longer use that device until the problem is corrected and HCSS is reloaded. If there are other devices available in the jukebox, HCSS will continue to operate; otherwise, migration and demigration will cease.

The defective device can be identified by the device number. The device number is posted in the console message that indicates a device was deactivated due to a device failure. Device numbers are sequential and should be in the same order as the SCSI IDs for the MO devices. By looking at the device number for the devices in the jukebox and comparing the order to the SCSI ID order, you should be able to easily identify the offending drive.

The Current Media debug parameter will replace the colon (:) found after the drive number with an exclamation point (!) for any failed drive. However, this drive number is not the same as the device number described above. The order of the drive listings should be in the same order as the SCSI Ids, though.

Decider Module (KDECDAEM.NLM)

The Decider Module for HCSS is responsible for the migration of file data to HCSS. It has three processes that periodically run to cause migrations: the Marking Process, the Polling Process, and the Warning Process.

Marking Process. The Marking Process scans the HCSS domain directories searching for candidates for migration. Candidates are placed on a list in a least-recently-used (LRU) order so that the most aged files are migrated first. Before a file is placed on the LRU list, it must first pass a number of criteria.

  1. For starters, the file must not be (1) already migrated, (2) currently opened, or (3) flagged Migrate Inhibit or Execute Only.

  2. In addition, the HCSS parameters Migrate Unarchived Files, Migrate Compressed Files Only, and Migration can augment the criteria for candidacy.

  3. Also, if a subdirectory is flagged Migrate Inhibit, all files in that directory and in subdirectories below the directory will not be placed on the LRU list.

The LRU list is complete when all the domain directories have been scanned. After the list is built, the Marking Process sleeps for a period defined by the HCSS parameter Marking Frequency. When this time expires, a new LRU list is built.

The number of files placed on the LRU list can be controlled by the Marked Files Limit parameter. By default, the Marking Process will place every candidate from migration on the list. However, since each file on the list uses 80 bytes of memory, this may place a heavy burden on memory. If Marked Files Limit is set to a value other than zero, only that many files will be placed on the list at a time. With a limit defined, the oldest file on the list - not necessarily the oldest file on the volume - is migrated first.

The Marking Process can be rather disk intensive while scanning a large number of files, say 100,000 files or more. It may become necessary to periodically increase the Marking Frequency as the number of files on the HCSS volume increases to reduce the frequency of intense disk activity.

Polling Process. The Polling Process wakes up when the period defined by the Polling Frequency parameter has expired. It determines the percentage of the HCSS volume capacity used by nonmigrated file data and compares that to the value defined by Upper Threshold parameter. If this threshold is exceeded, the Polling Process attempts to migrate the files on the current LRU list. The migration of the files on the LRU list continues until the Lower Threshold parameter is reached or the LRU list is exhausted. If the list is exhausted before reaching the lower threshold and the current volume capacity is now below the upper threshold, migration will not resume until the upper threshold is again exceeded.

The Decider can also be configured to migrate to the lower threshold once a day without exceeding the upper threshold. The time for this event is defined by the HCSS parameters Migrate To Lower Threshold Hour and Migrate To Lower Threshold Minute. At the defined time, the Polling Process will migrate the current LRU list. If the time is set to midnight (that is, both parameters are set to zero), the Polling Process will migrate the LRU list only if the upper threshold is exceeded.

Should a file fail to migrate, a message is posted to the server console screen indicating the failure and the file that failed to migrate. The reason for the failure may only be found in the log file, HCSSERR.LOG. The most likely reason for failure is that the media is full.

Note: Whenever the Migrate attribute for the HCSS volume is turned off and HCSS is attempting to migrate files, files will not migrate and error messages will accumulate on the server console screen.

The Polling Process and the Marking Process are mutually exclusive in operation. While one is running, the other is sleeping. The MONITOR utility's "Scheduling Information" screen can indicate which process, if either, is running. The process names markmain and pollmain identify the Marking and Polling Processes respectively.

Warning Process. The Warning Process wakes up when the period defined by the Warning Frequency parameter has expired. This process checks the percentage of remaining free space on each side of media. If this percentage is less than the value set in the Remaining Capacity Before Warning parameter, a message indicating the media that is getting full is posted to the server console screen and the error log file, and broadcast to user ADMIN.

The remaining free space is not just the number of free sectors. It is the total number of sectors on the side of media, minus the sum of sectors used and the number of migrated files. HCSS reserves one sector per migrated file to ensure that there is room on the media for the Info File when the media is exported. See the discussion on IMPEXP.NLM.

The Warning Process has a process name, updatemax, which can be monitored from the MONITOR utility's "Scheduling Information" screen.

Import/Export Module (IMPEXP.NLM)

IMPEXP.NLM is the media import/export module. It provides all the functionality to get media into a jukebox, format it, export it with migrated data on it, and later possibly reimporting the media while preserving its migrated data. This functionality is administered from NWAdmin, the Windows-based NetWare administrator utility.

In the context of HCSS, "importing" media is how media gets into the jukebox. "Exporting" is how media gets out of the jukebox. With a multidrive jukebox, import/export take place simultaneously with migration/demigration on other pieces of media. With a single-drive jukebox, migration and demigrations intended for other media are held off until the import/export completes.

Import/Format. All media in the jukebox must be formatted for use by HCSS. Formatting media is an option when media is imported. Formatting media resets the Used Sectors Bit Table, and creates the Stamp and the first Directory Segment. If the media contained migrated data before the format, the data will become unaccessible.The formatting process does not erase data that was on the media before the format. The data will be overwritten by future file migrations.

Note: Media must not be placed in the jukebox manually or by virtue of any application other than NWAdmin.

Once the media is formatted, it is assigned to a domain directory. The domain directory must have been created, via NWAdmin, before the media is imported.

Export. Before a piece of media is exported, HCSS makes sure the jukebox's mail slot is empty. If it is not, the export fails and a message is sent indicating that the mail slot is not available. Otherwise, HCSS proceeds with the export and attempts to migrate all files that are not currently migrated in the two media-label directory trees associated with the media. Should even a single file be unable to migrate, the export process will fail. Each file that could not be migrated will be posted to the server console screen and to the log file.

After all files are migrated, HCSS builds an Info File for each side of media. This Info File contains each migrated file's attributes and extended attributes. The Info Files are then migrated to the media. There will always be space on the media for the Info File because the Migration Process reserves one sector per migrated file for the Info File.

With the Info Files migrated, HCSS collapses the directory trees associated with the two sides of media. All files and subdirectories under the media-label directories, as well as the media-label directories themselves, are deleted from the volume. The media is then moved to the jukebox mail slot and the user is prompted from within NWadmin to remove the media from the jukebox.

Note: Media must not be removed from the jukebox manually or by virtue of any application other than NWAdmin.

Should an export fail, HCSS automatically reimports the media being exported, since files on either side of the media may have been deleted from the volume during the export.

Note: It is very important that a label be placed on the media cartridge identifying the labels placed on the media at format time.

Import/Assign. When exported media is reimported, the user is given the option to format the media or assign it to a domain directory. Assigning media to a domain directory recreates in that directory the media-label directory trees that existed for the media-label directories before the media was exported. Media can be imported to a volume of a server other than the server it was exported from.

HCSS scans the directory segments on each side of media, restoring the subdirectories and file entries to the HCSS volume. After the directory trees are restored, HCSS demigrates and processes each Info File. The Info File contains all the information necessary to restore the file attributes to their state before they were exported. The only exception is that the owner name for each file created when imported will be the server name.

Every attempt is made to restore the directory trees to their pre-export state. Should any type of failure occur during the import process, the user is informed at the end of the import process and the HCSS log file will contain a list of any affected files. Should a file's attributes fail to be restored, the date and time for that file are set to 00/00/00 and 00:00. Also note that empty subdirectories that existed before the media was exported are not recreated on import.

Note: During an import process, the two media-label directory trees are unaccessible to all users except ADMIN. When the import completes, this restriction is removed.

HCSS Snap-ins for NWAdmin

As mentioned earlier, there are six DLLs used by NWAdmin to administer HCSS. These DLLs are:

  • HCSSRES.DLL

  • HOBJSNAP.DLL

  • HNCPS.DLL

  • HMEDSNAP.DLL

  • HDECSNAP.DLL

  • HHLPSNAP.DLL

NWAdmin must be able to find these files. After a successful installation of NetWare, the DLLs will be in SYS:\PUBLIC, except for HCSSRES.DLL which should be in the appropriate language directory under SYS:\PUBLIC\NLS.

The NWADMIN.INI file must be modified to integrate these snap-in DLLs with NWAdmin. The following three DLLs must appear as shown under the category heading [Snapin Object DLLs]:

Vis HCSSDecider=hdecsnap.dll
Vis HCSSObject=hobjsnap.dll
Vis HCSSMedia=hmedsnap.dll

Executing NWAdmin with these snap-ins creates an extended list of options in the utility's "Tools" menu option. These additional options are used to create domain directories; export, import, assign or format media; and modify HCSS settable parameters.

Backing Up an HCSS Volume

Novell's SBACKUP utility can be used to back up and restore an HCSS volume in a variety of ways. If a file is migrated, SBACKUP can back up the migrated data, the DMKey, or both. If the file is not currently migrated, only the file data can be backed up.

Migrated data is backed up without demigrating the data. SBACKUP reads the data directly from the media via the RTDM. This allows the migrated data to be backed up without the data having to be moved onto the volume first. However, backing up a migrated file is considerably slower than backing up from the volume. It is recommended that files be backed up before they are migrated. (See the explanation of the Migrate Unarchived Files settable parameter.)

Backing up the DMKey significantly speeds up the backup process since each DMKey is less than 256 bytes and always resides on the volume. Since the data on the media is not affected by volume failures, only the DMKey needs to be restored. In the event of a catastrophic volume failure, access to the migrated data is regained by restoring the DMKey.

A typical backup scenario would be to back up file data before it is migrated, and at the same time back up the DMKey of files that have been migrated after being backed up in a previous session.

Note: A file will not have a DMKey until after it is migrated. A demigrated file will have a DMKey as well, but SBACKUP will ignore the key when doing backups since the file is not currently migrated.

SBACKUP verifies each DMKey with HCSS before anything is restored to the volume. If the DMKey is invalid and the file data was also backed up, the file data is restored to the volume instead of the DMKey. If the data was migrated at the time it was backed up, SBACKUP attempts to migrate the restored file.

HCSS Settable Parameters

The many settable parameters for HCSS come in two flavors:

  • Standard, documented parameters

  • Advanced, undocumented parameters

For the most part, the standard parameters modify the operations of HCSS, whereas the advanced parameters divulge information about what's going on inside HCSS. The HCSS parameters are not case-sensitive. The standard HCSS parameters are also settable from within NWAdmin.

Standard Parameters

Typing HCSS at the server console displays all the standard parameters and their current values. It also indicates the current state of HCSS (either Initializing or Up and running).

To modify a standard parameter, type at the server console HCSS followed by the parameter text and new value.

The standard parameters and descriptions are listed in Figure 5.

Figure 5: Standard HCSS parameters.


Parameter
Explanation

Eject MediaOverride = [ON | OFF] Default: OFF

During jukeboxinventory, HCSS ejects all invalid mediafrom the jukebox. If the media is actuallyintended to remain in the jukebox, settingthis parameter to ON will prevent HCSS fromejecting the media. A list of all media thatwould have been ejected is posted to theserver console and to the error log file.

Typically, the first piece of media will have alreadybeen ejected before the administrator knowsthere may be a problem. Setting this HCSSparameter from the server console beforeacknowledging the ejected media will preventany further ejections.

After the inventory completes and the HCSS is up and running. . . message is received, HCSS may beunloaded and the ejected media manually placedback into the jukebox. Correct the problem(for example, mount the volume, resolve anyhardware issues, or whatever), and reloadHCSS.

This parameter cannot be set fromNWAdmin.

Delete Through= [ON | OFF] Default: OFF

When turnedON, the Delete Through parameter disablesAsynchronous Deletes, and vice-versa. (Seethe discussion of AsynchronousDeletes earlier in this AppNote.)

MigrateUnarchived Files = [ON | OFF] Default: OFF

With this parameterset to OFF, HCSS will not migrate a fileuntil it has been backed up. More specifically,it will not migrate a file if the file'sArchive bit is set. There are two benefitsto leaving this parameter OFF. First is thecomfort in knowing that an archive of thedata exists before that data is migratedfrom a volume. Second, backup performanceis enhanced if the backup utility does nothave to demigrate files to archive them.

MigrateCompressed Files Only = [ON | OFF] Default: OFF

With this parameterON, a file will be eligible for migrationonly after it has been compressed. Thisfeature maximizes the storage capacity ofthe jukebox, but makes migration dependenton the compression system and its configuration.Files that cannot be compressed will alsomigrate. In short, with this parameter setto ON, only files that have the Co or Ccstatus are eligible for migration.

Note: A migrated file can not be compressed sinceits data does not reside on the volume.

MaximumTime In Drive = <value< Default: 30

This valuedefines the maximum amount of time, in seconds,that a piece of media will remain in a drivebefore it may be removed to service a requeston another side of media. This value mustbe larger than the value for Minimum TimeIn Drive. (See the discussion of ElevatorQueues earlier in this AppNote.)

MinimumTime In Drive = <value< Default: 20

This valuedefines the minimum amount of time, in seconds,that a piece of media will remain in a drivebefore it may be removed to service a requeston another side of media. A recommended valuefor this parameter is at least twice theamount of time it takes to get a piece ofmedia into a drive. This value must be smallerthan Maximum Time In Drive. (See thediscussion of Elevator Queues earlier in this AppNote.)

RequestIdle Time = <value< Default: 2

This valuespecifies the amount of time, in seconds,to wait for requests for media currentlyin a drive after the Minimum Time In Drive has expired before it may be removed to servicea request for another side of media.

After the Minimum Time In Drive has expiredand before Maximum Time In Drive expires,HCSS waits no longer than the RequestIdle Time for further requests for themedia currently in a drive. If a requestis pending on another piece of media andthe value expires, the current media maybe removed to service the pending request.(See the discussion of ElevatorQueues earlier in this AppNote.)

Migration= [ON | OFF | NOW] Default: ON

This parameterturns the Decider ON and OFF. If HCSS iscurrently migrating files, setting this parameterto OFF will stop migrations as soon as thecurrent migration completes. Migrations willcease until it is returned to the ON state.

The option NOW is an undocumented value. It causesthe Decider to stop the Marking Process ifit is running, and migrate the current LRUlist. The current LRU list may not be a completelist. If Migration was set to OFF,then NOW will set it to ON. (See the discussionof the Decider Module earlier in this AppNote.)

Upper Threshold= <value< Default: 80

HCSS will beginmigrating file data when <value<%or more of the volume capacity is used upin nonmigrated file data. (See the discussionof the Decider Module earlier in this AppNote.)

Lower Threshold= <value< Default: 50

Data migrationstops when either the LRU list is exhaustedor <value<% or less of the volumecapacity is used up in non-migrated filedata.

MigrateTo Lower Threshold Hour = <value< Default:3Migrate To Lower Threshold Minute= <value< Default: 0

These two parameterswork in concert. They define the time ofday that HCSS should migrate to the lowerthreshold. This migration will occur evenif the upper threshold is not reached.

This automatic, once-a-day migration to the lowerthreshold is disabled when both parametersare set to zero.

RemainingCapacity Before Warning = <value< Default: 20

HCSS will beginposting messages when a side of media has <value<% or less capacity remaining.The messages are posted to the server consoleand the log file, and broadcast to the user(s)ADMIN every Warning Frequency value.The message will identify the media thathas exceeded the remaining capacity limit.

PollingFrequency = <value< Default: 1

This valueindicates how long, in minutes, the PollingProcess should wait between checking to seeif the Upper Threshold has been exceeded.Since migration occurs when the upper thresholdis exceeded, extending this amount of timedecreases the resolution of migration events.

MarkingFrequency = <value< Default: 30

This valuedefines how long the Marking Process waitsafter building an LRU list to begin buildinganother. This is not to say that a list willbe built every <value< minutes.Depending on the number of files under theHCSS domain, the time required for constructionof the LRU list may be lengthy. This valuesimply defines the time between LRUlist builds. Building an LRU list may alsobe disk intensive. Therefore, it is suggestedthat this value be periodically increasedsomewhat as the number of files under theHCSS domain increases.

WarningFrequency = <value< Default: 2

This valueindicates how often, in minutes, HCSS willpost messages indicating a side of mediahas exceeded the Remaining Capacity BeforeWarning limit.

Marked FilesLimit = <value< Default: 0

This parameterdefines the maximum number of files thatcan be placed on the LRU list at any onetime. When this limit is reached, the MarkingProcess stops and the Polling Process beginsthe migration of the files on the list.

There are rare occasions when it may be desiredto limit the number of files placed on theLRU list. One occasion is when a large number(tens of thousands) of files have been placedunder the HCSS domain in a very short amountof time. Placing a limit on the list mayprevent the exhaustion of memory resourcesand expedite the migration of files.

Note:If this limit is less than the number offiles that cannot be migrated, no files willbe migrated.

A value of 0 indicates nolimit and the Marking Process will searcheach domain directory tree looking for migrationcandidates.

Advanced Parameters

In addition to the standard HCSS parameters, there are some advanced and debug settable parameters for HCSS. These parameters are useful when monitoring HCSS performance or debugging HCSS.

Note: These additional parameters are not documented outside of this AppNote and may change at any time without notification.

Typing HCSS DEBUG at the server console displays all the advanced parameters and their current values. To modify these parameters, type HCSS DEBUG followed by the parameter text and the new value.

The advanced parameters are defined in Figure 6.

Figure 6: Advanced HCSS parameters (undocumented).


Parameter
Explanation

Deny AllRequests = [ON | OFF] Default: OFF

Turning thisparameter ON will stop all media activity. Requests for migration, demigration, deletesand renames will be denied.

SucceedAll Deletes = [ON | OFF] Default:OFF

When this parameteris set to ON, all requests to delete a filefrom the media will be immediately succeededand never deleted from the media. This parameteris useful in some recovery operations. Itallows a migrated file to be deleted withoutthe media being in the jukebox, so long asthe support module is loaded.

Information= <value< Default: 0

This parameterturns on a variety of debug information regardingevents inside HCSS and the times associatedwith these events. This information is postedto the server console only. The <value< is a bit map used to indicate the type ofinformation to be posted. It must be at least0x01 and may be ORed with the modifiers.

0x01

Post createevents (migrations), open events (demigrations),deletes, and renames to the server console.The file associated with the event is identified.

0x10

A modifierto post a time stamp, including date andtime, of the event being posted.

0x20

A modifierto show the delta time between events.

For example, typing AHCSS DEBUG Information = 31" will post migrate, demigrate, rename, and delete events; the delta time between these events; and the time of day the event occurred to the server console.

Secure Media Time = <value< Default: 15

This parameter defines the amount of time, in minutes, that the media must remain idle before it is removed from the drive and placed back into a slot. The greater this value, the longer the media remains exposed to the environment since the protective gate on the cartridge is pulled away while the cartridge is in the drive.

Number Of Marked Files

This command has no value and provides information only. It simply returns the number of files currently placed on the LRU list. This list includes files that may have never been migrated or that needed to be remigrated. This list is rebuilt each Marking Process.

Additional Parameters

There are two other undocumented HCSS parameters/commands that are useful. These are typed at the server console using the syntax shown in Figure 7.

Figure 7: Miscellaneous HCSS parameters.


Parameter
Explanation

HCSS DEBUG Current Media

This parameter reports which media is in which drive in the jukebox. If the drive is empty, it willbe reported as such. This parameter also identifies a failed device by replacing thecolon (:) that separates the drive numberfrom the media label with a exclamation point (!).

HCSS MediaStatus = <full path to media-label directory<

This parameterreturns some statistics for the specifiedside of media. These statistics include thetotal capacity of the media in bytes, theused space in bytes, and the number of filesmigrated. The <full path to media-labeldirectory< is the full path to themedia-label directory. For example, referringback to Figure 2, one such path would beHCSS:\HISTORY\MAY.

Frequently Asked Questions

Q. Why is HCSS ejecting media during inventory?

A. There are a variety of reasons why HCSS would eject media during inventory. Some likely reasons are:

  • The HCSS volume is not mounted.

  • The HCSS volume has been recreated.

  • Media was manually loaded into the jukebox.

  • Domain or media-label directories have been deleted or renamed.

  • Device or driver errors have occurred.

The HCSS settable parameter HCSS Eject Media Override can be set to ON if you don't want HCSS to eject media from the jukebox.

Q. How do I stop "Media . . . exceeded remaining capacity limit" messages from being broadcast?

A. These messages will be broadcast so long as sides of media have free space less than the amount specified in the Remaining Capacity Before Warning parameter. These messages may become annoying. To eliminate them, move files out of the media-label directory tree or decrease the Remaining Capacity Before Warning value. Increasing the Warning Frequency parameter will not eliminate the messages, but it will make them less frequent.

Q. How can I get statistics on migrated files?

A. Both SERVMAN.NLM and NWAdmin can report the number of migrated files on a volume.

  • In SERVMAN, select the "Volume Information" menu option and then the HCSS volume. A window filled with statistics is displayed. Among the information is a Migrated files: value, which indicates the number of migrated files on the selected volume. Note that as the number of files migrated increases, so does the Extended directory space: value,also shown in this window. This is because the DMKey for each migrated file is stored in extended directory space.

  • From NWADMIN's browser, highlight the HCSS volume object. Select Details from the "Object" menu and then Statistics from the "details" dialog box. A box at the bottom center indicates the number of migrated files on the selected volume.

Getting the number of files migrated per side of media is accomplished by using the HCSS command HCSS Media Status, by using NDIR, or with the help of some NetWare function calls available to an application running at the client.

  • For details about using the HCSS command HCSS Media Status, see the discussion above.

  • Invoking NDIR with the "/M" option displays only those files that have been migrated. Typing "NDIR *.* /M /S" at a media-label directory lists all files currently migrated to the corresponding side of media. Use "/NOT M" to display those files not migrated.

  • The NetWare Client SDK contains several APIs for file migration. These APIs are defined in the Data Migration Services section of the client SDK manuals.

Q. Why does a file fail to demigrate?

A. If a file fails to demigrate, the original Open request that caused the demigrate will fail. The manifestation of this failure varies with the client operating system and the application requesting the migrated data.

Some of the more likely reasons for this failure include the following:

  • HCSS is not loaded.

  • There is no room on the volume for the file being demigrated.

  • The media containing the migrated file data is not in the jukebox.

  • A hardware failure has occurred.

  • The Deny All Requests parameter is set to ON.

The HCSS log file, SYS:\HCSSERR.LOG, should contain an entry indicating the cause of the failure.

Q. Why is HCSS not migrating files?

A. Some likely reasons for files not migrating include the following:

  • The Migrate volume attribute is not set to ON.

  • The Upper Threshold is not exceeded.

  • Migrate Unarchived Files is set to OFF and files are not archived.

  • Migrate Compressed Files Only is set to ON and files are not compressed.

  • Files are flagged Don't Migrate.

  • The files expected to migrate are outside the HCSS domain (that is, the files are not under a media-label directory).

  • Full media

Q. Why is the extended directory space increasing?

A. Each migrated file causes the allocation of some extended directory space for the DMKey. When a migrated file is deleted, the extended directory space is not relinquished. This space is not relinquished until the volume is dismounted and remounted. SERVMAN.NLM shows the amount of extended directory currently in use via the "Volume Information" option.

Remounting the HCSS volume occasionally frees up available extended directory space. It also improves file access performance since the volume has a chance to rebuild its hash tables.

Q. Why is NWAdmin having trouble with HCSS?

A. There are a couple of likely reasons why NWAdmin might fail to work correctly with HCSS:

  • If the "Tools" menu does not show the "HCSS" submenu option, make sure NWADMIN.INI is properly set up for HCSS.

  • If NWAdmin is complaining that the HCSS DLLs could not be loaded, make sure the DLLs are in SYS:\PUBLIC and that this directory is included in your PATH. If NWAdmin is not executed from SYS:\PUBLIC, place the HCSS DLLs in the directory from which NWADMIN.EXE is executed.

  • Server names and domain directory names containing underscores ( _ ) cause the HCSS snap-ins to work improperly.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates