Novell is now a part of Micro Focus

Features of the Novell Kernel Services Programming Environment for NLMs: Part Three

Articles and Tips: article

RUSSELL BATEMAN
Senior Software Engineer
Server-Library Development

01 Nov 1999


Covers programming interfaces for writing NLMs, including virtual machine management services, memory allocation and management, file and directory I/O and time-out services. Part three in a series of articles on the programming features and functionality of the Novell Kernel Services (NKS) programming environment.

Introduction

This is part three in a series that reviews the programming features and functionality of the Novell Kernel Services (NKS) programming environment from the perspective of my organization whose task it is to ensure developers have tools and technologies at their disposal for writing applications that run as NetWare Loadable Modules (NLMs) on NetWare, Novell's network operating system.

Part One explored the reasons for this recently-defined programming model in the NetWare Loadable Module (NLM) environment and it examined in some detail the various interfaces surfaced by NKS.NLM.

Part Two focused more closely on the programming concepts in the Novell Kernel Services environment including discussions of multithreaded programming correctness, latency and thread cancellation issues. Specifically discussed were the thread and context management primitives, synchronization including mutexes, read-write locks, condition variables and semaphores, and issues of per-thread data.

Part Three, this article, finishes the concepts started in Part Two by covering the remaining programming interfaces for writing NLMs. This includes the virtual machine management services, memory allocation and management, file and directory I/O and time-out services. Some discussion of platform-dependent and debugging interfaces is included.

Part Four of the series will discuss how LIBC sits atop NKS and make comments about programming at both levels. In addition, it will make pertinent remarks about differences in programming to LIBC as compared to CLIB.

In another article in Novell Developer Notes, I explained how the ten-year old CLIB programming environment had reached a fork in the road in its attempt to continue to support existing NLMs coded to it and at the same time move forward to embrace support for future multithreaded programming technologies. I also explained the reasons why we are freezing CLIB in its current state (plus bug fixes as they are needed) and move ahead with a new environment based on Novell Kernel Services (NKS) plus a standard C library environment (LIBC) loosely referred to as NKS/LIBC. As just noted, Part One of this series covered at the 64,000 foot level, the services offered by NKS. Part Two covered NKS thread programming in some detail. Now we will fill in the rest of the NKS feature story by discussing the remaining functional aspects beginning with Virtual-machine Management.

Note: At the risk of being misunderstood, let me clarify just what I mean by "freezing CLIB." Bug fixes are going to continue to happen just as before and with no de-emphasis because, after all, the majority of NLM software in existence currently depends on this library. What is not going to happen is a continual technology bulking in the form of added APIs, enhanced thread-programming technology, and the sorts of improvements that permitted, for instance, the Java Virtual Machine (JVM) to be ported atop CLIB by means of an enhanced process model. Instead, that effort will be directed at promoting NKS programming and making the necessary improvements to this new library and LIBC that will permit porting those same components, the JVM, Web servers and other new applications, to the new environment.

In Part Two, we established some a glossary of thread-programming terms. Some of that glossary will continue to apply to Part Three.

Virtual-Machine Management and Containment

In the last article, we mentioned a containment concept, the virtual machine, without getting into a definition of what it is or what purpose it serves on NetWare. Containment is the separation between complex executing entities like applications. It protects the entities from each other's antics such that serious mistakes on the part of one have no effect upon the other.

In most operating systems with a formalized opposition of user and kernel programming environments, containment is clean, and formal interprocess communication (IPC) mechanisms are required when crossing containment boundaries for the purpose of communicating or sharing information between entities. This is called process-style containment.

There is no process-style containment in NetWare, and, traditionally in fact, what containment there has been is fairly loose. As examples of containers in NetWare, we cite the NLM, whose resource tag collection permits the system to enforce accountability for resources used and not freed. There are resource tags for the threads an NLM creates, for its semaphores, allocated memory, etc. When an NLM allocates memory and doesn't free it up, NetWare issues a "nasty-gram" at unload because it knows about it. These are all manifestations of NetWare's notion of containment. However, nothing requires a thread to behave as if it were owned by its creating NLM, and the threads the NLM's start-up thread might create don't have to continue bound to that NLM and its code, although if an NLM unloads before terminating all its threads, an AbEnd occurs. Except for the NetWare 5 concept of a user address space, containment on NetWare never means protection from entities that go awry.

CLIB's containment is more formal and slightly better enforced, but many if not most CLIB applications share threads between themselves, and, in any case, these threads often run through other NLM code to access widely diverse services. CLIB's containment also fails to protect the operating system from NLMs and NLMs from each other.

The point is that there is a lack of formal containment on NetWare as compared to the process-containment models of UNIX or Windows NT. This makes NLM programming challenging, as noted in the kick-off article to this series, "The Future of Application Development on NetWare with NLMs," which appeared along with "Features of the Novell Kernel Services Programming Environment for NetWare 6 Pack Programming, Part One" in the September 1999 issue of Novell Developer Notes.

This problem of containment hasn't really been solved; indeed, much of NetWare kernel programming and NLM development relies on the weak containment. NLMs were originally conceived as an easy way to extend the operating system; it was only in the second breath that their ability to express network application functionality was realized and exploited.

For the reasons of correctness and scalability covered in these articles, the lack of containment has been tackled. Novell Kernel Services' (NKS) Virtual-Machine Management Services interfaces define a containment within which a portable service may be hosted. This means, basically, that an NKS virtual machine (VM) is created for each NKS application and that there are interfaces to tailor that environment. On Modesto and other NKS-equipped platforms, that containment is rigid and enforced because the design of Modesto is nothing like NetWare. Containment issues were a priority in Modesto's design.

Meanwhile, NetWare doesn't enforce containment, and NKS can't go very far to help. Consider, for example, the subdiscussion in the last article on thread cancellation. As pointed out, NetWare programmers can't rely on rigid containment and must manage thread migration explicitly when entering and exiting foreign, especially non-NKS, NLMs for their services.

It is anticipated that most applications written to NKS will want a more rigid containment like CLib's or better rather than the looser environment of underlying NetWare as already described and so NKS provides it. But, the limits to the containment that can be imposed must continue to be recognized and compensated for by the NetWare developer. Nevertheless, the protection aspects of containment are available to NKS/LibC-based NLMs through the use of the NetWare 5 protected address space just as they are to CLib-based ones. That is, an NKS application loaded in its own address space gains much of the protection traditionally afforded by the process-style containment of other operating systems.

Virtual-Machine Features

The virtual machine supports a multithreaded execution environment in which the NKS threads share memory, the native platform security context, open file descriptors, file and record locks, and an owned pool of threads. The virtual machine may be transformed into a deterministic or controllable state machine by disabling preemption and limiting the concurrency. The virtual machine as defined by NKS is expected to map rather naturally onto the typical task or process abstraction found in most operating systems, but with the caveats already noted when running on NetWare.

NKS doesn't surface interfaces to create virtual machines (look forward to the spawn interfaces of LIBC) nor does it support intervirtual machine communications (any sort of IPC). The decision not to do this was motivated by the desire to keep NKS platform-unspecific in its design and to leave such features to the higher-level libraries of the individual platform. The last article in this series will discuss to what extent and how interprocess communication (IPC) has been implemented atop NKS by LIBC and how this solves former problems of cooperative relationships between NLMs coding to CLIB.

The features of the VM interfaces of NKS include getting the number of online processors (on multiprocessor hardware), getting the identity of the CPU on which the thread is executing, disabling preemption, terminating the current virtual machine (like ANSI exit), managing the worker thread pool, and transforming the VM into a singly threaded machine.

Some of the interfaces are straightforward. For example, calling NXGetCpuId returns the identity of the processor executing the calling thread. If the hardware executing the thread has only one processor or only one processor is presently online, then the value returned will of course be 0 (for processor 0) since NetWare still has vestiges of processor-0 centricity. Among the thread management interfaces, while not discussed with them in the last article, NXThreadBind exists explicitly to provide hard affinity: where a thread is bound on a certain processor and will never execute on any other.

Other of these interfaces are less obvious in their use. Some exist really to emulate some of the useful execution characteristics of NetWare that make it a fast protocol engine and might be used by code ported from NetWare to another platform like Modesto.

The worker thread pool functions exist to give hints to the implementation as to the number of worker threads the application might want to use in the course of its execution. An implementation is free to ignore this, but it might be very useful on still other implementations. At the very least, it is a statement by the engineer as to one important aspect of the profile of the application.

In summary, the NKS virtual machine provides:

  • multithreaded and multiprocessor programming environment

  • a time-slicing environment, by default

  • thread execution priority where supported (not on NetWare)

  • shared file and directory handles

  • a per-VM worker thread pool

Memory Allocation and Management

While no assumption can be made about the existence of a virtual memory subsystem underneath the NKS implementation, NKS on NetWare does provide the only published interface to NetWare 5's virtual memory subsystem. Obviously, this isn't accessible from NetWare 4 which doesn't support virtual memory. Portability, as well as performance, encourage the application to manage its memory very carefully.

The basic memory-management functions allocate, reallocate, and free memory much the same as malloc, realloc and free. However, they accept an argument to specify alignment requirements which, on some platforms, may be an important performance variable.

#include <nks/memory.h>

void *NXMemAlloc( NXSize_t bytes, NXSize_t alignment );
void NXMemFree( void *memory );
void *NXMemRealloc( void * old, NXSize_t bytes, NXSize_t alignment );

In addition to these simple functions, others exist to allocate whole pages of memory and still others to control the state of that memory whether locked down or not.

#include <nks/memory.h>

void *NXMemCntl( void *start, NXSize_t pages, unsigned long flags );
void *NXPageAlloc( NXSize_t pages, NXBoolean_t lockIt );
void NXPageFree( void *memory );

The page allocation functions don't fail without a virtual-memory subsystem present, they just allocate large chunks of locked-down, "normal" memory.

Platform-Dependent and Debugging Interfaces

Because NKS was meant to be portable across several platforms and because it was to have only very reduced and unambiguous semantics, it is somewhat of a lowest-common denominator, with minimal interfaces to achieve the features sought by most applications. It goes without saying then, that, as defined, NKS won't solve specific problems of a given platform, in our case NetWare. Each platform of implementation has added (or may add) interfaces to shore up two areas.

The first area is debugging. There are only a couple of explicit debugging interfaces in NKS, as shown below. The first one is used to implement dead-lock detection among mutexes or read-write locks. Debugging synchronization was mentioned briefly in the last article. At this writing, the interface was still under development, so specific details will be had only in the documentation when it is finished. This is one of the very few NKS interfaces that hasn't been completely nailed down.

#include <nks/synch.h>

typedef struct
{
    char             name[NX_MAX_OBJECT_NAME+1];
    unsigned long    flags;
} NXLockInfo_t;

#ifdef NDEBUG
# define NX_LOCK_INFO_ALLOC( var, name, flags) \
           NXLockInfo_t var = { name, flags }
#else
# define NX_LOCK_INFO_ALLOC( var, name, flags)
#endif

Basically, this structure may be optionally passed to a number of the synchronization functions exposed in Part Two of this article series. This names the lock and gives it some characteristics that will be ultimately defined to help the library point out problems in an application's lock-acquisition hierarchy when compiled for debug. There is no overhead for any of this unless the application code is compiled for debug.

The second interface imitates ANSI strerror and returns, for a given error passed to it, a pointer to a string containing wording similar to what strerror returns for values of errno. While most all other strings in NKS are Unicode, this one isn't because it is for debugging and the strings passed back will never be translated, just as strerror's counterpart strings in LIBC and CLIB aren't translated into locale-specific wording since it is likely their translation would obscure rather than elucidate the error for the engineers using it.

#include <nks/debug.h>

const char *NXStrError( int error );

By definition, however, names prefixed with uppercase NX are usual, portable NKS interfaces. Each platform is free to augment the standard ones with its own, though the Portable Execution Specification document on which NKS is based encourages platform implementations to minimize the number of supplementary interfaces offered. According to the specification, when a nonportable interface is offered, it must be prefixed with lowercase nx.

NKS for NetWare offers a number of additional interfaces to solve problems of thread cancellation. These were discussed at some length in the last article and include nxCancelDisable, nxCancelEnable, and nxCancelCheck. In addition, NKS for NetWare offers a few additional functions for routine NetWare concerns as well as debugging. These are the following:

#include <nks/netware.h>

int nxGetStartUpParameters( ?a very long argument list? );
int _nxAssert( int abort, const char *expr, const char *fName, int lineNo );
int _nxPrintError( int abort, const char *msg );

#define nxAssert(abort, expr)     ((void) ((expr) |
               || (_nxAssert(abort, #expr, __FILE__,__LINE__), 0))
#define nxPrintError(abort, msg) _nxPrintError(abort, msg \
               ?\ from %s on line %d\n?, __FILE__, __LINE__)

nxGetStartUpParameters retrieves all the start-up details for the NLM which is sometimes useful to Novell-written and other system-level NLMs.

The other two interfaces are macros that furnish an enhanced assertion interface for debugging purposes. nxAssert is a macro interfacing a mechanism which will print a message to the screen based on the same criteria and content as ANSI assert. In addition, it accepts a separate argument to tell whether the application should abort at that point (or not), something ANSI's assert gives no control over. While assertion prints only conditionally to the screen, nxPrintError always prints a message, but again, will abort or not as desired by the programmer.

Printing interfaces are also added by NKS for NetWare and behave closely to ANSI printf except that they support no floating point (%e, %g, etc.). They do, however, support a number of additional conversion specifiers. The documentation to use is that for printf in LIBC (not CLIB).

#include <stdarg.h>
#include <nks/netware.h>

int nxPrintf( const char *format, ... );
int nxSPrintf( char *string, const char *format, ... );
int nxVPrintf( const char *format, va_list args );
int nxVSPrintf( char *string, const char *format, va_list args );

At this writing, an application could create and maintain application- (or VM-) wide clean-up functions &agrave; la atexit. Beyond mere atexit functionality, these could be added or removed under the complete control of the programmer any time before the application unloaded. As of this writing, these functions were being discussed for mainstreaming in NKS itself and will probably turn up there in some form at least as sophisticated as atexit . The admission here completes the list of interfaces still in movement in NKS.

Platform-specific interfaces could be considered an afterthought and of small importance; however, as we'll see in discussing file and directory I/O, platform-specific additions to NKS cannot be overlooked.

File and Directory I/O

The concept behind file and directory I/O for NKS was that the interfaces be portable, that is, common to all platforms, inherently multithreaded (atomic), and succinct. It is thought that most services would find them adequate if somewhat less functional than their POSIX or ANSI equivalents. In the last article in this series, we'll discuss how LIBC (POSIX and ANSI) interfaces are built atop these NKS functions.

The interfaces for file I/O are as follows:

#include <nks/fsio.h>

int NXCreateFile( NXDHandle_t dirHandle, const unicode_t *path, NXMode_t mode,
        NXFHandle_t *fileHandle );
int NXOpenFile( NXDHandle_t dirHandle, const unicode_t *path, NXMode_t mode,
        NXFHandle_t *fileHandle );
int NXCloseFile( NXFHandle_t fileHandle );
int NXRead( NXFhandle_t fileHandle, void *buffer, NXBufSize_t bytesToRead, 
        NXSize_t offset, NXSize_t *bytesRead );
int NXWrite( NXFHandle_t fileHandle, void *buffer, NXBufSize_t bytesToWrite, 
        NXSize_t offset, NXSize_t *bytesWritten );
int NXGetFileLength( NXDHandle_t dirHandle, const unicode_t *path, 
        NXSize_t *length );

Clearly, these are simple interfaces with the exception of the directory handle passed to the open function, which we'll explain in a moment. Notice that the read and write functions require an offset from which to start. In this, they are like POSIX pread and pwrite: these atomic operations avoid the problems of safeguarding a current file pointer against multiple threads performing I/O on the same file. That is, the problems from the library's point of view are solved; multiple-thread access of the same file is still problematic from the application's standpoint. The fact that the library doesn't have to set the safeguards in place frees it and the application from all sorts of misunderstandings, as implicitly discussed in the previous articles in this series. Practically speaking, the library maintains no mutex that could block when these functions are called from more than one thread simultaneously. Therefore, the library doesn't contribute to the opportunities an application has to find its threads killed, shutdown, or otherwise canceled while holding a lock an occasional problem in CLIB and still a problem in LIBC when using the thread-unsafe calls fread, fwrite, etc. The problem of simultaneous, multithread access is still very real and one the developer must solve.

In addition to the simple file I/O functions, there are more that create, delete and otherwise operate on directories as well as files. These are the following:

#include <nks/fsio.h>

int NXCreateDir( NXDHandle_t dirHandle, const unicode_t *path, 
          NXDHandle_t *dirHandle );
int NXCloseDir( NXDHandle_t dirHandle );
int NXRemoveDir( NXDHandle_t dirHandle, const unicode_t *path );
int NXRemoveFile( NXDHandle_t dirHandle, const unicode_t *path );
int NXGetAttributes( NXDHandle_t dirHandle, const unicode_t *path, 
          NXFileDirAttr_t *attributes, NXBufSize_t *xAttrSize );
int NXSearchDir( NXDHandle_t dirHandle, const unicode_t *path, 
          NXDirEntry_t **buffer, int numEntries, void *sequence, int *entryCount );

The directory handle passed to the open function merits explanation. Since NKS is completely unstateful as a library, that is, since it does not maintain information such what the current directory relative to which a path is to be interpreted as is evidenced in CLIB and LIBC by chdir, getcwd, etc., the information passed to the open call must somehow specify everything about the path that open needs to know to identify the file discretely. While this could merely be a full path and indeed, it can be that approach would unnecessarily saddle the application with the duty of maintaining a current directory as a string and spending time concatenating and in other ways building complicated path and name strings before calling open functions.

This concept of directory handle is based on a two-part, but three- or four-way view of any path to be maintained and passed to library open functions. The following tables illustrate the possible combinations that may be used to identify a file or directory using a directory handle and string path or name. First, consider specifying files. There are three ways to do this.


When the directory handle indicates...

the name consists of...

nothing because it is nil

the full path including the file name

the immediate parent directory

only the file name

an arbitrarily-distant parent directory

the rest of the path including the file name

And for indicating directories to NKS, there are four ways although, obviously, the fourth method is meaningless if calling NXCreateDir since the directory won't already exist.


When the directory handle indicates...

the name consists of...

nothing because it is nil

the full path including the directory name

the immediate parent directory

only the directory name

an arbitrarily-distant parent directory

the rest of the path including the directory name

the very directory itself

nothing; it is nil

Despite NXCreateDir and NXOpenDir returning directory handles, there remains a chicken-and-egg difficulty in getting into a file system in the first place. This "priming of the pump" is a platform-specific problem and involves questions such as discovering the existence of a file system, knowing what name space it supports, or even having the right to be in the file system in the first place. The answers must be found on a platform-by-platform basis, depending on how complex file system access is. NetWare is the most complex since traditionally it exposes the local DOS file system, an almost unlimited number of local, NetWare volumes each with from one to four name spaces, remote file systems on other NetWare file servers accessed via NetWare-core Protocols (NCP), and the nascent NetWare Storage Services (NSS) file system.

We are not going to deal with these questions in complete detail here. Instead, the documentation for NKS (in particular, NKS for NetWare) will bear that burden. Nevertheless, we will illustrate the solution which is in the form of a NetWare-specific interface, nxMakeDirHandle.

#include 

int nxMakeDirHandle( int connection, const unicode_t *volume, const 
        unicode_t *path, int namespace, NXDHandle_t *dirHandle );

The connection must be an authenticated one. The path is the only optional argument; it is optional since specifying the volume alone implies that the resulting directory handle will point to the root of that volume.

Once a directory handle is created, it may be used in combination with a name or path just as the current working directory is implicitly present in all file and directory path operations in POSIX and ANSI.

Directory handles are important resources. At least, they represent some memory use. On NetWare, they also represent a connection that is a very precious commodity. Consequently, it is unthinkable to discard them without first calling NXCloseDir, even when exiting an application.

A note on the last argument to NXGetAttributes: the extended size indicated concerns file-system specific (and, therefore, platform-specific) information that could be passed back if the buffer supplied were made big enough. We are thinking of revisiting this interface to make it simpler in this aspect. Each platform will document the potential of this interface and the information it might supply. In NetWare's case, the archive timestamp, the name space or the entry name in different name spaces are part of this information.

NKS file and directory I/O isn't quite so primitive as these last few paragraphs have made it out to be. It also includes notions of transactional file I/O and provides the only asynchronous file I/O interfaces available in the NKS/LIBC environment. Exposing these interfaces is left to the documentation.

Timeout Services

Before discussing timeouts, a quick note: In addition to timeouts, these services also provide a Swiss-Army knife time function that returns units since a specifiable epoch either boot or 1970 and is good for simple time-difference calculations. We suspect, however, that traditional ANSI time interfaces will be preferred for serious calendar time operations. It is accurate, nevertheless, and based on the same underlying information from the kernel and hardware. The name of the interface, which isn't shown here, is NXGetTime.

The timeout interfaces are the ones that really interest us in this article. They are for one-shot as well as cyclic uses and are very simple. The application supplies a function that will be executed as a call back by the system once the timer expires or the timeout as expressed has otherwise reached the appropriate moment.

The two following interfaces implement the time-out functionality in NKS:

#include <nks/time.h>

int NXTimeOutSchedule( NXTimeOut_t *tOut );
int NXTimeOutCancel( NXTimeOut_t *tOut, NXBoolean_t wait, NXBoolean_t 
        *status );

The timeout structure, NXTimeOut_t, contains fields to be filled in with the call back function's address, an argument to be passed to it when it is called and which might help identify which timeout is firing, and information about the type of timeout being set up including period and/or expiration.

Moreover, a timeout already set up may be canceled if it is determined that it should no longer fire. The behavior is tightly defined and well documented. When canceling a timeout, it is possible to cause the canceling thread to wait until the time-out has been successfully canceled.

Additional information as to when the timeout was successfully canceled is available too through the status argument.

Sample Code

We have written some example code for common operations in NKS and published this code with the individual NKS function documentation on the web in the SDK. The URL is http://developer.novell.com/ndk.

Conclusion

These are the remaining interfaces in Novell Kernel Services (NKS) that weren't covered in part 2, albeit with details especially oriented toward NetWare programming since that is the main thrust of the article series. Except where specifically noted, as in the case of the platform-dependent interfaces, most of the details hold for all platforms. The next installment of this article also the last will cover the relationship between NKS and LIBC, which sits atop the former, and the differences in programming to LIBC and NKS, and make relevant comments about and comparisons to traditional CLIB programming.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates