Features of the Novell Kernel Services Programming Environment for NLMs: Part One

Articles and Tips: article

RUSSELL BATEMAN
Senior Software Engineer
Server-Library Development

01 Sep 1999

First in a four-part series of articles which will review the programming features and functionality of the Novell Kernel Services (NKS) programming environment for NLMs. This article explores the purpose of this recently defined programming environment and examines the various interfaces surfaced by NKS.NLM.

Introduction
Novell Kernel Services for NLM Programming
The NKS Virtual Machine
Transport (Communications) Interfaces
Conclusion

Introduction

Novell's Server-Library Development group ensures that developers have tools and technologies for writing applications that run as NetWare Loadable Modules (NLMs) on NetWare. This series of articles reviews the programming features and functionality of the Novell Kernel Services (NKS) programming environment for NLMs. This article explores the purpose of this recently defined programming environment and examines in some detail the various interfaces surfaced by NKS.NLM.

Part two focuses more closely on the programming concepts in the NKS environment, including discussions of multithreaded programming correctness, latency, and thread cancellation issues.

Part three completes the concepts started in part two by covering the remaining programming interfaces.

Part four discusses how LIBC sits atop NKS and comments on programming at both levels. In addition, it discusses differences in programming between LIBC and CLIB.

Novell Kernel Services for NLM Programming

In "The Future of Application Development on NetWare with NLMs," in this issue of Novell Developer Notes, I explain how the ten-year old CLIB programming environment had reached an unfortunate milestone in its attempt to continue to support existing NLMs coded to it and at the same time move forward to embrace support for future multithreaded programming technologies. I also explain the reasons why we plan to freeze CLIB in its current state (plus bug fixes) and move ahead with a new environment based on NKS plus a standard C library environment (LIBC) loosely referred to as NKS/LIBC. Rather than reproducing those explanations here, I would like to revisit the history and motivation behind this new environment.

For the purposes of our discussion, LIBC on NetWare (LIBC.NLM) remains essentially identical to age-old CLIB.NLM. It will contain the ANSI interfaces and include as many POSIX interfaces as appear relevant, which should include most interfaces already long-surfaced by CLIB, name for name.

However, NKS surfaces new interfaces that are more complete, more correct, and more numerous, and replace the entire threading model of CLIB. What's more, all the interfaces in NKS lead users to a more correct implementation of multithreaded code not that the programmer cannot make mistakes with these interfaces, but that the interfaces are complete and semantically correct. NXThreadJoin, an interface providing fuctionality impossible in CLIB, offers an example of completeness; NXThreadSuspend offers an example of correct semantics, not allowing itself to be suspended as did CLIB's SuspendThread.

Originally, NKS was conceived as a portability layer (NPL) posited to support a broad set of architectural underpinning for all of Novell's services and applications written in C or C++. Whatever the internal use of this environment, the API set became more commonly associated with the next-generation platform code-named Modesto, where it is the interface to the operating system kernel. The API set was redubbed with its present moniker.

As noted in the other article, NKS promotes a set of threading interfaces that are more sophisticated and correct than those in CLIB. Indeed, it was decided to adopt the erstwhile NPL interfaces in replacement for CLIB's after examining other alternatives such as POSIX pthreadsand UNIX International. These were the most important interfaces attracting the Server-library Development team in addition to the synchronization primitives which could replace and augment the humble semaphore, the only such interface ever made available in NetWare or CLIB before NetWare 5. Nevertheless, for completeness and to provide an upward migration path to Modesto, however different from NetWare this platform will ultimately be, it was decided to implement not only the entire NKS primitives but also the NKS virtual machine as the basic process model.

The NKS Virtual Machine

The NKS virtual machine defines a containment within which an application or service may be hosted. The virtual machine supports a multithreaded execution environment with the threads in the virtual machine sharing the following system resources: memory, the native platform security context, open file descriptors, file and record locks, and a pool of anonymous threads. The virtual machine can be transformed into a deterministic (controllable) state machine by disabling preemption and limiting the concurrency. In this mode, replay-based fault-tolerance strategies can be supported such as SFT III. The virtual machine as defined here is expected to map naturally onto the typical task or process abstractions found in modern operating systems.

While a virtual machine is a key component of the NKS architecture, to enhance portability, only a small set of operations are defined for managing the virtual machine. These are as follows:

NXDisablePreemption	Disable preemption in the virtual machine
NXExit	Terminate the virtual machine
NXGetNumEngines	Get the number of on-line CPUs
NXGetPoolConcurrency	Get the maximum number of threads that may be created in the anonymous thread pool
NXSetPoolConcurrency	Set the limit on the number of threads that may be created in the anonymous thread pool
NXSingleThread	Transform the virtual machine to be a single threaded machine

NKS does not surface APIs for creating virtual machines or APIs for supporting intervirtual machine communication. This decision has been motivated by the need to keep the abstractions portable without sacrificing important platform-specific optimizations. All intervirtual machine communication will be supported via libraries atop NKS like LIBC, which is not discussed in this article.

NKS Thread Programming

The thread is the unit of execution in NKS. A thread is a sequential flow of execution control and is mapped onto the underlying NetWare unit of execution-also a thread, but differing depending on the version of NetWare supporting NKS. (There happen to be two very different types of threads used in NKS depending on the underlying version of NetWare. In NetWare 5, the thread is created by the new, multiprocessing kernel [MPK] whereas old versions host NKS using legacy threads.) A thread virtualizes the processing element of the underlying platform, and, on a multiprocessing platform, these threads can execute concurrently. The runtime state of a thread is defined by the context (discussed below).

The two classes of threads defined are application threads and anonymous threads. Application threads are threads explicitly created by the application, while anonymous threads are automatically created and managed within the virtual machine to support the NetWare notion of work.

One of the important transport optimizations possible when the communicating virtual machines (or a given virtual machine and some other external component in the system) are hosted on the same physical node is the use of a client thread to execute the code in the server. Anonymous threads support this optimization. A thread from outside the virtual machine can migrate into the virtual machine and become an anonymous thread in the virtual machine for the duration of the transaction.

NKS supports priority-based preemptive scheduling. The priority of a thread is the priority of the context it is currently bound to high, low, or medium. Each thread is assigned a unique identifier whose scope is local to the containing virtual machine.

A context abstracts all the runtime state of a thread that this context is bound to. Encapsulated by the context abstraction are execution stack, execution priority, hardware-specific register state, context-specific private data area, and a flexible infrastructure for managing context specific data including key-value pairs. This clear separation of the execution vehicle (the thread) from the work being executed (the context) provides an efficient and scalable infra structure for supporting services built as finite state machines.

A context gets bound to a thread as part of thread creation, as part of scheduling work to be executed by an anonymous thread, or by explicit binding of the context to a thread. A bound context gets unbound by explicitly binding itself to a different context, by the context in question being a work context and the work being completed (as signified by the return from the start function of the context), or by the hosting thread exiting. NKS guarantees that when it changes the binding status of a context as a result of the completion of work (or delayed work), or when the hosting thread exits, the state of the context is reset to its initial state.

NKS contexts can be created for use either as work contexts or normal contexts. Work contexts can only be bound to anonymous threads, while normal contexts can be only bound to application- managed threads. NKS treats work elements as first-class abstractions with semantics comparable to a regular thread; the scheduling priority, stack size and private data area can all be specified for a work context.

NKS also supports a lightweight work abstraction with semantics comparable to the traditional NetWare work-to-do abstraction. In addition to the simple private data area associated with a context, per-context data may be associated with one or more keys. While these keys and their associated data can are conceptually thread specific, they are in fact bound to the context and follow it rather than the thread.

This distinction between the running thread and context data is an important difference with other thread models but especially with the model in force on NetWare since its inception. The interfaces defined to manage the threads abstraction are as follows:

NXCancelLwWork	Cancel a previously scheduled lightweight work
NXCancelWork	Cancel a previously scheduled work
NXContextAlloc	Allocate a context
NXContextFree	Destroy a context
NXContextGet	Get the identity of the currently executing context
NXContextGetInfo	Get information on the specified context
NXContextReinit	Initialize a previously allocated context
NXCreateKey	Allocate a key for the calling thread/context
NXDeleteKey	Delete the specified key
NXGetKeyValue	Retrieve the value associated with the key
NXProcessInterruptSet	Process the specified set of interthread interrupts
NXScheduleDelayedWork	Schedule work to be executed at the specified time. The API for deleting a scheduled work can also be used to cancel delayed work
NXScheduleLwWork	Schedule lightweight work to be executed by an anonymous thread
NXScheduleWork	Schedule work to be executed by an anonymous thread
NXSetKeyValue	Associate the specified value with the specified key
NXThreadBind	Bind the thread to a specific CPU
NXThreadContinue	Continue the execution of suspended thread
NXThreadCreate	Create a thread
NXThreadDelay	Delay the invoking thread
NXThreadExit	Terminate the thread
NXThreadGetBinding	Get the CPU binding information
NXThreadGetId	Get the thread ID
NXThreadGetPriority	Get the thread/context priority
NXThreadGetPrivate	Get the thread/context private data pointer
NXThreadInterrupt	Interrupt the specified thread
NXThreadInterrupted	Check to see if the thread has been interrupted
NXThreadJoin	Wait for the specified thread
NXThreadSetPriority	Set the execution priority for the specified thread/context
NXThreadSuspend	Suspend the execution of the specified thread
NXThreadSwapContext	Get the context currently bound to a thread and bind a newly specified context.
NXThreadYield	Yield the CPU

NKS Memory Management

NKS surfaces both typed as well as untyped memory, though there are peculiar semantics across the NetWare kernel and user-address spaces. Untyped memory is potentially pageable unless it is explicitly wired down using NXMemCtl. Hence, to be portable, services are expected to manage their memory needs carefully. Virtual memory allocated and not locked down by an NLM running in ring 0 (the kernel) opens execution to preemption. All execution is open to preemption when the NLM is running in a user-address space.

NXGetCacheLineSize	Get the cache line size
NXGetPageSize	Get the platform-specific page size
NXMemAlloc	Allocate typed memory
NXMemCtl	Perform control operations on the specified memory
NXMemFree	Free memory
NXMemRealloc	Change the size of the memory allocation
NXPageAlloc	Allocate untyped memory
NXPageFree	Free memory

NKS Synchronization Services

NKS supports a comprehensive set of synchronization primitives. The composition of this set was based on an evaluation of the synchronization needs of some of our services that might be definitively ported to NKS. Mutual-exclusion and read-write locks, condition variables, and semaphores are supported. Spin locks were rejected because NKS is essentially a user-level package. Barriers were rejected because other locks suffice to implement the concept.

Mutual-exclusion locks are to be used to serialize access to a shared state. To enhance concurrency when the shared state is mostly read-only, we also support read-write locks. Because NKS is designed to be hosted both within the kernel as well as in user space, how a thread sleeps awaiting a contested lock is not specified. To deal with locking-related deadlock issues, NKS supports a hierarchy with respect to locks. An application is free to specify the order in which it expects to acquire the application-defined locks, and this order is asserted by NKS in debug mode.

To support performance tuning and debugging, the system also collects statistics with respect to locks (again, if appropriate, compilation switches are defined). Given the frequent use expected of some of these locks, the function names have been appropriately abbreviated instead of prefixing them with mutex or RwLock.

Condition variables can be used to synchronize threads based on arbitrary external state. Since the state used to synchronize is external to the condition variable, they can be used to solve a variety of synchronization problems. Classical counting semaphores can be used to control access to a set of identical resources. Here are the synchronization primitives offered by NKS:

Mutexes

NXAllocLockInfo	Statically allocate and initialize a mutex or read-write lock information structure
NXLock	Acquire the mutex
NXMutexAlloc	Allocate a mutex
NXMutexFree	Deallocate a read-write lock
NXMutexDeinit	Deinitialize the mutex
NXMutexInit	Initialize the mutex
NXMutexIsOwned	Check to see if the mutex is owned
NXTryLock	Try to acquire the mutex
NXUnlock	Release the mutex

Read-write Locks

NXAllocLockInfo	Statically allocate and initialize a mutex or read-write lock information structure
NXRdLock	Acquire the lock in the read mode
NXRwLockAlloc	Allocate a read-write lock
NXRwLockFree	Deallocate a read-write lock
NXRwLockDeinit	Deinitialize the read-write lock
NXRwLockInit	Initialize the read-write lock
NXRwLockIsOwned	Check to see if the read-write lock is held in the specified mode
NXRwUnlock	Unlock the read-write lock
NXTryRdLock	Try to acquire the lock in the read mode
NXTryWrLock	Try to acquire the lock in the write mode
NXWrLock	Acquire the lock in the write mode

Condition Variables

NXCondAlloc	Allocate a condition variable
NXCondBroadcast	Wake up all the threads blocked on the condition variable
NXCondFree	Deallocate the condition variable
NXCondDeinit	Deinitialize the condition variable
NXCondInit	Initialize the condition variable
NXCondSignal	Wake up one thread blocked on the condition variable
NXCondTimedWait	Wait on a condition variable no more than the specified time
NXCondWait	Wait on a condition variable

Semaphores

NXSemaAlloc	Allocate a counting semaphore
NXSemaFree	Deallocate a counting semaphore
NXSemaDeinit	Deinitialize a counting semaphore
NXSemaInit	Initialize a counting semaphore
NXSemaPost	Release a semaphore
NXSemaTryWait	Try to acquire a semaphore
NXSemaWait	Acquire a semaphore

NKS Time and Time-out Services

Both one-shot as well as cyclic time-outs are supported. In addition to the function listed below, the thread interface, NXScheduleDelayedWork, works on a time-out basis.

NXCancelTimeOut	Cancel a scheduled time-out
NXGetTime	Get time in seconds or microseconds since1january 1980
NXGetSystemTick	Get the interval between two consecutive clock interrupts
NXScheduleTimeOut	Schedule a function to be executed at the specified time

NKS File and Directory I/O

NKS file and directory I/O interfaces collectively provide a common minimal set of capabilities for using the file and directory functionality available on any platform (not just NetWare). This may tend to limit the depth of the interfaces in comparison to those usually subscribed to by NLMs. These capabilities consist of creating or deleting file and directory objects, navigating a hierarchical name space of directories to find files or directories of interest, reading from and writing to the files, and obtaining status information on files and directories.

The complete set of features of the legacy Novell File System (NFS) as well as the new Novell Storage Services NSS is not available through NKS. Appropriate interfaces (including NSS interfaces) are indeed used to access the directory or file opened. Instead, the NKS approach with respect to the delivery of the file-system services is to split out certain common operations that can be expected to be found on all platforms and file systems including NetWare and NSS, and surface them as NKS file and directory functions. Any Novell service that currently relies on deeper-provided capabilities can continue to use those implementations on the host platform. This mode of use does not require the use of the NKS interfaces described in this section. The remaining common operations available on the local file system (i.e., the platform's native file system) are accessed through the interfaces described here.

NKS is not equipped with such namespace composition operations as MOUNT, UNMOUNT, or REMOUNT, or with support for selecting from a multiplicity of naming syntaxes. It is designed largely so that the available file system can be used for performing simple read, write, create, delete, search, and file-synchronization operations.

Operations that enforce specific caching modes, such as read-ahead or controlling the length of time data is buffered for reading or writing by the local file systems, are excluded from this interface. This is based on the assumption that the services using this API will most likely use the local file system for very limited purposes and not be affected by, nor affect the performance of the local file system.

NXAbortFileTransactions	Rollback to the last committed state
NXCancelAsyncRW	Cancel a previously scheduled asynchronous I/O operation
NXClearTransactional	Clear the transactional attribute for the file
NXCloseDir	Close a directory open for searching
NXCloseFile	Close an open file
NXCommitFileTransactions	Commit the transactions (writes) against the file
NXCreateDir	Create a new directory
NXCreateFile	Create a new file or rewrite an existing file
NXEnableWriteCaching	Enable caching on file writes
NXDisableWriteCaching	Disable caching on file writes
NXFileLockUnlock	Perform file/record locking operations
NXGetAttributes	Get the attributes of a file/directory
NXGetFileLength	Get the length of a file
NXGetPathSep	Get the component separator character
NXIsFileTransactional	Check to see if the file is transactional
NXOpenDir	Open the specified directory for searching
NXOpenFile	Open the specified file
NXRead	Atomically position the file pointer and read
NXReadAsync	Issue an asynchronous read operation
NXRemoveDir	Remove the specified directory
NXRemoveFile	Remove the specified file
NXSearchDir	Read the contents of an opened directory
NXSetPathSep	Set the path separator character
NXSetFileTransactional	Set the transactional attribute for the file
NXWrite	Atomically position the file pointer and write
NXWriteAsync	Issue an asynchronous write operation

Transport (Communications) Interfaces

Specific communications interfaces were designed in NKS, but the feeling for now is that WinSock will become the common transport for communication on NetWare, as on other platforms where NKS will be found. WinSock is already available for NetWare 4.11 and, indeed, the NKS NCP requester uses it.

Conclusion

These are the interfaces that make up Novell Kernel Services on all platforms to which it has been ported. Most of the detail covers all platforms, and I covered very little NLM-specific detail. As noted in the introduction, part two in this series discusses writing NLMs in NKS, with particular attention paid to the thread-programming model.

Part three undertakes to cover the concepts of programming to the remainder of the library.

Part four talks about the relationship between NKS and LIBC, which sits atop the former, and the differences in programming to LIBC and NKS, making relevant comments about and comparisons to programming to CLIB.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.