Novell is now a part of Micro Focus

How to Use the Novell Kernel Services Programming Environment and Standard Run-Time Library: NetWare 6 Update

Articles and Tips: article

Russell Bateman
Senior Software Engineer
Server-library Development

01 Dec 2001

This AppNote crowns a series of articles that have previously been published in Novell AppNotes and Novell Developer Notes. These articles were a statement of programming features and functionality of the Novell Kernel Services (NKS) programming environment with a new C run-time library (LibC) for writing applications and other code as NetWare Loadable Modules (NLMs) on NetWare.


application development tools, NLM development, server applications


Novell Kernel Services, NetWare 5, NetWare 6





Prerequisite Skills

familiarity with Novell Kernel Services

Operating System

NetWare 6



Sample Code



This is a final update to a series of articles detailing concepts and functionality of the Novell Kernel Services and a new C run-time library to accompany it. The release time-frame is first customer shipment of NetWare 6, but support for this environment has already been released in the support packs to NetWare 5.x. A brief summary of each article in the series is given below:

"The Future of Application Development on NetWare with NLMs" discusses why CLib had reached a crossroads in its history and how it would pass the torch to NKS/LibC for future application development. (See

"Features of the Novell Kernal Services Programming Environment for NLMs: Part One" explores the reasons for this programming paradigm in the NLM environment and examines in surface detail the various interfaces promoted by NKS. (See

"Features of the Novell Kernal Services Programming Environment for NLMs: Part Two" focuses more closely on the programming concepts in the Novell Kernel Services environment including discussion of multithreaded programming correctness, latency and thread cancellation issues, and synchronization. (See

"Features of the Novell Kernal Services Programming Environment for NLMs: Part Three" finishes the concepts started in Part Two by covering the remaining programming interfaces for writing NLMs include the NKS virtual machine, memory management, platform- dependent and debugging features, file and directory I/O, and time-outs. (See

"Features of the Novell Kernal Services Programming Environment for NLMs: Part Four" discusses how LibC sits atop NKS and made comments about programming at both levels. It also made a few remarks about differences in programming to LibC and CLib. (See

"KLib: A Kernel Runtime Library" elaborates the development issues and techniques that leave the domain of strict application programming (based on CLib or LibC) and enter the domain of lower-or kernel-level code. (See

"New Features of the Novell Kernal Services Programming Environment for NetWare 6 Programming" exposes new features added to NKS since 1999. (See

Two years ago, in the kick-off article entitled "The Future of Application Development on NetWare with NLMs," I explained how Novell had reached a decision point with the then ten-year old CLib programming environment in its attempt to continue to support existing NLMs coded to it and at the same time move forward to embrace support for future mulithreaded programming.

In that article, I pointed out that:

  • CLib had fundamental problems in supporting multithreaded code such that applications written to it do not scale well across multiple processors. Were CLib to be modified to accommodate such support, the majority of existing applications coded to it would break including losing IPX/SPX, processor-0 centricity to which many NLMs were coded and some other services.

  • NKS was chosen as the thread model to remedy CLib's incomplete support of NetWare 5's new multiprocessor kernel.

  • NKS was also the upwardly mobile solution to Novell kernel support including NetWare 6 and the 64-bit next-generation platform code-named Modesto.

In the interim, development of NKS on NetWare and on Modesto has sharpened our focus. NKS has matured as well as adopted a C run-time library duplicating CLib within its own paradigm. This article seeks to summarize what NKS/LibC is today with particular attention paid to important aspects that have evolved since each of these articles. The result is as comprehensive and significant as the original CLib.

While this AppNote makes frequent mention of Modesto, it is not about Modesto nor even about the implementation of NKS on Modesto where NKS is the very interface to that kernel. The focus of this AppNote is upon NKS and LibC as they are implemented on and affect NetWare.


The resulting library has been called LibC. In its NLM form, it is named libc.nlm. It is essentially an execution kernel extending NetWare 5 and 6's Multiprocessor Kernel (MPK) augmented with system interfaces to that kernel plus user-level interfaces. What I mean by "execution kernel" is software that hosts the execution of applications and other programs by interfacing them with the underlying NetWare scheduler and solving problems of start-up, shut-down and data instancing. So, LibC, just as CLib before it, is far more than a library. In this article, I generally use NKS to reference to NKS-specific components and functionality. I use LibC to refer generally to the whole library and in this usage, LibC is ambiguous with NKS/LibC which I sometimes use to insist upon the close, inter-cooperative relationship between NKS proper and LibC.

LibC now consists of what were formerly four different libraries and which for a brief time existed separately:

  • KLib. The kernel-level interface collection set up to permit drivers, protocol stacks, libraries and other low-level code to consume standard interfaces formerly provided by CLib but not available to them as they loaded too early. This library was the object of a separate article in this magazine about one year ago.

  • UniLib. A new Unicode interface library that forms the basis of standard C wchar.h and wctype.h interfaces which are the preferred wide-character support offered by LibC.

  • NKS. The much-discussion solution to moving beyond nonpreemptive, uniprocessor NetWare programming.

  • LibC. The replacement for CLib which runs atop NKS' execution kernel.

KLib existed for only about six months and, regrettably, a service pack was distributed. We were really trying to begin supporting driver and low-level code programming on NetWare directly from the library, and it was thought that to do so, we should separate the kernel applicable components and load them earlier.

Because the decision was made to cut the ties to NetWare 4 from NKS/LibC, it was possible to retrofit the NetWare 5 kernel in v5.1 for some kinds of critical support the library needed to support low-level code properly-support that could not reach back into NetWare 4. Because of this decision, we were able to merge KLib into LibC making LibC the basis not only for its own functionality, but a fair number of functions also claimed by CLib. (More on this later.)

The sort of support that enables LibC to underlie any code written to NetWare is in the form of thread and NLM context (for the NKS virtual machine). Every NLM loaded and every thread that executes on NetWare now has this support whether it actually uses it or not though much of that support lies dormant until needed as the result of a call into the library. Even though KLib introduced mostly "contextless" interfaces, it is now the case that LibC supports applications and other code regardless of whether they explicitly set this up by linking a prelude object, and it also loads early.

To prove this to yourself, boot NetWare 6 with its new USB support. You'll find that LibC is already loaded by the time you have got that USB support (on which the USB mouse depends, for example) because the USB drivers make use of it.

Second, the legacy Unicode library NLM, unicode.nlm, was not particularly reentrant and it made copious use of function pointers to tailor its locale-specific functionality. As was laid out in an article published in August of 1998 (, "Scheme to Optimizing Calls to Functions through Pointers on the Pentium Platform," calling functions through pointers causes branch "misprediction" on Pentium with considerable overhead in lost clocks. This phenomenon helps make this NLM, which was written before Pentium, slower than acceptable. (It is always easier to replace a set of interfaces that have been around a long time with better adapted ones and avoid the mistakes of the past.)

Many components need Unicode support today and not all that was needed was made available by this library. UniLib is considerably faster in its individual support for translation of characters to and from the local code page and in casing support where it is very much faster. It is also a little bit easier to use than the old library.

In addition to support for Unicode with local code page and Unicode alone, LibC offers specific UTF-8 interfaces. UTF-8 is the direction of much of text processing relative to the Internet and other platforms. While we haven't got a completely native UTF-8 linguistic to all of our interfaces, you should probably notice that we are heading in that direction. Within a short time, most interfaces should support this alternative to Unicode and byte strings in the local code page by way of translation support, parsing functions and acceptance of file and directory paths.

NKS, the third component, has already been much discussed in these articles. In this article, only the evolutionary changes to it will be explored in detail and the particulars of its raison d'être won't be repeated.

LibC is, first, the name given to the C run-time library that runs atop NKS and gives it value from the point of view of most applications and portability. Second, it is the whole library because it is formally the C execution environment and interface to NetWare. This is a little like CLib giving its name to some six or seven NLMs that were its composites.

LibC, the individual component, is easy to describe because it consists mainly of:

  • Almost all ISO/IEC 9899:1999 (ANSI C '99) standard interfaces. Particularly missing is complex math functionality.

  • A fair number of ISO/IEC 9945 (POSIX '96) standard interfaces; mostly the ones that CLib already offered and, as with CLib, there are considerable semantic limitations.

  • Interfaces from other standards and would-be standards like Open Source.

  • Interfaces demanded by internal and third-party developers, occasionally specific to NetWare and, from the point of view of a new header, netware.h, interfaces that actually come from the NetWare OS.

The ones from the latter two sets will be treated as we go in this article.

Symbol Prefixing

As explained in an earlier article, prefixing became necessary because the NetWare Loader isn't sophisticated enough to discern between symbols of the same name coming from different NLM sources. A number of years ago, a prefixing syntax was introduced into NLM linking and this is the means by which printf can be exported by both CLib (as printf) and LibC (as LIBC@printf). Not all symbols from libc.nlm are prefixed. By and large, the interfaces belonging to KLib and to NKS are not. Thus, there is no LIBC@strcpy, but there is a LIBC@strtok because the latter cannot be the same one as used by CLib applications since it isn't reentrant and relies upon library context to sort out the problems of data instancing.

Development Tools and Requirements

When these articles first began appearing two years ago, there was a multiplicity of tools available. Watcom had only recently stopped shipping. Edinburgh Portable Compilers, Ltd. had recently become a player as had Metrowerks CodeWarrior. Then too were GNU C, Borland and Microsoft at least theoretical possibilities.

Today the scene has changed. CodeWarrior has entrenched itself more solidly as the premier NLM development platform, at least inside Novell. All new projects are done using CodeWarrior and ome old, but still active code has been ported where it made sense to do so.

The SDK will soon ship a prelude object for GNU use, but it, Borland and Microsoft remain largely an art practiced and supported by only a few outside of Novell. Our attitude remains that we will try to do all we can to accommodate such development, but targeting multiple platforms is beyond our ability and resources. In our SDK, for example, can be found libc.wmp and libc.ali, import and alias files to help the Watcom hold-out link using wlink.exe, which doesn't have support for symbol prefixing. Despite this, however, it is impossible at this time to envision LibC support for Watcom C++ because we cannot modify Watcom's complex runtime libraries which are essential to Watcom C++ development.

A lot of assembly code has been moved from PharLap to MASM or to CodeWarrior. I personally find the CodeWarrior assembler easy to use and I would have to go outside my tool set to do anything else. For the last two years, I've used only Metrowerks' command-line tools to develop NKS and LibC. As of this writing, a source-level debugger was soon to be released in the Metrowerks/Novell CodeWarrior Professional Developer Kit NetWare. This will be version 3 of that PDK.

Bundling of Development Aids

CodeWarrior comes with stationery for its integrated development environment (IDE) that help the developer create an NLM project quickly. The project is typically already set up with a main, statically linked runtime library (or two libraries if C++ is used), a stubbed linker command or definition file and settings. The Novell Developer Kit (NDK) provides the import file or files referenced from the definition file, the headers to compile with (the path to which is established in an IDE setting), and various preludes for CLib, one single prelude for LibC. As this is already well set up by the Metrowerks/Novell Professional Developer Kit (PDK), there is no need to discuss it here beyond mentioning that the stationery for individual LibC-based projects is comprehensive and growing continually to cover not only basic C or C++ applications, but also lower-level code like libraries.

Link Options

While I'm on development tools, let me cover some linking issues peculiar to LibC use that will be at least as awkward to cover in another section as they are here. There are a number of suppositions made; these are:

  • At link time, the type of string used in NKS file and directory I/O and still other interfaces must be chosen. This is done using the UNICODE_STRING link flag (0x02000000) or not (defaulting to ASCII). Linker flags are set using the FLAG_ON command in CodeWarrior, the NLMFLAGS command in Watcom.

  • The PSEUDOPREEMPTION flag (0x00000008) must be used or LibC will not load the NLM. This is basically a ruse to force you to realize that what you are writing isn't your grandfather's NLM. Your NLM must be written to be multithread-safe and preemptive (able to survive preemption).

  • The AUTOUNLOAD flag (0x00000040) should be considered for library NLMs. A library NLM is one that usually doesn't contribute active threads beyond its initialization phase, exports at least one interface for use by other NLMs, and is usually auto-loaded by these NLMs. This flag makes it so that the NLM will come down automatically if it was auto-loaded in the first place. It also helps it better participate in dynamic symbol use involving the dlfcn.h interfaces (an Open Source standard).

  • The TERMINATE_STAY_RESIDENT flag (0x01000000) can be added to an NLM, such as a library, so that even after its active threads die away, the NLM doesn't unload. The reasons thread might die away range from they have finished initialization (the initial thread running main, for example) or a calling thread may have cause the library to start up a new thread performing some information-gathering, bookkeeping, maintenance or other task and now that thread is finished. The NLM would otherwise unload at this point.

Dynamic Library Support

Evoking the existence of dlfcn.h interfaces in LibC returns us to the issue of prefixing discussed earlier. In order for dlsym to discover the right symbol to import into its caller, library NLMs loading and participating in this way must standardize on a predictable prefixing scheme. For the purposes of these interfaces, and this would be a good scheme to follow whenever prefixing, the name of the NLM having symbols to contribute should be used. Succinctly put, symbols should be exported either without prefix (as most do today) or prefixed with the name of the NLM providing them. And this should be done in uppercase since case is respected. The linker syntax for prefixing exported symbols is:

   EXPORT            foo,

This link directive will lead to foo, MYLIB@bar, and MYLIB@foobar being listed in the NetWare Loader's publics tables upon load of (presumably) mylib.nlm. Function dlsym will attempt to look up the demanded symbol using a prefixed name corresponding to the source implied in its first argument and, that failing, without any prefix.

Linking Low-level Code

LibC support for low-level code grows out of the developer's need to control start-up and shut-down more closely from an NLM implementing a driver, protocol stack or library. In the past, CLib's prelude provided the _cstart_ entry point with no ability to control the special conditions present at NLM start-up or shut-down.

LibC's prelude, libcpre.o, is specially engineered to allow the driver to create its own entry points as known by the loader while at the same time, making them uniform for all. This is because whether hand-rolled start-up and shut-down code exist or not, the start routine is _LibCPrelude and the stop routine is _LibCPostlude. Beyond this and in addition to or instead of the main that applications code, the driver can contain entry points for _NonAppStart and _NonAppStop as well as _NonAppCheckUnload. Prototypes for these functions exist in a new header, netware.h, and the arguments are passed through as is by the library just as if these were called directly from the NetWare Loader.

Therefore, any piece of code whether low-level like a driver or top-level like an application can cause the aforementioned entry points to be called. If the NLM doesn't code a _NonAppStart function as would be the case for most application NLMs, then it isn't called. If the NLM has a main, then it will simply have an active thread-the normal situation for the application, but unusual for a driver or a library. This is not, however, impossible. When the NLM doesn't have a main, then no main function is called, no active thread is created and the NLM remains loaded assuming it returns zero from its _NonAppStart function.

Linking Application Code

Applications continue to link as usual by including libcpre.o and coding a main; nothing changes over how it was done in CLib. Just as the CLib application tells the linker its entry points are _Prelude and _Stop, the LibC application tells the linker its entry points are _LibCPrelude and _LibCPostlude.

Here is a good chance, however, to point out that LibC applications coding a main will unload after all threads-and not merely the one that was executing main-have run to completion. This is different from CLib which unloads the NLM once the main thread has terminated.

For drivers, protocol stacks, libraries and applications or any other NLM, the point of linking with libcpre.o is to get adequately initialized context set up so that the NLM can make any calls it wants into the library as early as at start-up. The lack of context at start-up was the reason so-called CLib NLMs couldn't control their own start-up and shut-down.


At the time of this writing, Metrowerks and Novell were on the point of coming out with a fully functional source-level debugger operated from the CodeWarrior IDE in PDK 3. I have some experience with this debugger, having used it on the Macintosh years ago and more recently on Windows. What is new in the debugger and what has made it challenging to implement has been the need to cover the peculiar combination of multiple threads, processors and address spaces on NetWare.

Threads can marshal down into the kernel from one address space and, because they are kernel-created, get coopted for use in another address space. Threads can and do often migrate from processor to processor, particularly between processor 0, where all NLMs start, and other processors. Threads return to processor 0 in certain cases to run code such as the legacy file system that hasn't been upgraded to operate on other processors. All of this makes for difficult debugging.

NLM developers have long had to use the NetWare System Debugger which is a kernel-level tool unable to interpret debug records. Following execution is done with a disassembler which can be tedious even for those familiar with Intel assembly.

The prefixing problem created by the flat symbol namespace on NetWare leads in turn to a problem in debugging. The order in which NLMs are loaded establishes which symbols the debugger looks at without greater context. This has always been a problem since more than one NLM can, when linked for debug, have a main that is visible in the debugger. However, now that LibC is exporting almost as many standard symbols as CLib, just break-pointing printf can be a real challenge.

To disassemble your NLM's main function, you probably had to type

   # u mynlm.nlm|main

This is still the case. However, you used to be able to set a breakpoint on printf just by

   # b=printf

and go on confident that you would stop the next time your NLM (or someone else's) called printf. This would be in CLib. LibC exports a printf under the name LIBC@printf and now that we're all writing to LibC,

   # b=LIBC@printf


   # b=libc.nlm|printf

would be a very annoying way to set a breakpoint. To solve this problem, there is a new global variable in NetWare v5.1 SP3 and NetWare 6 named preferredModule. By setting this variable to the module handle of the library NLM of your choice, or to your own NLM, you can control where the debugger symbol look-up goes first. By default, LibC sets this variable to itself when it loads (for my team's convenience and because we invented the concept), but nothing stops another library or NLM from grabbing it later, so this may lead to a free-for-all situation in which little control can be maintained.

Note too the (long-observed) tendency for debug symbols to be inserted into the symbol list rather than added on by the NetWare Loader. This means, typically, that, in addition to the preferred- Module semantic, the mere fact that your NLM has been more recently loaded than another will ensure preference of your symbols over those of another (including the library, but not overriding those of the module designated by preferredModule).

Basically, this means that if you export a printf from your module, referring to it in a debugger command will require prefixing it (as you have prefixed it, say MYLIB@printf) or you will get LibC's, but if you export a zzyzzx from your NLM, referring to it in the debugger will probably get you the one you want even if, say CLib exported such a symbol because CLib loads very deep (early on) in the boot process and your NLM probably won't be loaded for debugging until much later.

To work around these problems, a set of interfaces are available from the library.h interfaces now promoted by LibC. Specifically, it is possible to get the module handle for any NLM by name or from your own NLM. The function that returns the module handle for your own NLM is getmodulehandle. If necessary, you could set this up yourself for another NLM, perhaps a library that supports your application.

                void        *handle;
        extern void                *preferredModule;

        err = findnlmhandle("libc.nlm", &handle);

        if (!err)
            preferredModule = handle;

Additional debugging interfaces at this lower level include the EnterDebugger call which, when encountered, halts the processor and drops into the debugger. This is used often as a call to printf is used for in debugging except that no information can be printed to the screen.

Another trick in making your NLM easier to debug with the low-level debugger is to order the objects in your NLM using CodeWarrior's LINK_ORDER command (which is also an option in the IDE project window, Target Settings). If you code main (or _NonAppStart if you don't have a main) at the top of its C file, and order this file at the top of the link, the following console and debugger commands will get you to the first "debuggable" instruction in your NLM (my own server happens to be named after the Celtic bard "Taliesin"):

   TALIESIN: load -d mynlm

    # g mynlm.nlm

For the up-and-coming CodeWarrior source-level debugger, there is a debugger setting accessible from the project window that specifies the function in which to set the first, temporary breakpoint at which the debugger will halt when first brought up. In Windows programs, for example, this would usually be the first source line in main. The various stationery for LibC set this up accordingly for IDE use.

Adding a tutorial on using the low-level debugger exceeds the scope of this article, so the present information will suffice as it hasn't already been covered. However, it may be useful to point out again that dropping in the debugger interrupts everything and if you have LAN connections, these will soon time out which may be annoying.

NKS Functionality

Now let's examine, header by header, both the NKS interfaces that have been changed or refined and the LibC features that have been added since last Fall. Most functionality in NKS has changed somewhat; some, like File and Directory I/O, has changed extensively. In LibC, mostly new features have been added to shore up the interface needs of developers.

In addition to functionality, we paid close attention to the uniformity of names and cleaned up the types we were using. Gone are the artificial types NXLong_t (64-bit guaranteed width) and NXBoolean_t (with NX_TRUE and NX_FALSE) in favor of using more pointedly specific types such as NXOffset_t, NXBufSize_t (64-bit guaranteed buffer size), size_t, ssize_t, uint64_t, and bool from ISO/IEC 9989:1999 stdint.h and stdbool.h where you would expect a standard. While we have a lot of abstract types, especially structures, but also scalars, we haven't resorted to confusing or ungainly Hungarian representations and have employed a standards-based type wherever it could be justifiably used instead of making up our own.


This section was new last year. Beyond its support via NXCopyEnv for NXVmSpawn, nothing has changed.


The file and directory I/O section is what has seen the greatest improvement. Already last year, it had changed somewhat. Over the last few months it has been completely overhauled to include what we've learned about file systems other than our own, about virus scanning, archive and retrieval, extending a file's length on open, handling sparse files, and more.

We plan additional interfaces to be added like call-backs at certain events for virus checking, but none of the present interfaces present at FCS of NetWare 6 will change any further. Some interfaces below comport an Ex- suffix. These permit a great deal of latitude in specifying the file system operation including the complex operations alluded to earlier like "extents" and sparse files, asynchronous and vectored I/O, etc. There is to little room here for a detailed exposé; this will be left to documentation.

Some interfaces have WithHandle- suffixed versions. These perform the operation named by the call but through a handle to the already open object (file).


Creates a bookmark into the file system for use in combination with a path to identify a file system entry. Formerly NXCreateDirHandle.


Injects a different pathname context than the default into a path context. This is roughly equivalent to the concept of namespace in CLib.


Frees a path context.


Gets the first or next entry in a directory search. Formerly NXDirSearch.


Deallocates resources associated with a directory search.


Initializes a directory entry search.


Gets attributes of a file by path; there is also a call that works through the handle of an open file.


Sets the attributes of a file or directory; there is also a call that works through the handle of an open file.


Cancels an asynchronous I/O operation in progress.


Closes a file, console/device, FIFO, etc.


Allocates file "extents" through a handle to an open file.


Flushes data through any buffering in the file system below.


Gets the length of a file; there is also a call that works through the handle of an open file.


Simple version of open file.


Complex, fully functional open file.


Deletes a file system entry; there is also a call that works through the handle of an open file.


Renames a file system entry; there is also a call that works through the handle of an open file.


Sets the length of a file; there is also a call that works through the handle of an open file.


Locks a range of bytes in a file.


Locks a range or several ranges of bytes in a file with more control over the detail of the locking.


Unlocks a range of bytes in a file.


Unlocks a range or several ranges of bytes in a file with more control over the detail of the locking.


Simple read of file, console/device, FIFO, etc.


Full-control read of file, console/device, FIFO, etc. of vectored ranges.


Simple write of file, console/device, FIFO, etc.


Full-control write of file, console/device, FIFO, etc. of vectored ranges.


Opens a device. Presently on NetWare, the only supported device is the console.


Opens/creates a first-in, first-out file by name. The name can be communicated to or known by participating NKS VMs. In this case, it works as a sort of named pipe.


Creates a hard link in the file system.


Creates a symbolic link in the file system.


Macro calling NXDeviceOpen to open a console.


Macro calling NXFileOpen to create a file.


This section has been somewhat impoverished over what it had been advertising. Some functions were mainstreamed into NKS proper, others dropped as LibC was integrated (nxPrintf, etc.), but interface wrappering and cancellation point handling remain as necessary shims on NetWare to overcome the lack of containment discussed in earlier articles.


Exports an interface via ExportPublicSymbol.


Exports an interface wrapped such that callers will gain exporter's context while executing it.


Is VM loaded in a protected address space on NetWare?


Returns size of memory allocated using NXMemAlloc.


Tears down the wrapped, exported interface.


Macro to wrap an interface without exporting it (special case of nxExportInterfaceWrapped).


Macro to balance previous visually.


This does not mean platform-specific, but relevant to the host platform. Functionality that had hitherto been prefixed as part of the VM or Thread Model interfaces migrated here. Some additions were made; by and large these need no discussion beyond available documentation.


Gets length of the memory cache line in effect.


Counts the processors present.


Returns the identity of the currently executing processor.


Gets the length of the memory page in effect.


Gets the interval between two consecutive clock interrupts.


Nothing was changed in this section except for enforcing explicit semantics of recursion for mutexes. This is to say that, by default, a mutex is not recursive and in debug mode (as defined during compilation), an assertion will occur whenever a mutex is used recursively without having been explicitly created as recursive. If the mutex is allocated as recursive, recursive use passes in silence. We failed to flesh out the hierarchical debugging aids promised in past articles; this remains on the list of things to do for future releases.


Almost nothing changed in this section. We added a macro, NX_THREAD_CREATE, that implements a call to NXContext- Alloc followed by one to NXThreadCreate with some standard defaults.

We also rejected and removed the concept of private data area for a thread in the way it was originally implemented because of the redundant, but more useful existence of key-value data pairs (64 in the NetWare implementation).


A lot of work was done on this section. For one, we've almost completely abandoned the concept of implementing backward- compatibility with the nonpreemptive, uniprocessor environment of NetWare. Beyond the original artificial impetus for designing it into NKS, we have had no call for it. In fact, as noted elsewhere in this article, an NLM won't be loaded by NKS on NetWare without the PSEUDOPREEMPTION link flag set.

Another area where there has been a lot of work and progress is in the spawn interface. It has been adjusted slightly from what it was a year ago in response to evolving understanding of just how it will work and be used.

The following are the functions that have changed importantly or significantly:


Retrieves information about the worker thread pool.


Returns the type of string currently in force.


Registers any number of exit handlers.


Changes information for the worker thread pool.


Spawns a new VM with a specific environment in effect.


Unregisters a previously registered exit handler.

LibC Functionality

LibC is the repository of the bulk of the work in terms of sheer functionality in support of developers of Open Source ports, new code and others porting old code to NKS/LibC.

Standard ISO (ANSI, POSIX, etc.) Headers

There are non-standard interfaces that logically extend standard ones. Most standard headers include a few of these be they from ctype.h, stdio.h, stdlib.h, string.h,time.h, and so on.

For example, stdlib.h now offers functions that implement atomic operations, including atomic_add, atomic_bts, atomic_btr, atomic_dec, atomic_in c, atomic_sub, and atomic_xchg; orthogonal-width rotation functions rotl8, rotl16, rotl32, rotl64, rotr8, rotr16, rotr32, rotr64; useful boundary or debugging functions like stackavail and stackwatermark; and more.

stdio.h interfaces have been enhanced with handling of wide-characters. The contents of FILE have been rendered completely opaque as individual field access became a problem in CLib that kept us from being able to maintain interfaces accepting FILE *. Functions flockfile, ftrylockfile, funlockfile, getc_unlocked, getchar_unlocked, putc_unlocked and putchar_unlocked have been added. setmode, an interface in unistd.h that is tightly coupled by fileno to stdio.h interfaces, now works in LibC. It could not be implemented for CLib.

string.h adds versions of its functions dealing more accurately with multibyte strings in the local code page such as Lstrcat, Lstrchr, Lstrcmp, Lstrcoll, Lstrcpy, Lstricmp, Lstrlen, Lstrlwr, Lstrncat, Lstrncmp, Lstrncpy, Lstrnicmp, Lstrpbrk, Lstrrev, Lstrtok_r, Lstrupr.

time.h is augmented by mkgmtime, a sort of mktime that interprets its broken-down time argument as if it were Greenwich Mean instead of local time rounding out the standard time interfaces flowchart (see the "nowhere" arc in Figure 1, page 80, in "Time Functionality in the Standard C Library," Novell AppNotes, September 2000 ( which leaves gmtime a dead end and no easy to construct UTC times by hand. To boot, dos2calendar and calendar2dos eliminate the rather confusing alternatives to this process offered by CLib by exposing two simple structures and a union for easy reading of the DOS date and time results.

In fact, most of the additions to these standard interfaces, along with the simple standard interfaces themselves, are consumable even by CLib applications since they require no context. Creating a suitable import file to notify the linker of a CLib application is, however, left as an exercise to the developer.

Some of these headers are broken out and discussed separately in the following list where specially relevant.


As begun in CLib, there are extensions to this rather simplistic standard interface that permit the caller to tailor the post-assertion action to aborting, entering the debugger or proceeding from the assert. Moreover, a function interface capable of returning a value after proceeding is available though not within the ISO-prescribed syntax (assertf). This is useful for several situations, the most important of which is software testing.


These interfaces form the bulk of our support for creating, authenticating and maintaining NCP connections for performing remote file I/O (as does CLib) or explicit exchange of NCP packets. The create_identity interface presumes a fully distinguished NDS name which LibC cannot furnish. The developer is obliged to gain such a string either from the Lightweight Directory Access Protocol (LDAP) interfaces available on NetWare or from elsewhere. This function also has emerging support for Novell Multiple Authentication Services (NMAS).

This article does not cover issues arising over the legacy Novell Directory Access Protocol (NDAP), also known as DSAPI, in opposition to LDAP. Because it has been widely called for, it is being ported atop LibC, but some semantics will necessarily change. As this work is currently in progress, these changes cannot be accurately discussed here and a discussion of them will be relegated to a later article. In any case, NDAP could be used to obtain the fully distinguished name just spoken of, or it could be used just as it is atop CLib as it will be fully integrated with LibC though without so much allegiance to state and thread architecture as was the case with CLib.


Creates an identity by logging in; identity encloses authentication credentials which are used to authenticate as many NCP connections as are ever needed by the caller; there is no need ever to juggle these connections as in CLib. The identity can be passed notably to NXGetPathContext.


Disposes of the identity and all associated connections.


Opens a session over which raw NCP packets may be exchanged.


Disposes of an NCP session.


Exchanges an explicit NCP packet over the session; caller builds the packet according to public domain documentation.


Combines an identity with a path context (from NXGetPathContext) and injects it into the VM as the current working directory. This sets up an environment of user identity, file system location and file system rights so that normal POSIX operations can operate with meaning. CLib required constant juggling of NCP connection, NDS identity and other states in order to achieve this.


Thread-only version of setcwd.


A new NetWare-specific error value is available, filesyserrno, which is set deep inside NKS or LibC whenever errno just isn't going to say enough about what went wrong. If execution wandered down into the new Novell Storage Services (NSS) file system, for example, the value will be one from that component.


This is the Open Source Group's dynamic library function management facility. Some of this doesn't fit well into NetWare; it is there to help Open Source ports and to provide a better interface to the NetWare Loader than is done by using ImportSymbol from CLib. We've found, as noted elsewhere, that the utility of this interface is greatly increased by NLMs playing according to two rules:

  • Symbols offered by a dynamic (NLM) library must either be unprefixed or prefixed according to an official scheme which is that the prefix be identical to the name of the NLM (minus its extension) in upper case.

  • A library offering symbols must link with the AUTOUNLOAD flag. This is because the dlfcn.h interfaces are not integrated into the NetWare Loader where enough table information is available to know whether it is safe to unload the library. The Loader is intelligent enough to unload a library automatically if this flag is set and if no library entry points are presently consumed. A function from netware.h, SetAutoUnloadFlag, is also available; calling it from the initialization code achieves the same thing.


Opens (and loads, if necessary) an NLM library that will presumably offer interfaces to be consumed dynamically by name. A handle is returned.


Closes and unloads an NLM library.


Imports a pointer to a named interface in the library whose handle is passed.


Returns a message explaining the last error to occur in this facility. In LibC, this is per thread.


CLib had precious little support for other libraries than itself and the cross-platform libraries. A primary goal of LibC was to fix this problem based on years of experience attempting to support third-party libraries in the mix with CLib. This functionality section fulfills the goal making it possible for libraries to register themselves, register a destructor function and easily get a pointer to their instancing data.

The procedure for working with instance data is simple, though not automatic as it would be if we were writing a DLL on Windows NT. For example, a library's function is called on a thread. The library doesn't magically know where it keeps data for that thread or its application so it attempts to get the data through a special interface. If the data is missing, it probably means that it was never created and set, so the library sets it up. Here is a very cursory implementation of a library entry point using its own data which it associates with a specific caller. Remember, the calling thread comes from the library's client rather than from the library itself. The library, early on during start-up, registered itself by calling register_library and received a discrete handle from LibC.

#include <library.h>

    extern void                *libhandle;                    
	// from register_library()

    int lib_func( void )                                    
	// exported library entry point
        libdata_t                *data;

        if (!(data = get_app_data(libhandle)))
            data = malloc(sizeof(libdata_t));

            if (!data)
                errno = ENOMEM;
                return -1;

            initialize the data...

            set_app_data(libhandle, data);

        use the data now set or already (merely) got...

        return 0;

The sort of data to be stored and retrieved this way depends on the library's unique needs. If, for example, the library were so intrusive in its actions as to need to maintain per-thread information for its caller in addition to per-VM (that is, per-NLM) information, it could call NXKeyCreate and store the resulting key in the per-VM data returned by get_app_data. Then, in the working part of the function, it could manipulate that per-thread data using the other NKS key-value data pair functions.

Obviously, this could get rather extensive and the code written above to check for and get (or create and set) the library's data on behalf of its client application might well be relegated to a special function call to be performed at the beginning of every library entry point. This was the direction taken in the special library stationery included in CodeWarrior PDK 3 for NetWare. The code in this programming template is copiously commented for a number of problems in library development.

LibC itself does something very similar at the beginning of most of its entry points. Certainly, functions such as strcpy don't operate on data outside what is passed via the argument list and so don't merit this treatment, but printf, for example, must know what the caller's standard output console (stdout) is along with some other stateful information that is maintained by the library. Moreover, errno, a per-thread data item, is also set potentially by printf. Solving these data instancing problems is the one area where NetWare programming is challenging as compared to most other environments. library.h was created to make this easier than before with CLib.

Conceptually, a library can be simply thought of as "dead code." When its code is executed, it isn't by the library itself (that is, by its threads), but by the threads of its client applications. Libraries typically don't have any active threads anyway. Consequently, when a library function calls malloc in the standard C library, for example, the memory allocated gets charged to the calling thread's application (NLM) and not to the library. When that application unloads, it takes that memory with it and the memory is invalid. Poorly written libraries botch this situation by continuing to access that memory from other threads running through the library code when it cannot easily be known whether in fact the allocating application has come down already or not. A second way this situation can become severe is for the library to communicate that memory address to another application than the owning one. The library developer must guard against this by enacting good library design, architecture and implementation.

When a library function calls malloc in the standard C library the memory allocated gets charged to the calling thread's application.

If the library needs to allocate memory for itself, but cannot know when and how much it will need until these are known as a result of being called by a client application, that memory can be allocated using lib_malloc which takes the library's own handle (the one returned from register_library) and uses it to charge the new memory against the library rather than against the application. Obviously, to guard against the inverse of the situation described in the preceding paragraph, the library cannot communicate that memory address (or any part of it) to client applications unless that memory is going to be carefully conserved until after the last library client unloads.

Both these potentially dangerous situations have additional ramifications for libraries and applications crossing protected address space domains on NetWare. Indeed, the use of a library in the kernel by an application loaded in a protected address space is a very advanced programming exercise as compared to most of what we're talking about here. Such a discussion outstrips the scope of this article.

The use of some of these interfaces, particularly the last six or so listed below, goes well beyond mere library development; developers of any application, driver, protocol stack, or library would need them.


Registers a library returning a handle by which it is known to all NKS virtual machines as long as the library is loaded.


Unregisters the library.


Fetches a pointer to the library's client application-specific data area known and interpretable only to it.


Sets a pointer to the library's client application-specific data.


Returns flags containing information about the potentially available context of the calling thread, whether it is from a CLib application, an NKS/LibC application, Java, etc. or a combination thereof. See example in Library stationery of CodeWarrior PDK 3 for NetWare.


A clear-allocation operating on the library's own behalf rather than on behalf of the calling thread's VM as calloc would do.


free companion to these functions.


Allocation operating on the library's own behalf rather than on behalf of the calling thread's VM as malloc would do.


msize companion to these functions.


Reallocation operating on the library's own behalf rather than on behalf of the calling thread's VM as realloc would do.


Obtains the named NLM module handle with optional address space constraint.


Returns the identity of the current address space.


Gets the module handle for the calling thread's NLM.


Returns the name of the calling thread's NLM.


Returns the file system path of the calling thread's NLM.


Is this NLM loaded in a NetWare protected address space?


Some developers wanted access to the same information monitor.nlm uses to create its displays. Some of the more complex information used by this NLM is already available through the Media Manager which is public. These interfaces give some more and the intent is to add to these as needed.


Returns information about the active processors.


Returns information about the file system. More is available through the Media Manager.


Returns information about memory.


Returns information about (up to 12) LAN boards.


Returns information about the NetWare OS (AbEnd count, server processes).


Returns information about volumes. More is available through the Media Manager.


Years ago, there existed a published document for developers of drivers on NetWare. This documentation exposed a number of interfaces from the NetWare OS that were more or less cast in concrete (not likely ever to change). This header isn't really a reproduction of that document, but it does expose a small subset of NetWare OS interfaces directly that a developer might want to use. Those chosen for export come from the original driver development lists augmented with other useful ones identified from among those that have demonstrated the utmost stability.

This header is accompanied by an import file shipping in the NDK with LibC, netware.imp. The types and prototypes in this header have been modified to lessen coercion and other tricks used in-house to get around problems especially with the rest of LibC attempting to move development toward standard and more portable types. The original prototypes often use LONG to hold pointers. This has been changed.


LibC supports the use of NUT, the rather outdated, but still useful blue-and-white text-based interfaces used by such utilities as Monitor, NWConfig and Install. In fact, there is stationery for a simple application using NUT in CodeWarrior PDK 3. However, because LibC is trying to move toward standards, it was necessary to adapt the original header with the result that a new one is distributed with LibC.


There is far less opportunity to manipulate screens from LibC than CLib. CLib had a full, low-level package interfacing the NetWare OS screen handlers. The emphasis is now on remote management via browser for which there is an SDK, or on Java-based utilities. As LibC supports the interfaces in nwsnut.h, it neglects lower-level ones that go much beyond simple printf.

These functions are:


Clears the screen.


Prints to the NetWare System Console, or Console Logger (NetWare 6) screen.


Returns mode of screen (SCR_AUTOCLOSE_ON_EXIT).


Gets one character from key-press.


Gets one (low-level) key-press.


Gets the screen handle for the NetWare System Console or Console Logger (NetWare 6) screen.


Gets the screen handle for the application's screen.


Gets string input from the keyboard.


Sets the cursor at the indicated screen coordinates.


Boolean TRUE or FALSE that a key was pressed.


Pends the screen until any key is pressed.


Pends the screen until Escape is pressed.


Puts a character to the screen.


Puts a string to the screen.


Sets (changed) mode of screen.


Rings the bell.


Pushes back a character to the application's keyboard.


Pushes back a (low-level) key press to the application's keyboard.


Returns the column coordinate of the current cursor position.


Returns the row coordinate of the current cursor position.


Returns the row and column coordinate of the current cursor position.

Each NLM that starts up get one screen only, as named during link using the SCREENNAME command. If this command was not used, there is no screen; output goes the System Console which, on NetWare 6, is the logger screen, and input is disallowed since input to the keyboard from the console would be ambiguous if not actually impossible to handle with applications attempting to read from the console as well.

When an application that has linked with libcpre.o and also a screen name starts up, a screen is created to which stdin, stdout, stderr, cin and cout are wired. The latter two, which should not be confused with C++ objects, are the hard console pointers used by cprintf and other interfaces. These remain even after the standard three have been closed or, are present despite any command-line redirection that may have been specified.

It is possible to open a screen from an application that didn't link a screen name. This is useful in the event an NLM library or other service provider wishes to implement an optional debug screen. This must be done through macro NXConsoleOpen.

While the screenwise offerings of LibC are small, nevertheless, there is a small number of screen functions that will handle all simple tasks. These include clearscreen, consoleprintf, getcharacter, getkey, getstring, gotorowcol, kbhit, pressanykey, pressescape, pressenter, putcharacter, putstring, ringbell, wherecol, whererow, and whererowcol. These were not given names very similar at all to the ones in CLib for a couple of reasons including that those in CLib aren't particularly standard and these were given names obvious of their functionality without being so close to those in CLib as to throw the caller into confusion about which call comes from which library.


See ws2nlm.h.


Normally, nothing would need to be explained about a standard interface. However, this one is, in addition to the few POSIX-mandated fields, a treasure house of information about the calling thread's NLM (NKS VM).


This is an ISO/IEC 9945-1 (POSIX) header. In addition to its standard interfaces can be found:


Cause the current working directory to change for the calling thread independently of all other threads in the VM (NLM). Later, calling it with nil returns the behavior of the calling thread to rely on the single current working directory of the process (or VM or NLM).


Construct a directory-delimited path based on one or more input strings. The strings assembled are server and volume names, directory path and file or directory name. Similar to CLib's _makepath.


Analyze a pathname by splitting out its contents into one or more strings as may be used by construct to reassemble the path. A flags argument is returned with a number of comments about the contents of the pathname (whether it was UNC, contain forward or backward slashes, DOS drive letter, etc.).

unilib.h utf8.h

These are not the preferred interfaces for manipulating strings. NetWare's wide-character solution is Unicode and that means using wchar.h. However, some of these functions operate independently of local code page, so when the string needs to be translated into or out of some other code page than the local one, these must be used. UTF-8 will one day doubtless become the string mode par excellence for applications, but for now, these basic functions that translate in and out of ASCII and Unicode, plus the ones that imitate functionality from string.h, will have to suffice.

wchar.h wctype.h

These, on the other hand, are the preferred interfaces for manipulating wide-character strings respective to the local code page.


LibC has fully integrated WinSock for use by applications. WinSock can be used directly (outside the library) or through Berkeley interfaces integrated with the library's file, pipe, etc. descriptors. The choice is basically keyed off the inclusion of sys/socket.h or ws2nlm.h. The former gets a socket from socket that must be closed using close, while the latter gets a socket from socket, which is only a preprocessor redefinition of WS2_32_socket, that must be closed by closesocket, a redefinition of WS2_32_closesocket.

What Is Missing and Future Directions

No discussion on the state of things would be complete if it did not also include comments on what goals we've missed but still plan to reach, and where we plan to move in the future. Especially over the past year, we've taken many suggestions to heart and shifted gears accordingly. Sometimes it's been easy, such as the monitor.h interfaces which represent mere additions to the library as suggested by one heart-and-soul NLM programmer who doesn't work for Novell. In other cases, this has required a great deal of tear-up and rewrite, such as our effort to implement NDAP (netnlm32.nlm) and other cross-platform library support. Some of the greatest contributors have been the Novell SysOps, a group of talented engineers who aren't on the Novell payroll. Other important contributions, especially in shaping the direction of the new library, have been the Open Source ports.

The areas where we feel there is not yet enough support are locale, characters representation, NetWare management, and file system.


Despite a plethora of NWL-prefixed functions, CLib had practically no locale support. LibC was designed to overcome this and ships with locale support for a number of locales as can be seen on the path c:\nwserver\nls, but still falls way short of the mark. One problem is a lack of intelligence for additional locales, but though we know where to go to gather this intelligence, we have yet to decide how to encapsulate it. Mostly, we just haven't had the time to finish the solutions we have chosen.

Presently, the setlocale call functions a lot more than in CLib, but there are still disappointing results such as the failure of strcoll to response afterward to any change in locale setlocale may have resulted in. Presently, strcoll operates on the underlying locale (the same one as the server) and Lstrcoll does the same thing, but is more careful about the prospect of characters being composed of more than one byte. As this implementation is going forward to Modesto, we feel the greatest need to retain this as one of our most important problems to solve.

Character Representation

The question of ASCII, multibyte local code page, Unicode, and UTF-8 really tie in with the preceding problem of locale support. In fact, that locale support already mentioned has been held up precisely because of character representation.

Novell has had a Unicode solution in place since NetWare 4 primarily in support of Novell Directory Services (NDS). Translation in and out of the current locale has been provided. However, since the explosion of the Internet, simply moving between ASCII and Unicode seems a bit near-sighted. Many developers have asked us for UTF-8 support which we've quantified in the utf8.h interfaces, but this really does not go far enough. The convenient relationship between ASCII and UTF-8 is too much to pass up and we must enable more of our interfaces to accept implicitly UTF-8 strings. Whereas NKS started out with a default of Unicode and an ASCII option, we are questioning that commitment and asking ourselves if UTF-8 shouldn't have at least an equal share of our attention if not become the default.

UTF-8 is easy to use with the right interfaces to manipulate individual character representations, storage effective as long as the actual bulk of string data doesn't shift substantially from Western alphabets to Eastern systems, and it doesn't require special retooling from ASCII as does Unicode in the source code. UTF-8 might become the dominate representation at Novell within the next couple of years and this will affect both NetWare through LibC use as well as Modesto.

NetWare Management

As we accelerate toward our next-generation platform, one tends to forget that we are still in the business of supporting NetWare. Indeed, NetWare doesn't become obsolete with the next platform any more than the V-Rod terminated the other Harley-Davidson lines. Instead, it promises exciting performance and the opportunity to do things in a new environment where we can increase our wingspan and our speed. And, just as the new V-Rod, we hope it will have the same throaty roar of NetWare as we twist back on the right hand grip.

So, CLib had a number of interfaces to gather information about NetWare. There are applications that want to do so. We have tried to move those pieces of information forward that were consumed from CLib and add new ones such as are found in monitor.h. This effort simply isn't finished. There is more to come.

File System

In order to claim a useful set of file system interfaces, we have to beat CLib at what it has offered. Direct File System (DFS) and other deep-down interfaces aren't yet offered by LibC, so the new environment is ill-adapted at supporting back-up and virus checking. This will change thanks to NKS and developments in file system technology. We have already added the equivalent of readv and writev which presently only exist in CLib as interfaces to descriptors allocated via socket. These are done by NXReadEx and NXWriteEx and support files, FIFOs, devices and the console.

As an example of what we plan that goes further beyond CLib, consider flattened I/O streams that eliminate the need for a back-up application to get file information, read the file, read extended attributes, get more namespace-specific information, and on and on, then find a format in which to store it. Consider instead, a flag to NXFileOpenEx that causes subsequent calls to NXRead to roll out all this information into a compact stream for direct consumption by back-up software. Consider another flag to NXFileOpenEx that reverses the process by enabling the back-up software to squirt this same stream back through NXWrite.


This is an outline of all that has changed over the final months of NetWare 6 since the others in this series of articles. NetWare 6 formally introduces NKS/LibC as the preferred way to program to NetWare moving it from the NDK futures area to one on a par with CLib with NetWare 5 service pack 4 support shortly behind. More than other software perhaps, a library is never quite done and the driving force in direction is spread between forward mobility to the next-generation platform, the continued march of NetWare, and suggestions from the developer community.

* Originally published in Novell AppNotes


The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates