Novell is now a part of Micro Focus

Features of the Novell Kernel Services Programming Environment for NLMs: Part Four

Articles and Tips: article

RUSSELL BATEMAN
Senior Software Engineer
Server-library Development

01 Dec 1999


Discusses how LIBC sits atop NKS. Also discusses differences in programming to LIBC as compared to CLIB. Fourth in a series of articles on the programming features and functionality of the NKS programming environment.

Introduction

This is the last installment of a series for the review of programming features and functionality of the Novell Kernel Services (NKS) programming environment.

Part One explored the reasons for this recently defined programming environment in the NetWare Loadable Module (NLM) environment, and it examined in some detail the various interfaces surfaced by NKS.NLM.

Part Two focused on the programming concepts in the NKS environment and included discussions of multithreaded programming correctness, latency, and thread cancellation issues.

Part Three of the series finished the concepts started in part two by covering the remaining programming interfaces.

Part Four, this article, discusses how LIBC sits atop NKS and makes comments about programming at both levels. In addition, it makes pertinent remarks about differences in programming to LIBC as compared to CLIB.

The New Standard C Runtime Library on NetWare (LIBC)

In the September 1999 issue of Developer Notes, I explained how the ten-year-old CLIB programming environment had reached a milestone in its attempt to continue to support existing NLMs coded to it and at the same time move forward to embrace support for future multithreaded programming technologies. I explained the reasons why we have frozen CLIB in its current state (plus bug fixes) and moved ahead with a new environment based on NKS plus a new standard C library environment (LIBC) loosely referred to as NKS/LIBC. While it seemed clearly enough stated at the time, some approached us to verify, on the one hand, our continued support for NLM programming, and, on the other, our support of CLIB and applications written to it.

As noted in the article just mentioned, CLIB continues to be maintained and to host the vast majority of applications currently written for NetWare. This will persist as the state of things for a long time to come. CLIB will always be the recipient of Novell's testing organizations and will receive from our group the attention that it always has. We still fix bugs in CLIB 4.11 from time to time, and it hasn't been two years since we fixed a bug in 3.12 and released an update for it.

What CLIB won't be is the object of great technological advancements. It won't be endowed with interfaces to support shared or virtual memory in NetWare 5. It won't ever surface interprocess communication (IPC) interfaces for use between two NLMs. It won't directly support WinSock, etc. Basically, it won't evolve past its present level of functionality; we won't bulk it with any new technology.

The present effort in some quarters to port open source code to NetWare, along with projects to make existing NetWare applications and services scale across multiple processors, has led us to make progress in areas like IPC, and shared and virtual memory as well as cooperative support of NetWare WinSock from the library. (Previously WinSock and the standard C library were unknown to each other for purposes like select, read, and write.) NKS offers support for some of these features, and LIBC solves others. Together, they support NetWare 5's multiprocessing kernel, virtual memory, address space protection, and other new technologies. Together, they point toward the future Novell platform, code-named Modesto, whose SDK is still a ways off.

C/C++ Programming Emphasis and NetWare

Due to the excitement of welcoming a Java Virtual Machine (JVM) and the Java programming paradigm to NetWare more than two years ago, it may have appeared to some that C and C++ programming was falling in our priorities. A lot of effort was put into gaining acceptance and developers to Java programming on NetWare.

Nevertheless, programming in C went on within Novell as well as without. For example, the JVM sits on top of the NetWare C runtime libraries: what it needed the rest of the developer community got in the way of enhancements and bug fixes. Almost two years ago, even before Watcom had announced they would not issue versions of their popular environment beyond v11.0, Metrowerks announced its support of code warriors on NetWare by creating a professional developer's kit for NLM development using their IDE. Not following Watcom's direction, Metrowerks tools let the C and C++ developer code to Novell's standard library rather than supplying their own. Similarly, EPC has promoted development tools of their own for creating NLMs and have followed suit by not substituting a proprietary C library in place of Novell's CLIB.NLM.

I hope that these facts lead to the conclusion that Novell hasn't de-emphasized C/C++ programming to NetWare. While it isn't as portable as Java for network applications, it results in the best performance performance we can still brag about because of NetWare's own advantages in that arena. NetWare 5 introduced a real protection technology making running application code safer by guarding the operating system with its file system, Directory and other services from applications the system administrator loads in ring 3 whether because they are unstable, untried or just safer to run there. AbEnd prevention and recovery continues to develop in NetWare. These are all improvements for the C/C++ programmer and should make the case that this paradigm isn't going to disappear.

LIBC Context

Those who've programmed to NetWare recognize that one of the more difficult aspects of NLM programming is library context. Over the years, one of the most frequent questions I have had to deal with is library context. This question goes back to the discussion of containment in Part Three of this series and to the peculiar nature of NetWare programming. I am launching right into this discussion because in writing this article, I realized that a great deal of library context issues arise and can't be passed over as if they were well understood. This discussion is, therefore, quite germane to the rest of the article.

Since programming on NetWare is not user programming, but of the kernel variety, context and reentrancy are a big problem. In a proper user environment, errno, the current working directory, stdin, stdout and stderr, are all transparently handled by the operating environment made up of the user containment and DLLs or static-linked libraries. On NetWare, an NLM application is really a kernel extension since, when running in ring 0 at least, it operates within the same address space as the NetWare operating system and all the other NLMs running at the same time.

Consequently, there can be no global variable definition in the standard C library thus:

int    errno;

because no sooner than one thread, erroring out of a function, could set it, another would come along and change it, probably before the first could exit the function, check its return code and then ask for errno's value.

So, where is such a value kept? The answer is in thread-specific data maintained by the library for each execution unit like a thread (errno) or the application as a whole (the current working directory, stdin, -out and -err). This is why the standard header, errno.h, for LIBC contains the following declaration and macro:

extern int *__errno( void );

#define errno   *__errno()

When errno is referenced in code, after preprocessing, it isn't really a global integer variable at all, but a call to a function that returns a pointer to a location in the library's thread-specific data maintained for the calling thread and a dereference of that location. Thus

errno = -1;

results in (after preprocessing):

*__errno() = -1;

Obviously, a thread from NetWare itself, which doesn't use the library and doesn't have the library's thread-specific data, is going to get an undefined value when it calls __errno() to get a pointer to it. The subsequent dereference of the return value as a pointer will result (most likely) in a page fault.

Because of its very nature, NetWare CLIB programming often results in calls into the library without context. An application might register a call-back or otherwise lead the operating system to create and schedule a thread or work-to-do on the application's behalf executing its code. In LIBC, it is the same. As noted in the discussion on containment in November's article, we are requesting that the developer control these situations more closely and use some bracketing functions to tell NKS when an application thread is migrating outside the NKS virtual machine containment (the library and the application) and reaching into a service that isn't NKS-aware (meaning, doesn't itself sit atop NKS) and that acquires locks, resources, etc. so that NKS won't kill or suspend the thread while it is holding these foreign resources. At the same time, these bracketing functions may permit the library to maintain context on behalf of these threads in some cases since they serve as more or less a wrapper. We hope that by BrainShare 2000, we'll have more experience with the feature and more to say about this.

In the meantime, if your application contains code that will be called back or executed by a non-NKS thread (this does not include NKS work-to-dos or time-outs as they expect this), this code must not make calls into NKS or LIBC that require context or those calls will return an error, in most cases, NX_ENOCONTEXT orENOCONTEXT. This problem isn't much different than what an application suffered in CLIB.

LIBC with NKS: What to Expect

LIBC is the name of the new standard C runtime library that replaces, in the context of Novell Kernel Services (NKS), CLIB. LIBC.NLM exports the same entry points as CLIB, particularly all of CLIB.NLM and some of NLMLIB.NLM and THREADS.NLM, namely, all interfaces promoted by ANSI as well as much of POSIX and additions to both these standards. To continue the reduced semantics of NKS, LIBC does not promise compatibility with CLIB except that inasmuch as CLIB complied with ANSI and POSIX, LIBC should be identical. However, over the years, CLIB has been buried in semantic expectations. Many developers coded around bugs or holes in its interfaces. Others found ways around it in areas it didn't cover well.

To fix the bugs meant breaking existing applications and so this wasn't done or it was done in careful ways. One example of this is fputs which ANSI says returns the number of characters put, but which returned 0 or -1 on success. The second argument to open was not implemented in POSIX-compliant fashion and the semantics arising from combinations of O_CREAT, O_EXCLand others were different than those expected by developers porting from other platforms.

Thus, we do not guarantee LIBC to behave identically to CLIB in every interface. We have followed ANSI and POSIX as well as CLIB in developing the semantic for LIBC, but have preferred ANSI or POSIX where a stated meaning was given.

LIBC context, which we've already discussed in detail, sits atop NKS context. LIBC threads are nothing more than NKS threads. Indeed, there are no threading primitives in LIBC. POSIX pthreads, for example, are not part of LIBC. Nevertheless, LIBC maintains contextual values like errno separately from the NKS context created by NXContextAlloc. We speak of per-thread data with respect to LIBC; we really mean per-context data in the sense discussed thoroughly in Part Two of this article series.

LIBC errno values and NKS errors are identical at the symbol and numeric levels. The values from 0 through 99 are reserved for the implementation of ANSI, POSIX and extensions to these, 100 to 1000 for Novell extensions to errors (like ENOCONTEXTand ENAMESPACE).

Besides threading primitives, all functionality that conflicts with NKS does not survive into LIBC unless there is another compelling reason to put it in such as having been popularized by CLIB. This targets principally any notion of UNIX threads, pthreads as noted, synchronization and scheduling primitives. These are not offered.

An area of overlap, however, is memory management. Despite NXMemAlloc, LIBC promotes malloc. The latter merely calls the former. However, NKS doesn't surface any notions of shared memory. LIBC is expected to have a functional subset of sys/mman.h include shm_open, et al.

Interprocess communication (IPC) hasn't been completely designed yet. It is essential to open source ports that have been undertaken over the last couple of years. Various work-arounds have been devised, but in the end, the load of separate NLM applications at the request of others with shared descriptors probably only the standard ones and intercommunication is inevitable and we are putting the finishing touches on a design, which will require some effort on the part of porting engineers since it won't be fork/exec, that should be straightforward enough to release to the greater developer community. What I can say is that this functionality almost certainly will require ring 3 loading (separate NetWare 5 address spaces). It probably won't be available in the kernel.

LIBC file I/O is, as it was for CLIB, the POSIX set including open, read, write, etc. augmented by that specified by ANSI (fopen, etc.). These are more tightly integrated than in the older library, for example, Perl uses a setmode call that simply couldn't be implemented before. It works in LIBC. The file I/O interfaces are built atop NKS' set with additional reaching down into NetWare where (very infrequently) necessary.

The interfaces read, write and others are also useful to non file-type descriptors including consoles, pipes, shared memory and sockets. These are integrated just as they were in CLIB, but also, socket now returns a descriptor to a WinSock object and manipulates that descriptor just as CLIB supported a Berkeley-like socket descriptor and interfaces. The goal is a tighter integration of NetWare's WinSock library for coding that does not have to be any more aware of socket origins on NetWare than on other platforms.

Pipes will continue as before few developers ever used them as they only surfaced in support of the Java Virtual Machine implementation beginning on NetWare v4.11 except that named pipes will be added and these will be held in shared memory as part of our IPC offerings. They won't be created in the file system, but they will be specifiable as if they were. The details will appear at a later date, perhaps at BrainShare, perhaps in another Developer Notes article. Obviously, there will be a special open call, perhaps named_pipe, that will take a path.

The reason I am announcing a number of technologies is because we must have them and it is certain they will be there in the 6 Pack time frame. What I cannot do at this point is state their exact interfaces nor how they work as their designs are not all finished at this writing.

LIBC with NKS: What Not to Expect (But You Can Expect it Anyway)

NKS really concentrates on the issues of multithreaded programming correctness, multiprocessor scalability and portability to NetWare's future platform. What it has not addressed primarily is backward compatibility to features found in old CLIB. This has not escaped our attention, however.

We have made ourselves a list of every interface to be found in the CLIB suite of NLMs (FPSM.NLM, LIB0.NLM, THREADS.NLM, REQUESTR.NLM, NLMLIB.NLM, NIT.NLM and CLIB.NLM). We have also examined our cross-platform libraries to see what features they offer. Our conclusion has been to continue promoting the technologies that are cross-platform and let the aging ones go. However, this statement is insufficient to communicate our vision so a short expos is in order about how functionality will be made available to NKS/LIBC programs.

First, CLIB context agent exists that will permit access from LIBC to most older interfaces, specifically those requiring library context, in CLIB. This agent brokers enough context on behalf of its caller to enter CLIB and return with information. One of the principal target libraries is the NetWare Interface Tools library or NIT. These interfaces have, by and large, been replaced by the cross-platform calls long available to both workstation and NLM programmers and NIT is more or less considered obsolete. Nevertheless, it doesn't stretch the imagination very far that someone would want to use it even if we officially discourage it.

Second, the cross-platform NLM libraries are being re-engineered to sit atop NKS or CLIB indiscriminately. This fact obviates much of the worry that we would lack specific NetWare information technology. This achievement was made possible by the design and implementation of a new requester in NKS, written last Spring, that performs the role of NCP broker for these libraries as well as for the remote NCP file I/O capability in NKS (and, hence, LIBC).

The remaining missing functionality is being mulled over. Some solutions have been found. Let's choose the example of namespace technology.

Because namespace support in the file system postdates CLIB, the latter's support for it grew somewhat awkwardly as Java and others found out. In NKS, we have adopted the attitude that by default, applications make file system calls in the long namespace. For example, the path argument to LIBC's fopen, opendir, stat and NKS' NXCreateDir, etc. is expected to be a long name or, at least, findable in the long namespace. Indeed, NetWare 5 loads the long namespace by default on its system volume. Support for the primary (DOS, 8.3) namespace is provided and invoked when the path names presented are uniform in case (upper or lower) and do not contain long names (all path components conform to 8.3) and the NetWare volume simply doesn't support the long namespace. New errors, NX_ENAMESPACEand ENAMESPACEhave been added to warn of namespace conflicts (or illogical use) in a bad path.

As noted in part three of this series, NKS still supports file I/O to the DOS partition, local and remote NetWare volumes and is expected to interface NetWare Storage Services (NSS) volumes more directly than does CLIB. Moreover, NKS file I/O is 64-bit ready. We are contemplating extra-standard additions to POSIX and ANSI so that 64-bit I/O can be done at that level also.

There are other interfaces to functionality not represented by NKS or LIBC, but we are dealing with each whether through the context agent already mentioned or directly when the lack of this functionality would cripple.

Now, back to how NKS/LIBC work and how to target them.

NLM Symbol Prefixing

Anyone experienced with NetWare-loadable module (NLM) programming and the flat symbol namespace on NetWare has been wondering how we can succeed in providing a new printf, strcpy, fopen, readdir, socket, etc. for developers to use when CLIB.NLM already exports these. For the answer, we must return seven years to an internal effort prompted by the development of Platform-independent NetWare (PIN). In those days of porting NetWare to HP's PA RISC, Dec's Alpha, Sun Solaris and Apple PowerPC, it was thought that most vendors would want to use their own standard C library and other libraries for the best performance.

In order to do support vendor-specific libraries without conflicting with CLIB or other NLMs already exporting the same names, the linker design was enhanced to permit the linking of code referencing symbols that might be got from different sources. (In fact, this was also the original reason CLIB.NLM was broken into six or seven smaller NLMs including Threads, Requestr, etc!) This enhancement took the form of prefixing by which is meant that a library contributes exported entry points prefixed with its name or other appropriate string of characters. The library's import file, in LIBC's case, LIBC.IMP, lists the entry points, but states a prefix in a special format prescribed by a document, The NLM ToolMaker Specification, which is communicated on a non-disclosure basis from Novell to tool vendors like Metrowerks, Inc. and Edinburgh Portable Compilers, Ltd.

Later, the linker matches up the entry point used in the code with the one indicated by the import file. So, the linker prepends the prefix and ultimately creates a symbol dependence with a name that won't conflict in the loader because, in theory at least, prefixes are supposed to be discrete and registered with Novell Developer Support. For example, LIBC exports printf, but tells the linker that it wants the name presented to the loader as LibC@printf. Similarly, an NLM linked using LIBC's import file, LIBC.IMP, is linked to ask for this symbol even though the reference within the actual code is printf.

Watcom's WLink was written long before prefixing or even the document describing it was invented. Unfortunately, while WLink was the object of many enhancements over the years, prefixing was too dormant a concept and was never added. On the other hand, WLink was enhanced to offer the ALIAS directive which helps us create at least an awkward work-around for the lack of prefixing in a limited respect. The utility of aliasing will be noted in a few moments.

In the NetWare system debugger, these symbols show up prefixed and are long to type. However, they can usually be accessed via their unprefixed names since the debugger finds the most recent occurrence of a name. Since in NetWare 5 CLIB is loaded very early on (as compared to its load sequence on older versions of NetWare), setting a breakpoint on printf will almost surely result in a breakpoint on LIBC's entry point rather than CLIB's. This makes debugging CLIB applications more tedious and LIBC applications less so.

If you are debugging a CLIB-based NLM and you keep running into LIBC symbols when you type the names of common functions for setting breakpoints, disassembly, etc., you know what you need to do: unload LIBC and applications depending on it.

Choosing LIBC or CLIB

Choosing which C runtime library to code to is a function of several factors. Beginning with NetWare 6 Pack, NLM development falls into one of several categories or expedients as illustrated by the following table:


Development activity
Options
Comments

Maintain existing NLM without recompilation or relink (not marked with MPKXDC)

CLIB

Nothing is required; the NLM will continue to run atop CLIB and all versions of NetWare (so far). It will not execute anywhere but processor 0.

Maintain existing NLM without recompilation or relink (marked with MPKXDC)

CLIB

Nothing is required; the NLM's threads will continue to behave as before including frequent funnelling to processor 0.

Port existing NLM

Mark with MPKXDC

Continue to use CLIB

Port to NKS/LIBC

Port to NKS requires new thread model and consequent recoding. In either case, NLM threads can run on other processors, but with the frequent funnelling caveat noted if CLIB is used.

Port code from another platform including open source

Mark with MPKXDC

Port to NKS/LIBC

Port to CLIB

Encouraged to port only to NKS/LIBC. LIBC should be more fully ANSI and POSIX compliant than CLIB. Threads can run on other processors, but with the frequent funnelling caveat noted if CLIB is used.

Create new NLM

Mark with MPKXDC

Use NKS/LIBC

Use CLIB

Encouraged to port only to NKS/LIBC. LIBC should be more fully ANSI and POSIX compliant than CLIB. Threads can run on other processors, but with the frequent funnelling caveat noted if CLIB is used.

Import Files

It is possible to use Metrowerks CodeWarrior and EPC to develop NLMs that use CLIB or NKS/LIBC. In both cases, it is merely necessary to include NKS.IMP and LIBC.IMP in the build process. Because of prefixing, the printf you called in your code will be LibC@printf if you include LIBC.IMP instead of CLIB.IMP. If you include NKS.IMP and CLIB.IMP, the results will be unworkable.

Prelude Code

You must, as always, link a prelude into your application. For CLIB, this was PRELUDE.OBJ or NWPRE.OBJ. For NKS and LIBC, there are others. Before continuing on with this discussion, I'd like to stop bringing CLIB into it. Developing to CLIB has already been discussed exhaustively and for many years. We've already contrasted CLIB development with LIBC development. This will avoid confusion from this point on.

If you are making calls into LIBC, use LIBCPRE.O. During prototype and early-access stages of NKS, we distributed first a PRELUDE.OBJ then a NKSPRE.OBJ which we have or will soon discontinue. This happened because LIBC was lagging behind NKS on which it sits. Linking with NKSPRE.OBJ won't permit calling into LIBC, so it shouldn't be used. You lose nothing, however, by linking LIBCPRE.O even in the unlikely event that you don't use LIBC calls that require context.

The development environment you use makes almost no difference. There are work-arounds for developing to LIBC using Watcom, but they are awkward and, in any case, not supported if Watcom's statically-linked library is included.


Environment
What Novell provides
Comments

Metrowerks C/C++

NLM header filesLIBCPRE.OLIBC.IMPNKS.IMP

Use either the IDE or command-line tools. (LIBC.NLM is built using CodeWarrior command-line tools.)

Edinburgh Portable Compilers (EPC) C/C++

NLM header filesLIBCPRE.OLIBC.IMPNKS.IMP

Contact Novell Developer Support.

GNU C/C++

NLM header fileslibcpre.o

Use of GNU is ad hoc and perhaps not for the faint-hearted. I have included it here because we supply the prelude code, however, we do not provide any other support at this time.

DLLs (in Cor C++)

Special NLM headersDLL loading on NetWare-contact Developer Support

None.

Watcom C

NLM header filesLIBCPRE.OBJLIBC.ALILIBC.WMPNKS.IMP

WLink does not support prefixing except through use of the ALIAS link directive. Because `@' is overloaded in linker syntax, the work-around is difficult and requires quoting. Do not attempt to include Watcom libraries, therefore, no C++!

Sample Code

We have written some very rudimentary examples for common operations in NKS and published this code on the web with the SDK. However, this sample code can be found nearer by this article at another URL listed below. Our goal is to equip the documentation with more and more examples over time. For now, at least examples of creating context, creating, suspending, rescheduling, yielding and joining threads, using synchronization variables in NKS, key-value pairs, file and directory I/O all promised in the preceding articles of this series have been made available.

Sample code (only) downloads for this article series: http://www.novell.com/coolsolutions/tools/index.html

SDK downloads (including code samples): http://developer.novell.com/ndk

Conclusion

This is, then, how LIBC and NKS relate. In the CLIB world, one is hardly aware that BeginThread is exported by THREADS.NLM while printf comes from CLIB.NLM. We just call it CLIB. In the case of NKS/LIBC, they will function together on NetWare. LIBC's thread model is the one exposed by NKS. When we published NKS' prototype two years ago in NPL.NLM, we built it atop CLIB because we could hardly imagine anyone writing code only to NKS without the broader standard C library.

Out goal is to make porting to NKS/LIBC painless when coming from CLIB and far less painless when coming from another platform than going to CLIB by tackling the areas of containment and IPC the absence of a solution to which made porting to NetWare more exciting than most wanted.

While this is the last article in a whole series devoted to the topic of new library technology for NetWare, it is doubtless not the last you'll hear of it. There have been presentations at BrainShare and other strategic developer conferences over the last two years. At each conference, we've talked about NKS more and more. In the most recent, its interfaces were used to illustrate how to write multiprocessing applications for NetWare because they are the only ones that can ensure both scalability on NetWare as well as a path to our future platform. More will likely be written.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates