How to Write Start-up Code for NLMs

Articles and Tips: article

Russell Batemen
Senior Software Engineer
Server-library Development
rbateman@novell.com

01 Aug 2002

This AppNote covers the writing of NLM start-up code of all sorts, describing how it has been done in the past-with or without CLib and LibC-for drivers, libraries, applications, and other kinds of NLM programs. It also discusses NetWare Loader flags and other link options. It culminates in suggestions for how best to write NLM start-up code for NetWare 5.1 and higher.

Introduction
Let's Talk History
What Is Start-up Code?
Specifying Code Entry Points to the Linker
CLib Start-up Code
LibC Start-up Code and Data Instancing
Writing Start-up Code in the 21st Century
Coding the Application main
Conclusion

Topics	NLM development, server applications, kernel development, driver development, library development
Products	LibC, NetWare 5, NetWare 6
Audience	developers
Level	intermediate
Prerequisite Skills	familiarity with C and NLM programming
Operating System	all versions of NetWare
Tools	none
Sample Code	yes

Introduction

If you know anything about writing NetWare Loadable Module (NLM) programs, you know that it is different from the usual user-level programming environments of UNIX and Windows NT. Making it less different might make it slightly less exciting for engineers, but more appealing to marketers and management who are mostly interested in delivering solutions-the end of the road rather than the journey. Over the last couple of years, LibC development has increasingly pointed to a day in which the NLM programmer's craft no longer resembles black magic. One of LibC's principal goals has been to minimize the differences with user-level environments and become a more transparent host to Open Source ports and all other kinds of programming.

However, writing NLMs is still kernel-level programming. Even if the "mystical" elements are fewer than they might have been in NetWare 3, they are still there because some bugaboos remain. When I say bugaboos, I am talking about the following preoccupations that NetWare kernel programmers (a bit redundant because almost all NetWare programming is done in the kernel) must know about in order to do it right:

Start-up code
Data instancing

As I say, LibC's goal has been to eliminate these bugaboos. The purpose of this AppNote on how to write start-up code is basically to describe the magic wand in such detail as to demystify it. It not only presents the history of writing NLM start-up code, but also helps you understand what to do on current and future NetWare platforms.

Before going on, let me say that while I admit that NLM programming is often (and usually) kernel programming, I am not deprecating the protected address space model available since NetWare 5. Having applications run in their own address space is a very good goal.

This AppNote was written as much for developers who are new to these issues as for those who have already written start-up code. The objective is to provide in one place a lot of information that can't otherwise be found together and to make it accessible to developers no matter what their level of experience is. Thus I hope to help you avoid the frustration of playing 20 Questions in the Novell newsgroups with the experienced developers who answer questions in the NLM forums, at least on the issue of start-up code.

Let's Talk History

NetWare 2 had virtually no way for third parties to add any services. Novell offered some "extras" via the value-added process (VAP), but VAPs were hard to write and nigh-on impossible to debug. Few were ever written other than by Novell engineers. About the only third-party VAPs that come to mind are a set written cooperatively by Dayna Communications and Novell for NetWare 2.15 to host the Macintosh client directly from NetWare. But even that was a collaborative effort with Novell engineers.

Now jump ahead to 1989 and the early days of NetWare 3. For engineers, a new era was about to begin with NetWare Loadable Modules. The NLM model allowed NetWare to directly support simple-to-write extensions to its kernel, with no need to relink and no need for a lot of internal information. Compared to VAPs, NLMs were exceedingly easy to write!

With the advent of NLM programming, engineers quickly saw the utility of creating a C library; thus CLib was born. However, CLib was not gestated overnight-it took months just to get a standard C library with basic functionality up and running as an NLM. Architectural assumptions had to be made for the hosting of other NLMs calling into it because NetWare itself was not an operating kernel. It was an extensible file server with the ability to track transactions. It hosted the fastest protocol support around, had a modest scheduler, and provided numerous useful but disparate services such as queue management, a primitive object directory (called the Bindery), and some generic advisory synchronization. But it was not a complete operating kernel, and that is what CLib needed and therefore had to provide for itself.

Eventually, there began to be two ways to write NLMs: with CLib or without. Typically, applications wrote to CLib; everything else-including drivers, libraries and other "system" software- wrote without it. The salient characteristic of a CLib NLM was that it was linked with prelude.obj, an object compiled from CLib's idea of how NLM start-up code should be written. Thus CLib NLMs do not write their own start-up code because that code is in prelude.obj which they link (more about this later).

What Is Start-up Code?

Start-up code is what each NLM must have in order to allocate necessary resources and perform its initializations before it can successfully run.

Operating systems have to be boot-strapped so that when the computer is turned on, something will happen to get the necessary software loaded into memory and start it executing. You veteran programmers may remember back to your early school years and how the inner workings of a computer seemed like black magic at first. Similarly, when you wrote your first main program in C, you probably wondered what was going on inside the computer to get main called with its arguments.

In my own first UNIX system experience in the early 1980s with SunOS (not Solaris), Ultrix, BSD and Xenix, I discovered by using dbx and adb that main itself was called from something called _cstart_ which I immediately realized I hadn't written. So, years later when I came to Novell and NetWare, I learned what our equivalent of _cstart_ was because I had to write it. Having written a couple of print drivers for the Macintosh before then, I knew there was nothing special about the names _cstart_ and main.

So it is with NLM start-up code: there is nothing special about main-it is just another entry point among several you can choose from as the starting point of an NLM. It so happens that when code objects are linked together to create an NLM, a file is written in a special format. The first few bytes of this file are precisely known and when read by the NetWare Loader, it notes the offset of three functions of crucial importance: the start function, the stop or exit function and, optionally, the check-unload function. Along with all the other offsets in the file, these get fixed up to be memory addresses. The Loader then calls the start function and passes it a number of arguments that might be useful to it (as illustrated in the prototype code below):

int Start(             void            *loaddefinitionstructure,
            void            *errorscreen,
            const char            *commandline,
            const char            *directorypath,
            size_t            uninitializeddatalength,
            void            *nlmfilehandle,
                           int         (*readfunction)(
                           int         conn,
                           void         *filehandle,
                           size_t         offset,
                           size_t         nbytes,
                           size_t         *bytesread,
                           void         *buffer ),
            size_t            customdataoffset,
            size_t            customdatasize,
            int            messagecount,
            const char            **messages );

Later on, when the Loader is made aware that the NLM should be stopped from executing (we call this "unload"), the stop (or exit or shutdown) function is called after a successful call to the check-unload function:

void Exit( void );
int  CheckUnload( void );

All of this is included in our present topic: we consider exit and check-unload code to be part of "start-up and shut-down."

In the thirteen years of NetWare 3 and its successors, the arguments to these functions have not changed. Though few of the arguments are ever really used by many NLMs as they start up, we'll discuss them in detail here. (It might be helpful to refer to the preceding prototype for Start as you go through the explanation of the arguments.)

loaddefinitionstructure. The load-definition structure is an internal NetWare structure maintained by the Loader discretely for each NLM. It contains myriad essential pieces of information such as the name and type of the NLM, what address space it is in, where its code and data are in memory, and so on. As an internal structure, it must never be accessed even for read operations as its definition must be free to change. In general, you examine this structure in the NetWare System Debugger using the .m command.
errorscreen. If an NLM were to write out any messages about problems it encounters loading up, this would be the screen to do it on. One way to do this would be:
```
OutputToScreen(errorscreen, "NLM start-up error: %d\n", err);
```
OutputToScreen is prototyped in LibC's screen.h. This is almost always the System Console screen prior to NetWare 6 when the active system console screen became the Console Logger. errorscreen is usually the same handle you get by calling LibC's getconsolehandle. In NetWare 6 it is no longer possible to write to the System Console screen, but only to the Logger.
commandline. This is exactly what the NLM was loaded with and remains constant throughout the NLM's existence in memory. It is a mere string, as opposed to an argument list such as main's argv, which is manufactured by CLib or LibC on behalf of the starting NLM.
directorypath. This is the path to the directory containing the NLM that is being loaded. In an NLM linked with LibC's prelude, which we will be discussing later, this is used to set up the initial current working directory. For CLib NLMs, this is ignored and the first current working directory is sys:. You can get this live in an NLM by calling LibC's getnlmloadpath.
uninitializeddatalength. Uninitialized data is that data which is static (or global) to your NLM, but the starting value of which you do not explicitly specify to the compiler. This specifies its length. Unfortunately, without internal knowledge of the load-definition structure, you cannot discover the offset (address) of where this data starts if you want to zero it out. However, all environments (CodeWarrior, Watcom, and so on) have documented global variable(s) to deal with this situation. In addition, some linkers (such as later versions of Watcom's wlink.exe) zero out the data by default; others have a switch to cause it to be zeroed out (CodeWarrior's mwldnlm.exe uses -zerobss). If you link LibC's libcpre.o, this is done for you anyway (but not for re-entrant NLMs, as their uninitialized data may already have become the object of assigned values that should not be changed). This field is rarely used by any NLM start-up code.
nlmfilehandle. This is the NetWare OS's file handle on which the NLM binary is open. It's what is passed to the read function if that argument is ever used. This handle is not compatible with the descriptor argument to LibC's or CLib's POSIX functions like read, fcntl, and so on. Neither is it a stream such as would be passed to fread or fseek.
readfunction. This is the function that the Loader suggests an NLM should use to read any data directly out of its binary. CLib and LibC use this function, along with the nlmfilehandle, to read NLM information such as the description, stack size, thread name, screen name, and other information placed in the NLM at link time so that it can respect these parameters in creating the NLM. For example, both libraries create for the consuming NLM a screen with the name it was linked with. So the following link commands determine the name of the screen as it will appear at the console when the Alt key is held down, and in the screen list:
```
 SCREENAME                  "MyNLM"         [Metrowerks CodeWarrior Linker]
OPTION SCREENAME                  `MyNLM'         [Watcom Linker]
```
customdataoffset and customdatasize. These are mostly unused and will not be discussed here. Custom data of any sort can be placed inside an NLM binary for consumption by the NLM itself or anything else in any way it wants. For example, some NWSNUT-based NLMs (the C-Worthy interface MONITOR.NLM uses) put interface information such as strings in here.
messagecount and messages. These will not be discussed because NLMs typically get their messages by calling ReturnMessageInformation, a NetWare Loader call, rather than through these arguments. CLib returns this information using LoadLanguageMessageTable and LibC via the uname interface.

With this information, you can now write basic NLM start-up code that will run on any version of NetWare. As an example, we'll use a simplified version of LibC's start-up code in an effort to be realistic and give an idea of just what sorts of things are accomplished at start-up. This is not the advised way of doing start-up. The point behind this code isn't to educate you as to exactly what LibC does, but familiarize you with the sorts of things that are done at start-up. Feel free to review this code before going on with the next topic.

LibC Start-up Code

int _LibCStart
(
   T_LoadDefStruct               *NLMHandle,
   T_ScreenStruct               *screenID,
   const char               *cmdLineP
)
{
   int      loadedProtected, *debugLibraryP;
   char       *p, locale[MAX_LOCNAME_LEN];
   void      **preferredModule;

   // clear our zero-initialized data (how we do it in CodeWarrior)...
   memset(&__NLM_BSS_Start, 0, &__NLM_BSS_End - &__NLM_BSS_Start);

   loadedProtected = FALSE;

/*----------------------------------------------------------------------------
** The result of this function governs how LibC and CLib do their
** initializations and perform many other functions! Global SyntheticVersion
** is exported for use by both libraries.
**----------------------------------------------------------------------------
*/
   CalculateSyntheticVersion(NLMHandle);

/*----------------------------------------------------------------------------
** Check for flags on command line (for now, only one switch):
**
**   Load LibC.NLM -d --create debug screen for tracing
**----------------------------------------------------------------------------
*/
   for (p = (char *) cmdLineP; *p; p++)
   {
      if (*p == '-')
      {
         char  curr, next;

         curr = *++p;

         switch (curr)
         {
            case 'd' :   // create a debug screen and turn on debugging...
               next = *(p+1);
               if (next == ' ' || next == '\0')
                  debug = TRUE;
               break;

            default :
               break;
         }
      }
   }

/*----------------------------------------------------------------------------
** Hop into the NetWare System Debugger if the symbol is defined and set.
** Also, if available, set the preferred module handle to LibC so that we see
** LibC symbols in the debugger by default rather than CLib's or someone
** else's.
**----------------------------------------------------------------------------
*/
   debugLibraryP   = ImportPublicObject(NLMHandle, "DebugLibrary");
   preferredModule = ImportPublicObject(NLMHandle, "preferredModule");

   if (debugLibraryP && *debugLibraryP)
      EnterDebugger();

   if (preferredModule)
      *preferredModule = NLMHandle;

/*----------------------------------------------------------------------------
** Gather up some NetWare OS interfaces that aren't the same from version to
** version.
**----------------------------------------------------------------------------
*/
   InitOSInterfaces(NLMHandle);

/*----------------------------------------------------------------------------
** Load message strings in appropriate language. This function calls Return-
** MessageInformation, but upon failure, gives us a pointer to all our strings
** in English by default. These defaults are embedded inside us. In theory,
** this cannot fail.
**----------------------------------------------------------------------------
*/
   if (LoadMessages(NLMHandle, screenID))
      return -1;

/*----------------------------------------------------------------------------
** Initialize the library's global data structure so it can be filled in not
** only here, but by more initializations all along this function.
**----------------------------------------------------------------------------
*/
   memset(&LibCGlobals, 0, sizeof(LibCGlobals_t));

   LibCGlobals.signature                           = `LibC';
   LibCGlobals.handle                           = NLMHandle;
   LibCGlobals.console                           = screenID;
   LibCGlobals.CleanupLibCResources   CleanupLibCResources;

/*----------------------------------------------------------------------------
** Now do some version-specific initializations. Set flags describing the
** platform underneath us. This used to be more important when we spanned
** more than NetWare 5 and 6 because MP, VM and protected memory were absent.
**----------------------------------------------------------------------------
*/
   if (InitMPKInterfaces(NLMHandle))
   {
      OutputToScreen(screenID, MPK_KERNEL_ERR);
      return -1;
   }

   LibCGlobals.flags |= kRunningMoab;
   LibCGlobals.flags |= kMultiprocessing;
   LibCGlobals.flags |= kVirtualMemory;
   LibCGlobals.flags |= kProtectedMemory;

/*----------------------------------------------------------------------------
** Determine whether we're loading in a protected address space (ring 3) or in
** the kernel (ring 0). This will make a difference for whether we marshal
** interfaces or not. We are our own ring 0 nub in support of instances of us
** loaded in ring 3.
**----------------------------------------------------------------------------*/
   if (loadedProtected=nxIsLoadedProtected())
   {
      LibCGlobals.flags                  |= kLoadedProtected;
      loadedProtected                   = TRUE;
   }

/*----------------------------------------------------------------------------
** These resource tag allocations must be done before we can perform our init-
** ializations, especially allocation of any memory. There are other resource
** tag allocations, e.g., for our marshalling code.
**----------------------------------------------------------------------------
*/
   LibCGlobals.allocRTag = (rtag_t) AllocateResourceTag(NLMHandle,
                           "Private Memory Allocations", AllocSignature);
   if (!LibCGlobals.allocRTag)
   {
      OutputToScreen(screenID, ALLOCRTAG_ERR);
      return -1;
   }

   if (debug)
   {
      LibCGlobals.scrRTag = (rtag_t) AllocateResourceTag(NLMHandle,
                           "LibC Debugger Screen", ScreenSignature);
      if (!scrRTag)
      {
         OutputToScreen(screenID, SCRRTAG_WARN);
         goto NoScreenRTag;
      }

      OpenScreen("LibC Debug Trace", LibCGlobals.scrRTag,
                                       (void **) &LibCGlobals.debugScr);
NoScreenRTag :
      ;
   }

   LibCGlobals.workRTag = (rtag_t) AllocateResourceTag(NLMHandle,
                                       "Work", WorkCallBackSignature);
   if (!LibCGlobals.workRTag)
   {
      OutputToScreen(screenID, WORKRTAG_ERR);
      return -1;
   }

   LibCGlobals.eventRTag = (rtag_t) AllocateResourceTag(NLMHandle,
                                       "Private Events", EventSignature);
   if (!LibCGlobals.eventRTag)
   {
      OutputToScreen(screenID, EVENTRTAG_ERR);
      return -1;
   }

   // when available, NetWare Remote Management Portal is supported...
   if (!loadedProtected && !InitPortalInterfaces(NLMHandle))
      LibCGlobals.flags |= kHTTPStackUsed;

/*----------------------------------------------------------------------------
** Interface initializations must be done by here in order that the stubs may
** be set up to lower the cost of calling external interfaces through function
** pointers. Once these are implemented, they can be used. Nota bene: These
** may not consume LFS or NSS interfaces as those aren't even present when we
** make up the stubs as the system volume won't be mounted at this point when
** we load on NetWare 6. (On NetWare 5, the system volume usually has been
** mounted by the time we load.)
**----------------------------------------------------------------------------
*/
   if (ImplementStubCalls(NLMHandle, loadedProtected))
   {
      // our ring 0 nub not loaded! This is basically impossible, but...
      OutputToScreen(screenID, RING3_LOAD_ERR);
      return -1;
   }

   InitializeDebugger();

   if (!(LibCGlobals.flags & kLoadedProtected))
      InitRing0Globals(NLMHandle);

/*----------------------------------------------------------------------------
** Link this address space and LibC globals into the list.
**----------------------------------------------------------------------------
*/
   LibCGlobals.addrSpaceEntry = _create_address_space_entry(&LibCGlobals);

/*----------------------------------------------------------------------------
** Register to find out as soon as the system volume loads. There are things
** we need to do then including import legacy (traditional) file system
** interfaces and create LibC's own file and directory I/O context.
**----------------------------------------------------------------------------
*/
   LibCGlobals.sysVolMountID = RegisterForEventNotification(
                              LibCGlobals.eventRTag,
                              EVENT_VOL_SYS_MOUNT, EVENT_PRIORITY_OS,
                              (void *) NULL, ReportSysVolMount, NLMHandle);

/*----------------------------------------------------------------------------
** Bring in the NetWare legacy file system functions. If these aren't present,
** we're being loaded before the system volume mounts and will need to set up
** on the system volume-mount call-back when the functions become available.
**
** Now, on NetWare v5.1, the system volume is probably already mounted. We can
** determine this on the basis of the legacy file system interfaces being
** present, which they will be if the volume has already mounted. We'll just
** call the report function to do the work and unregister for the event
** notification as if we had been notified. Though present in NetWare 5, NSS
** doesn't provide an API set--we just call it through the legacy file system.
**
** In ring 3 this will have been futile since the system volume would
** obviously be up already, but it's well behaved for both NetWare 5 and 6 and
** we just clear the registration.
**----------------------------------------------------------------------------
*/
   if (InitLFSInterfaces(NLMHandle))
      return -1;

   ReportSysVolMount((void *) NULL, NLMHandle);

   InitNSSInterfaces();
   InitClusterInterfaces();

/*----------------------------------------------------------------------------
** If on NetWare 5 or later OS and loaded in ring 0 (kernel), set up the
** marshalling layer and also export explicitly those functions that only work
** from ring 0 and aren't implicitly exported via link statement.
**----------------------------------------------------------------------------
*/
   if (!loadedProtected)
      MarshalAndExportInterfaces(NLMHandle);

/*----------------------------------------------------------------------------
** Now perform trivial initializations and allocations for library components.
** Failure of these should be harmless, denying functionality rather leading
** to catastrophe.
**----------------------------------------------------------------------------
*/
   EstablishPageSize();

   LibCGlobals.minorTicksPerSec = _minor_ticks_per_second();

   _get_load_handle(RunningProcess, &LibCGlobals.system_handle);

   InitReqFunc();
   InitializeStandardConsoles();
   InitWinSockInterfaces();
   InitBSDInterfaces();

   if (InitUniLibGlobals(NLMHandle)) // no table files in...
   {                                 // ...C:\NWSERVER\NLS?
      OutputToScreen(screenID, UNITAB_INIT_ERR);
      return -1;
   }

   if (InitEnvironmentVariables(NLMHandle))  // mandatory
   {
      OutputToScreen(screenID, ENVVAR_INIT_ERR);
      return -1;
   }

   AllocateLibCGlobalLocks();
   AllocateFloatingStringLock();
   AllocateTZSetLock();
   SetTZDefaults();
   InitMathConstants();

/*----------------------------------------------------------------------------
** Register for notification of change of locale. We will empty our double-
** byte character table, our upper-case table and our collation table all
** based on the local code page. Then we will reload these plus re-derive the
** lower-case table. We will also change the language of our messages (to the
** extent that any of our messages are actually translated or otherwise
** localized).
**----------------------------------------------------------------------------
*/
   InitHostLocale(NLMHandle);

   gHostLocaleChangeID = RegisterForEventNotification(LibCGlobals.eventRTag,
                                 EVENT_LOCALE_FILE_CHANGED,
                                 EVENT_PRIORITY_OS, (void *) NULL,
                                 (void (*)(void*,void*)) LoadHostLocale,
                                 NLMHandle);

/*----------------------------------------------------------------------------
** Register to clean up resources at certain points.
**----------------------------------------------------------------------------
*/
   if (!(LibCGlobals.flags & kLoadedProtected))
   {
      LibCGlobals.cleanupEventID = RegisterForEventNotification(
                  LibCGlobals.eventRTag,
                  EVENT_MODULE_UNLOAD_POST_EXIT_ROUTINE
                     |EVENT_CONSUMER_MT_SAFE,
                  EVENT_PRIORITY_APPLICATION, NULL,
                  (void (*)(void*,void*)) CleanupLibCResources, NLMHandle);

      if( !LibCGlobals.cleanupEventID )
      {
         // NetWare 5 does not implement EVENT_CONSUMER_MT_SAFE correctly
         LibCGlobals.cleanupEventID = RegisterForEventNotification(
                  LibCGlobals.eventRTag,
                  EVENT_MODULE_UNLOAD_POST_EXIT_ROUTINE,
                  EVENT_PRIORITY_APPLICATION, NULL,
                  (void (*)(void*,void*)) CleanupLibCResources, NLMHandle);
      }
   }

/*----------------------------------------------------------------------------
** Initialize offsets used for swapping context in NXThreadSwapContext. This
** is good for everybody using this and the offsets can be calculated ahead of
** time. Also, we can do some work ahead of time for clients that are going to
** wrapper functions they export with code thunks that will ensure correct
** resource attribution.
**----------------------------------------------------------------------------
*/
   gOffsetOf_ctx_mctx                        = offsetof(ctx_t, ctx_mctx);
   gOffsetOf_ctx_current                        = offsetof(ctx_t, ctx_current);
   gOffsetOf_ctx_lock                        = offsetof(ctx_t, ctx_lock);
   gOffsetOf_ctx_stackInfo                        = offsetof(ctx_t, ctx_stackInfo);
   gOffsetOf_thr_stackInfo                        = offsetof(thr_t, thr_stackInfo);
   gOffsetOf_thr_ctx                        = offsetof(thr_t, ctx);

   InitThunkOffsets();

/*----------------------------------------------------------------------------
** Set up the library's notion of what the host locale is. Each client started
** up (each that has a main) will be endowed with the C locale per ANSI.
**----------------------------------------------------------------------------
*/
   derivelocale(NULL, NULL, locale);

   if (strcmp(locale, "C") || ReadLocale(locale, &g_default_locale))
      g_default_locale = g_C_locale;

   return 0;
}

Specifying Code Entry Points to the Linker

The start-up function is specified to the linker thus (we're still using the code from LibC listed above as our example):

START                _LibCStart                [Metrowerks CodeWarrior Linker]

    OPTION START     _LibCStart             [Watcom Linker]

It will come as no surprise that shut-down (or exit) basically reverses all that start-up does. Consequently, I am not offering LibC's shut-down sequence. It is important on NetWare to deallocate almost all resources before shutting down; otherwise, the NLM either crashes (is unloaded without freeing all its semphores), is subject to a hand-slapping or console "nastygram" about unfreed resources (such as memory), or places a permanent drain on kernel resources. This is because NetWare does not have a formal process model. Unloading an NLM is really a dynamic detach of a kernel extension. It is not simply a matter of tossing a process' pages in the trash as some other operating systems do.

The exit function is specified to the linker thus:

EXIT            _LibCExit                [Metrowerks CodeWarrior Linker]

    OPTION EXIT    _LibCExit             [Watcom Linker]

The final member of our NetWare start-up/shut-down troika is the check-unload function. Although most NLMs don't have one, it is fairly common nevertheless. The Loader calls the check-unload function whenever it is asked to unload an NLM. By providing this function, the NLM is basically saying, "I probably know better than the one trying to unload me whether it is wise to do so or not." Obviously, this could lead to a sort of program arrogance and the pitfalls are to be avoided: an NLM must be able to be taken down with only a small number of reasonable expectations.

The check-unload function is specified to the linker thus:

CHECK                CheckUnload                    [Metrowerks CodeWarrior Linker]

    OPTION CHECK     CheckUnload                 [Watcom Linker]

The check-unload function needs merely return 0 if it is okay to unload the NLM. Returning anything else is a signal to the Loader that there would be danger in unloading presently. The Loader issues a warning to the console to this effect; it is not usually necessary for the NLM check-unload function to do this. Neither CLib nor LibC have a check-unload procedure, for example. Note that since NetWare 5, it has been virtually impossible to get CLib down and leave something meaningful executing on the server. With LibC, it is completely impossible. So there is no reason for these libraries to be particularly cautious about going down themselves since it can only happen at server shutdown.

Here are some peculiarities of the check-unload function:

The function should execute without blocking. Operations that block include allocating memory with AllocSleepOK, malloc (try Alloc), acquiring a mutex and opening files or performing other I/O on them.
The function must not call any complex CLib functions because it does not have context to do so even if the NLM is a so-called CLib NLM. Functions such as strcpy and memset are obviously alright because they operate on data passed in. Complex functions tend to block anyway.
LibC functions may be called if the check-unload function is a LibC NLM. However, care must be taken not to block; this still rules out many library functions and knowing which ones block and which do not is not always completely obvious.
Messages from a check-unload function can be printed to the NetWare Console (Logger) screen using OutputToScreen or consoleprintf. CLib's ConsolePrintf is also consumable.
Check-unload functions do not work properly with a code offset of zero. The start and exist function offsets in the NLM format can be at offset 0, but since the check-unload function is optional, having it at offset zero is ambiguous with not having one at all. To avoid this problem, don't code this function at the top of the source compiled into the first object to be linked. If you are like most, your short NLM with few files will find your main coded in the first file specified to the linker and it will be somewhat natural to put your check-unload at the top of that file. Don't do this.

Below is a table that illustrates different types of code in NLMs and what sort of start-up entry points are most likely.

Type of Code	Start	Exit	Check
Application	main	atexit-registered functions and the like	any or _NonAppCheckUnload if guaranteed context wanted
Library	DllMain	DllMain	_NonAppCheckUnload
Driver (disk, LAN, etc.)	_NonAppStart	_NonAppStop	_NonAppCheckUnload
Other (protocol stack, namespace, etc.)	_NonAppStart	_NonAppStop	_NonAppCheckUnload

CLib Start-up Code

In this section, I'm not talking about CLib's own start-up code, but rather the code it creates for an application to consume by linking prelude.obj (or nwpre.obj). The point I was going to make about using Clib is that using it takes the problem of writing start-up code out of the developer's hands.

There are two reasons for doing this. The first is that it makes the CLib programming environment more like programming on other platforms such as UNIX and Windows NT where you don't do this for yourself. The second, more important, reason is that CLib is in a sense an "operating kernel" running on NetWare and so it needs to provide all of this to its consumers who want to behave just as applications do on other operating systems. CLib needs to carefully create a complex web of data structures that endow an application with all it expects from an environment with a standard C library. When Clib was created back in 1989, calling it was not something that could be done lightly. Standard C functions require data instancing to work, and the NetWare kernel provided no solution for this.

LibC Start-up Code and Data Instancing

LibC is the same in that it can provide everything an application needs in the way of instance data to run and consume a standard C library. Let's talk for a while about the data instancing both LibC and CLib do for the consuming application.

First, let's imagine an application calling read. This POSIX function takes a file descriptor as its first argument. The descriptor came from open, but how is the value of that simple, numeric descriptor chosen? Each NLM, just like each UNIX process, has its own descriptor table. Each and every NLM gets descriptor values 0, 1 and 2 wired up to indicate standard input, -output and -error. Therefore, there has to be something NLM-specific about descriptors if the 1 in every NLM that has one indicates a different screen than every other.

Each NLM receives a descriptor table that is unique and that is maintained for it by the library (whether CLib or LibC). The descriptor table is part of its NLM or application-specific instance data, or basically what we call its global variables. Other pieces of global data include the current working directory, its current locale, the signals it has registered to handle, one or more functions to be called upon exit, and so on. Add to that list a myriad of things that the library puts in that aren't even part of standard C, such as a pointer to the translation table to help get from the local code page to Unicode or UTF-8, or a pointer to an error handler to be called when math functions fail so that the application can handle such errors in its private way. There are many other obscure pieces and types of data maintained thus, just for the library's convenience.

Thread data is instanced as well. The most obvious is probably the "global" variable errno that is maintained separately for each thread. Others include (but are not limited to because there are yet many more):

Where the call to strtok left off (the third argument to strtok_r).
Ibid for unitok.
The current random seed and value in use for srand.
The results of the last calls to localtime, gmtime, and asctime.
A semaphore for use in implementing delay.
The last error to occur in dlfcn.h interfaces.
h_errno. as set by the likes of gethostbyaddr, gethostbyname, etc.
Storage for the last time tmpnam was called with none offered by the caller.

The standard C interfaces as documented by ISO/IEC 9899:1999 (ANSI) expect this sort of behavior. POSIX does not back each contextual case with reentrant interfaces, such as strtok_r, that would remove the problem.

While the details of start-up code executed on behalf of the library consumer are very complicated, it might be useful to reduce them to a simple set of steps. This is done in the code listing below with LibC's prelude code-the code LibC executes on behalf of NLMs that let it start them up. This comes with a disclaimer: this is not all the code there is, nor is it to be used to write start-up code for a LibC application. Despite this, LibC overcomes the problem that CLib created of disallowing an NLM's own start-up code.

Start-up Code by LibC on Behalf of Other NLMs

/*

** This is the NLM information structure that contains prelude version and

** flavor, the width of long doubles and wchars as well as some other stuff.

** Its principal use is to help identify when the NLM was linked which may

** determine some behavior in the library. For example, DllMain only became

** an option in Consolidated Service Pack 8 (NetWare 6 SP2, NetWare 5 SP5).

*/

NLMInfo            kLibCNLMInfo =

{

    NLM_INFO_SIGNATURE,    // signature

    LIBC_FLAVOR,                    // flavor

    LATEST_VERSION,                // version

    sizeof(long double),    // sizeof_long_double

    sizeof(wchar_t),                // sizeof_char

    0,                            // argc

    NULL,                            // argv

    NULL,                            // env

    FALSE,                        // callMain

    FALSE,                        // callStart

    FALSE,                        // callStop

    FALSE,                        // callCheck

    NULL,                            // vm

    FALSE                        // callDllMain (new for Nakoma or just before)

};



/*

** This is the reentrant flag. The first time loaded, it will be normal. It is

** set to true immediately and thereafter remains true even though an NLM,

** when linked with the REENTRANT flag, may be reinitialized over and over

** again. This flag is checked to avoid re-zeroing out the uninitialized data

** some of which may have been usefully initialized by the second load.

*/

#define TYPE_NORMAL                            1 // just an application

#define TYPE_REENTRANT                        2 // means don't reinitialize stuff...

#define TYPE_MULTILOAD                        3 // (actually undetectable at this point)



static int                 sNLMType = TYPE_NORMAL;

static void                *sNLMHandle = (void *) NULL;// save load-definition structure...

static void                *sDllHandle = (void *) NULL;// for DllMain() operation...



int _cstart_( void )

{

    return (*main)(kLibCNLMInfo.argc, kLibCNLMInfo.argv, kLibCNLMInfo.env);

}



int _LibCPrelude

(

    void                *NLMHandle,

    void                *errorScreen,

    const char                *commandLine,

    const char                *loadDirectoryPath,

    size_t            uninitializedDataLength,

    void                *NLMFileHandle,

    int            (*readRoutineP)(),

    size_t            customDataOffset,

    size_t            customDataSize,

    size_t            messageCount,

    const char                **messages

)

{

    int            err;

    T_SaveData                saveCtx;



    err        = 0;

    sNLMHandle = NLMHandle;



    if (sNLMType == TYPE_NORMAL)

    {

#if defined(__MWERKS__)

        extern unsigned char                            __NLM_BSS_Start, __NLM_BSS_End;



        (void) memset(&__NLM_BSS_Start, 0, &__NLM_BSS_End - &__NLM_BSS_Start);

#elif defined(__WATCOMC__) || defined(__GNUC__)

        extern unsigned char                            _edata, _end;



        memset(&_edata, 0, &_end - &_edata);

#else

        ;

#endif



        // Tool Maker Specification (Addendum) start-up initialization hook...

        if ((err = __init_environment(NULL)))

            return err;



        sNLMType = TYPE_REENTRANT; // (in case OS Loader sends us through again)

    }



    // some environments insist on "(CLIB_OPT)" for redirection which isn't LibC

    Delete_CLIB_OPT_FromCommandLine((char *) commandLine);



    // if pre-start fails we are finished, this NLM will not load...

    if (LibCPreStart(NLMHandle, errorScreen, main, _NonAppStart, _NonAppStop,

                                _NonAppCheckUnload, &kLibCNLMInfo, &saveCtx, DllMain))

    {

        return -1;

    }



    // this happens if the caller codes its own start-up code (a driver)...

    if (kLibCNLMInfo.callStart)

    {

        err = _NonAppStart(NLMHandle, errorScreen, commandLine,

                        loadDirectoryPath, uninitializedDataLength, NLMFileHandle,

                        readRoutineP, customDataOffset, customDataSize,

                        messageCount, messages);

    }



    // this happens if the caller has linked a DllMain() (a library)...

    if (kLibCNLMInfo.callDllMain)

    {

        sDllHandle = (void *) register_library(NULL);

        err        = (!DllMain(sDllHandle, DLL_PROCESS_ATTACH, NLMHandle));

    }



    // this happens if the caller has linked a main() (most applications)...

    if (kLibCNLMInfo.callMain)

    {

        err = StartNLMInNKS(NLMHandle, errorScreen, commandLine,

                        loadDirectoryPath, uninitializedDataLength, NLMFileHandle,

                        readRoutineP, customDataOffset, customDataSize,

                        &kLibCNLMInfo, _cstart_);

    }



    // pass error for clean-up in case of non-start...

    LibCPostStart(&kLibCNLMInfo, &saveCtx, err);



    return err;

}



void _LibCPostlude( void )

{

    void         *saveVM;



    (void) TerminateNLMFromNKS((void *) kLibCNLMInfo.vm);



    if (kLibCNLMInfo.callDllMain)

    {

        (void) DllMain(sDllHandle, DLL_PROCESS_DETACH, sNLMHandle);

        (void) unregister_library((int) sDllHandle);

    }



    if (kLibCNLMInfo.callStop)

    {

        saveVM = _set_vm_context(kLibCNLMInfo.vm);

         _NonAppStop();

        (void) _set_vm_context(saveVM);

    }



    // Tool Maker Specification (Addendum) shut-down initialization hook...

    (void) __deinit_environment(NULL);

}



int _LibCCheckUnload( void )

{

    int    err = 0;

    void         *saveVM;



    if (kLibCNLMInfo.callCheck)

    {

        saveVM = _set_vm_context(kLibCNLMInfo.vm);

        err    = _NonAppCheckUnload();



        (void) _set_vm_context(saveVM);

    }



    return err;

}

Writing Start-up Code in the 21st Century

With NetWare 6, it became possible for NLMs to have their own start-up code in place of or in addition to coding main, which was all CLib-based NLMs could do. This was done to open LibC to writing drivers and other low-level code like protocol stacks and libraries. Previously, driver writers were afraid to consume interfaces from CLib, even simple ones like memcpy, for two reasons:

CLib wasn't usually available as early in the load process as it would need to be to support drivers.
Code might be brought in from CLib that would cause a server Abend for lack of context.

Because of the first problem, drivers simply didn't link with CLib's prelude object. Another reason for this, in addition to the second problem, was a perception (not altogether erroneous) that CLib brought with it performance-crushing baggage that drivers preferred not to risk enduring.

With the advent of LibC, this all changed. For one thing, LibC loads before almost any other NLMs that aren't also linked into the server itself. For another, LibC creates, maintains, and disposes of context (another name for the underlying structures of data instancing) in a much lighter way than Clib did. These lower-level services can now link with libcpre.o and still write their own start-up code, even though varying amounts of LibC's will run as well. In this way, they have full access to LibC's complex interfaces (such as printf or fopen) if they want them. Of course, just because LibC is there doesn't mean all functionality will be available immediately. For example, if no NetWare volumes have been mounted yet, it will not be possible to perform file I/O.

In the second code listing above, the bold-faced functions are those that a developer would code if desired. If the NLM is to be an application, perhaps only main will be coded just as for a CLib NLM. If the NLM is to be a driver, perhaps only _NonAppStart, _NonAppStop and, optionally, _NonAppCheckUnload. No main need be coded.

If the NLM is to be a library, perhaps DllMain, just as for Windows NT, will be the only of all these functions to be coded. Writing libraries is a topic for a separate article, so I will not go into it here. DllMain handles start-up and shut-down for the library itself as well as process-attach, process-detach, and thread-detach for its "clients." Thread-attach is handled manually because help for accomplishing this is absent from NetWare kernels.

Because I'm not going to get into library technology, I'll just mention the simple fact that _NonAppStart is written identically in every way to any start-up code that might ever have been written by an NLM anytime between NetWare 3 and NetWare 6. The only constraint is that the start-up, shut-down, and check-unload functions receive the names, _NonAppStart, _NonAppStop, and _NonAppCheckUnload. Everything else can be done identically to how it was done before, which makes it very easy to move such code to LibC, presumably in search of modern synchronization and threading primitives unavailable before. This imposition is made because LibC has no other way of knowing the name of the start-up code.

The Cost of CLib or LibC Start-up Code

Some mention of the cost of all this "extra help" from CLib or LibC has been made. The point need not be belabored, but LibC only allocates resources for an NLM incrementally as needed. Callers to LibC do not need context in order to function, reversing the situation in CLib. In the worst case, only a few thousand bytes are incurred by consumers of LibC. LibC consumers that don't have a main consume less than those that do need a main.

Coding the Application main

The most common type of NLM codes main. Such an NLM is referred to as an application. Nevertheless, many have written libraries that code a main in which the active thread is used to accomplish library data-structure initialization before being parked either on a semaphore or by calling

ExitThread(TSR_THREAD, status);

While this was overkill, it did permit the library itself to be coded to CLib and to consume well-known functions such as malloc for memory allocation. To do it in a lighter way (without CLib), less well-known functions such as AllocSleepOK had to be used. LibC eliminates the gulf between itself and non-applications like libraries and drivers by making it always possible to code to malloc except under specific circumstances (again, a topic for a separate discussion).

So, what does it really mean to code main? Whether any of the other possible NLM entry points is present or not, this means that the NLM will:

Have an initial thread that runs until it terminates. Once this thread runs off the last right brace of main, the NLM will be terminated. To avoid terminating in this manner, link the NLM with the TSR flag. Other threads created and run to termination in the NLM will not bring it down; the initial thread is special in this way.
Get called with arguments parsed.
Get standard consoles (input-, output- and error) set up, including any redirection that might have been specified on the command line such as wiring standard output to a file instead of a screen by default.
Get environment variables set up.
Have many other full environment capabilities.

Not having a main is essentially programming a lite NLM. It's what drivers, protocol stacks and libraries do.

Conclusion

The purpose of this AppNote has been to discuss the history of start-up/shut-down code in NLMs, discuss the effect of using the standard programming environment (CLib and now LibC) on this code, and to urge all developers strongly to consider linking LibC's libcpre.o. This takes nothing away from their ability to code their own start-up and shut-down sequences; in fact, it adds greater clarity and freedom in doing it.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.