SERVER MEMORY: The NetWare 4 Memory Architecture
Articles and Tips: article
01 Apr 1995
If your responsibilities include interpreting MONITOR statistics and error messages, you need to understand NetWare 4's memory architecture. Here are some of the details you need to know.
For NetWare 4, Novell redesigned the server memory architecture to enhance performance and memory protection, and to reduce memory fragementation. Four major changes from NetWare 3 include:
- A new memory organization that eliminates some of the NetWare 3 memory pools (Permanent, Semi-Permanent, and Alloc Short-Term) to reduce memory fragmentation
- A new memory map that allows NetWare to write-protect operating system and NetWare Loadable Module (NLM) code space
- A new memory allocation scheme designed to be more in-tune with production server behaviors
- An optional memory domain feature which allows you to envelop and protect server applications
Memory Organization
NetWare uses a memory organization that divides the server's memory, beginning at address 0 to the end of physical memory, into useable units called "pages." In Intel-based systems, pages measure 4KB in size; on other hardware platforms, pages can vary between 512 bytes to 16KB. In every case, NetWare uses the host CPU's native page size.
Figure 1: NetWare 4 allocates memory in pages.
NetWare's paging organization should not be confused with virtual memory, in which unused pages are swapped to disk.
After operating system initialization, all remaining memory is given to the caching subsystem. As NLMs are loaded and request memory, NetWare takes the requested memory from cache in page-sized increments. When NLMs exit or discard unneeded memory, their pages are returned to cache (see Figure 1). Because the cache uses all memory not allocated by the OS and related NLMs, none of the server's memory goes unused.
This new organization overcomes several shortcomings in prior versions of NetWare. In NetWare 3, some memory pools permanently allocated memory that could never be returned for reuse by other NLMs. In NetWare 4, all memory allocated by NLMs is completely returnable and reusable, which greatly reduces memory fragmentation. In NetWare 3, memory pages allocated by NLMs were split up and shared my multiple NLMs. This interleaved organization made it difficult to reuse returned memory pages until all NLMs were finished with each shared page. To avoid this problem in NetWare 4, pages allocated by an NLM are not shared with other NLMs.
The NetWare Memory Map
NetWare uses a CPU facility that allows physical memory to be reorganized into a more efficient order. This process, called mapping, results in two or more views of memory:
The actual memory, called physical memory
The new view(s) of memory, called logical memory
The data structure providing translation between these two views is called a memory map. Using memory maps, NetWare reduces memory fragmentation by assembling non-contiguous blocks of physical memory into contiguous blocks of logical memory.
Figure 2 is a visual representation of how NetWare organizes server memory. NetWare loads its operating system code high in logical memory (in the F5000000 and F8000000 address spaces). All NLM code is also loaded into logical memory between F0000000 and F5000000, while the NLMs' data regions are mapped down in the physical memory area. With all of the code mapped high, NetWare takes advantage of memory write-protection services provided by the CPU to protect the OS and NLM code pools from being overwritten.
Figure 2: The NetWare 4 memory map.
Memory Allocation
NetWare's Alloc Subsystem manages free memory made up of blocks of memory left-over during previous allocation processes and blocks that have been de-allocated by NLMs. NetWare 3 managed free memory with one linked list made up of memory blocks in sizes ranging from very small (8bytes) to very large (>4KB). After a period of time, this free list became very large, including hundreds of small pieces of memory at the beginning of the list.
Each time an NLM allocated a block of memory, the Alloc Subsystem would traverse this free list looking for the first best fit. Ideally, this allocation process should take less than 100 instructions, but the size of the free list was often requiring thousands of instructions to find the best fit. This same process was repeated each time memory was returned to the free list.
To overcome the inherent latency involved in traversing large linked lists, NetWare 4 manages 77 separate free lists for each NLM. These free lists are based on size, and include:
An array of 64 free lists in 16-byte increments for memory blocks up to 1KB in size
An array of 12 free lists in 256-byte increments for memory blocks from 1KB to 4KB in size
A single free list for memory blocks larger than 4KB in size
This private allocation scheme can have a significant impact on memory performance in several ways.
First, the OS is now able to anticipate memory requests and prepare memory in advance. When an NLM allocates memory, the OS rounds the request up to the nearest 4KB increment and then doubles that amount before taking the total amount from cache.
This chunk of cache memory is then split up to satisfy the NLM's request. The remaining memory produced from doubling the request size is placed in one of the NLM's free lists in anticipation of similar requests of the same size. These extra memory blocks can have a significant impact on performance because requests serviced from the NLM's free lists are much less expensive in terms of CPU time than those serviced from the server's cache subsystem.
Second, this design is supported by our research findings that individual NLMs favor specific sizes of memory blocks. For example, one NLM may request memory in 32-byte and 2KB blocks, while another may request only 4KB blocks. This private allocation scheme provides each NLM with its own collection of right-sized blocks.
Memory Domains
DOMAIN.NLM is an optional memory feature, in the form of a NetWare Loadable Module (NLM), that allows you to protect the server's integrity while running untrusted NLMs. By loading DOMAIN.NLM, you create an "OS" domain surrounding the NetWare operating system, and an "AOS_PROTECTED" domain in which you can run untrusted NLMs (see Figure 3).
Using the CPU's segment and paging registers, DOMAIN sets the NLM's code segment register to the end of the NLM code space. This removes the NetWare OS, OS data space, cache control structures, and other unprotected NLMs from the NLM's view. DOMAIN also sets the protected NLM's data segment register to the end of physical memory to protect logical memory. Any violation of these boundaries will produce a segmentation violation or page fault, but will not interfere with the server's operation.
DOMAIN should not be used for anything besides protecting the OS from untrusted NLMs, because there are no other benefits.
Figure 3: The OS and OS_Protected domains.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.