Novell Storage Services (NSS) Performance Monitoring and Tuning
Articles and Tips: article
Asia-Pacific OS Mentor
Novell, Inc.
mculver@novell.com
Thanks to Paul Taysom, Randy Stokes, and Mike Hasleton of Novell for their invaluable assistance with this AppNote.
01 Jul 2002
This AppNote introduces various monitoring commands and tuning parameters that system administrators can use to optimize the performance of the Novell Storage Services (NSS) file system.
- Introduction
- Important Concepts to Understand
- Command Line Options for Monitoring NSS
- Performance Tuning Parameters
- Conclusion
|
Topics |
file system, Novell Storage Services (NSS), server management, performance tuning |
|
Products |
Novell Storage Services (NSS) |
|
Audience |
network administrators, support technicians |
|
Level |
advanced |
|
Prerequisite Skills |
familiarity with file system concepts |
|
Operating System |
NetWare 5.x, 6.x |
|
Tools |
none |
|
Sample Code |
no |
Introduction
Novell Storage Services (NSS) is a key technology in NetWare 6. Most customers who are running NetWare 6 have automatically chosen to run NSS without hesitation; however, little information is available on how NSS works or how to tune it. One common misconception about NSS is that it is slow. In truth, NSS is considerably faster than most file systems, including NetWare's traditional (legacy) file system in some cases.
NetWare 6 is pre-configured for a "typical" 100-user environment. As with most products Novell makes, it is a good idea to spend a little time tweaking the NSS settings for your particular environment. Under certain conditions, the performance of NSS can be tuned to great benefit. What's amazing is that you can keep the flexibility and extensibility of NSS while maintaining NetWare's legendary performance.
NSS tuning can be proactive, to get the highest possible speed from your storage subsystem, or reactive, because there is a problem. No matter which camp you are in, this AppNote should help you understand each of the tuning parameters available in NSS and how to use them. Hopefully you will learn how to tune NSS for your environment and how to optimize its performance.
Note: The information in this AppNote is based on the version of NSS 3.x contained in Service Pack 2 for NetWare 6.0. Where possible, I have made references to older versions of NetWare and NSS. However, as a general rule, the older your product versions are, the less this information will apply.
Important Concepts to Understand
This section provides a quick overview of some important concepts to understand in relation to NSS.
How Hashing Works in NSS
In NSS, a hashing algorithm is used to map each cache entry to a unique number. This number is a reference or location in a list of entries so that NSS knows where to read or write information in the file system. Each of these hash table entries is called a "hash bucket" in NSS.
For a hashing algorithm to be effective, it must be fast and it must be reversible. When an entry is written that has been keyed from its name, NSS must be able to use the same algorithm to know where to look in the list of entries to obtain the information when it is searched for later.
In NSS, the effectiveness of the hashing algorithm is determined by what percentage of buckets are used and how evenly the entries are distributed across all buckets. It is always faster to look up an entry in a larger list of hash entries than a longer chain within a single hash entry.
Figure 1 shows a chain of entries in hash buckets.
Chain of entries in hash buckets.
As a fictitious example, assume there is a file named COST.RTF. A hashing algorithm could be used that takes every two bytes in the file name, combines the value, and totals the values to get a hash bucket number. Thus, for this example, 275 is the result of the hashing algorithm:
CO = 77 ST = 66 .R = 34 TF = 98 Total = 275
NSS now knows which hash bucket to look in when searching for this file: bucket 275.
It doesn't take much thought to realize that this example would be a bad hashing algorithm. It would allow clusters to form at certain points in the hash table list, while buckets at the beginning and end of the table would be empty. NSS employs a more sophisticated hashing system than the example given above.
System and User Cache
In NSS 3.x, the cache performance statistics are divided into SYS (system) and USER cache. System data is everything that isn't user data. It includes the roll-back ZLOG (Journal) and the roll-forward Purge Log, the various databases and structures where NSS keeps information about files-for example, the name tree, the beast tree, and the visibility list.
By default, 30% of the available cache buffers are reserved for system blocks. This is not a function of the ZLSS file system (the default file system for NSS), but applies to all Loadable Storage Systems (LSSs). If user data from cached files leaves no space for the system data, the file system performance can suffer.
The cache algorithms that NSS uses are very advanced. Under normal circumstances, you should see a 98% (or better) hit rate for user data cache and better than 98% for system data. In some cases it is usual to see lower cache rates-for example, in situations where backups have run, or when large file copies have just occurred, or in a business that deals with large files that consistently flush the cache. If you want to raise the cache statistics in these situations, you will need to dedicate more RAM to NSS (as explained later in this AppNote).
How Hashing Works for the File System Cache
NSS allocates a hash table for the file system cache based on the amount of memory in the server. This hash table size is not configurable; however, adding RAM will increase it.
Unlike with the Name Cache, NSS usually does not have issues with the length of chains in the File System Cache. Usually there are far more hash buckets available than entries used. Therefore, it is common to see a small percentage of buckets used and a low maximum entries in a bucket.
If there are more than 10-15 cache buffers in a bucket for an extended period of time, you should consider dedicating more RAM to NSS file system cache.
How Hashing Works for the Name Cache
By default, NSS 3.x allocates 2111 hash buckets for the name cache. This is a fixed amount, unlike the number of cache entries that could possibly be stored, which also starts at 2111 but which can be increased to 65,521.
Increasing the name cache size will increase the hash table size. However, the relationship of entries to hash buckets is not proportional in the name cache. Thus, if you increase the name cache size to 65,521, the actual hash table size will only increase to about 30,000.
Increasing the number of entries in the name cache will, by the very nature of the algorithm, cause greater lengths of chains to develop, as more entries are hashed to the same value in the table. Therefore, increasing the name cache will actually increase the maximum entries in a hash bucket.
It is always faster to look up a hash table than to walk through chained entries in a hash bucket. Novell is currently in the process of changing the size of the hash table in the name cache to more dynamically reflect the number of entries. Further, Novell plans to increase the current limit of 65,521 cached names.
There is always a trade-off in speed between looking up the entry in the b-tree and the time taken to walk through a chained list in a hash table. Usually the cache will win out. On average, it takes twice as long to search the name tree than to look up a cache entry for name resolution.
Additional Notes on the Name Cache
NetWare runs two name caches in NSS: ASCII and UniCode. Most attempts are made on the ASCII cache, as the Client32 code was written for traditional volumes only capable of ASCII lookups. Despite the fact that Windows NT uses UniCode natively, Novell's implementation of the Microsoft Common Internet File System (CIFS) will also convert requests for NSS to ASCII. (Novell plans to change its implementation of CIFS to make requests in UniCode.)
NSS runs both UniCode and ASCII caches. For requests that come in ASCII format for which the file name is not found in the ASCII cache, NSS will convert the request to UniCode and search the cache before going to scan the Name b-tree.
Note: In NetWare 6 SP1 and SP2, cache performance statistics on the ASCII Name Cache are incorrect, showing an almost impossible value for "Num cache hits" due to a high "Num positive cache hits" value. The "Cache hit percentage" is also incorrectly calculated. Novell plans to resolve these issues in NetWare 6 SP3.
To determine whether the cache size is adequate, look at the Cache hit percentage. Under normal usage this usually stays in the high ninety percent range. If this appears to be too low and the number of Victim Selects is high, the cache is thrashing (being consistently flushed by new data). Increasing the cache size will help performance.
Command Line Options for Monitoring NSS
Before you worry about tuning NSS, you must first determine whether NSS tuning is required and what aspect of the file system needs tuning. There are two ways of determining if NSS requires tuning:
If your server suffers from poor performance and you suspect the storage subsystem
By monitoring the storage subsystem using the cache performance statistics tools provided with NSS
The NSS command line options described in this section will help you monitor its performance and thereby assist in determining what, if any, tuning is required. These are all server commands entered at the server console.
Note: You can abbreviate these NSS command options by providing just enough of the command to be unique. For example, "NSS /CA" would be unique enough to be an abbreviation for the "NSS /CacheStats" command.
NSS / ResetStats
This command can be used to clear the file system cache statistics and have them start anew. It is useful for checking the results of your changes in the file system cache.
Before executing this command, you should note how long the server has been up. If it has been up for weeks or months, you will need to wait a significant period of time before regarding the new statistics as valid-at least two full working business days, preferably a week.
Note: This command affects only the file system cache. Other cache statistics (such as those displayed via the /NameCacheStats command) are unaffected and will require a system reboot to clear.
NSS / CacheStats or /Cache
This command gives information on the performance of the file system cache. Most importantly, it gives information on cache hits, cache misses, and the number of cached entries.
Figure 2 shows an example of the statistics displayed when the NSS /CacheStats command is run at the server console.
Output from the NSS /CacheStats command.
Following are descriptions of the listed settings.
Min cache buffers. This is the minimum (fixed) number of 4KB cache buffers assigned to NSS. NSS will never allocate less than this number of cache buffers. (Compare this setting with "Num cache pages allocated" below.)
Num hash buckets. This is the number of hash buckets used for tracking cache buffers. To quickly look up cache buffers, NSS uses a hashing algorithm to place entries into hash buckets and then chains the hash buckets together to form a quick link to the information required. The number of hash buckets is proportional to the amount of free memory available when NSS is loaded. (For more information, refer to the "How Hashing Works in NSS" section of this AppNote.)
Min OS free cache buffers. NSS ensures that NetWare keeps at least this many 4KB pages available for the traditional NetWare file system and NetWare Loadable Modules (NLMs). It does this by giving pages back to the OS until either the OS has enough pages or NSS has reached its minimum number of pages (see "Min cache buffers" above).
Num cache pages allocated. This is the current memory used by the NSS file system cache, expressed in 4KB pages allocated to NSS.
To calculate the amount of memory NSS is currently using for file system cache, multiply the "Num cache pages allocated" by 4KB.
Cache hit percentage. Commonly referred to as the cache hit ratio, this is total number of cache hits divided by the number of attempts to get a file cache buffer from the cache. In equation form, that's CacheHit / (CacheHit + CacheMiss).
Cache hit. This value represents the number of times NSS found the desired file block in file system cache. (For more information, refer to the "System and User Data Cache" section of this AppNote.)
Cache miss. This value represents number of times NSS was not able to find the desired file block in file system cache. (For more information, refer to the "System and User Data Cache" section of this AppNote.)
Percent of buckets used. This statistic represents the current number of hash buckets that contain cache buffers. Since NSS allocates the hash size based on the amount of memory, this is more informational for Novell than for the administrator, as you cannot configure the buffer cache's hash table size.
Max entries in a bucket. This is the longest chain of cache buffers contained in a single bucket. The hashing algorithm used should place few entries in each bucket to improve search performance. However, larger chains can be formed. (For more information, refer to the "How Hashing Works for the File System Cache" section of this AppNote.)
Total entries. This is the total number of cache buffers currently being used. The rest are on a free list.
Some people are concerned when the server appears to be idle but the User hits are growing by hundreds or thousands per second. Keep in mind that server-based applications such as Novell eDirectory are always accessing the file system, even if you disconnect the network cable.
NSS /Status
The NSS /Status command can be used to see helpful "at-a-glance" information about NSS configuration. It includes most of the important NSS settings you will need to know about NSS on a server.
Figure 3 shows an example of the information displayed by this command.
Output of the NSS /Status command.
You can think of the NSS /status command as a configuration report or CONFIG.TXT for NSS.
NSS /NameCacheStats
The NSS Name Cache is responsible for caching the Name Tree information. This is the information that is read when you perform any kind of search by file or directory name. The name cache maps a name to a ZID. Directory listings do not do this as much as normal file opens that must resolve each name in the file path.
Note: A ZID is a unique ID number to reference file meta-data. This information is held in a "beast" object in the beast tree; therefore a ZID references a beast. Throughout the NSS statistics, you will see reference to the letter Z. This reference reinforces Novell's opinion of NSS as the last file system you'll ever need.
Some people are confused when increasing the amount of Name Cache does not improve the performance of getting directory listing information. The reason is that the enumeration of directories requires information in the name tree that is not cached. It is therefore probable (depending on the client) that once the name is fetched from the name tree during the enumeration process, it will request further information about the file from a tree or structure outside of the name tree. The process of fetching this additional information may be done by ZID or by name. If it is done by name, the name cache will be involved to get the ZID; however, NSS may still have to look up non-cached information about the file by the beast ZID.
If you want to see how your name cache is performing, use the NSS /NameCacheStats command. Figure 4 shows an example of the information displayed by this command.
Output of the NSS /NameCacheStats command.
Here is a description of the statistics displayed by this command.
Num hash buckets. As with the file system cache, hash buckets are used to enable quick lookups for name cache entries. NSS uses a hashing algorithm to place the entries into buckets and then chains the entries together to form a quick link to the information required.
Max cache entries. This value indicates the maximum number of Name Tree cache entries that can be cached; this is controlled by the /NameCacheSize parameter.
There is one cache entry for each name being cached. This value does not relate to hash buckets, as the number of entries chained from a bucket depends on how many names hash to the same bucket. A single name entry is never found in multiple cache buckets. (Keep in mind that NSS runs both UniCode and ASCII name caches for speed.)
Num entries in use. This value indicates the current number of Name Tree cache entries that have been cached. Except at startup time, this will generally be within one or two entries of the maximum setting.
Num entries in unused list. This is a system test statistic left over from the initial development of NSS. These statistics were put in to track possible conflicts and problems so that the engineers could address them quickly and make sure everything was working correctly. Novell plans to remove this parameter in NetWare 6 SP3 and higher, since it has no bearing on system performance.
Num packet allocates. The default name size is 32 characters. If a name longer than this requires storing, NSS will have to allocate a special packet for it. This parameter keeps track of these allocations.
Num packet reuses. This is another system test statistic left over from the initial development of NSS. Novell plans to remove this parameter in NetWare 6 SP3 and higher, since it has no bearing on system performance.
Num unused packets freed. This parameter tracks the freeing of special packets that are used when a name is longer than 32 characters.
Num times count change. This is the number of times NSS attempted to add to the name cache while it was already in the process of adding.
Change count (num adds). This is the number of positive adds made to the cache. When a name is found, it is added to the cache and this value is incremented.
Num negative adds. This is the number of negative adds made to the cache. When a name is not found, it is added to the cache and this value is incremented. (Negative and positive cache are discussed in more detail in "Num positive cache hits" and "Num negative cache" hits below.)
Num duplicate adds. You will rarely see this counter increment. It is a leftover from development to track when the same name cache entry attempts to add on two threads at the same time.
Num victim selects. This is the number of names (files/subdirectories) that have been forcibly removed from the name cache because a new entry had to be added and there wasn't enough room. Typically the entry that is "victim selected" is the one that has not been accessed for the longest amount of time.
If this statistic is high and constantly increasing, it is a good indicator that your server is flushing its name cache very quickly. In this case, you should increase the name cache significantly.
Num removed entries. If a file/subdirectory is deleted, it will be removed from the cache. Likewise when a file/subdirectory is created, it will be removed from the negative cache. Either action will increment this counter. (Negative and positive cache are discussed in more detail in "Num positive cache hits" and "Num negative cache hits" below.)
Num attempts on cache. This is the number of attempts that have been made to resolve data from the Name Tree cache. When a request is made to search the name tree cache, this counter is incremented.
Num cache hits. This counter tracks the number of times NSS finds a result of a search against the name tree cache. If you subtract the number of cache hits from this total, you get the number of cache misses.
Cache hit percentage. This is the number of attempts on cache divided by the number of cache hits. Usually this percentage should be in the high 90s for a server that has been up at least a week; if it is not, you should add more name cache.
Num positive cache hits and Num negative cache hits. When a file system operation attempts to resolve a name it will find either an object associated with the name or it will not: either way NSS will cache the results.
A cache "hit" occurs when NSS finds a name in cache during a search. If the name represents a file that exists, that is a positive name cache hit. If it represents a file that does not exist, that is a negative name cache hit.
Many applications continually try to open files that do not exist (such as optional configurational, backup, and temporary files). The amount of time it takes to determine if a file does not exist can be as slow as the amount of time it takes to find a file that is there. Therefore, having a negative name cache can speed up overall system performance.
Percent of buckets used. This statistic represents the current number of hash buckets that contain cache buffers. This is useful in determining how much name cache should be added.
Max entries in a bucket. The cache uses a hashing algorithm that maps names to buckets. At times, multiple names map to the same bucket. In this case, all names mapped to the same bucket must be searched to see if the entry is in cache.
This value indicates the longest chain of name cache entries contained in a single bucket. The length of the chain is a function of the hashing function. If the percent of buckets used is low and the chain value is high, you would have a bad hashing algorithm. This has not been seen in NSS.
If you allocate more cache entries for NSS, you should expect to see longer chains because it does not allocate a new bucket for each new name cache entry. On a normal NetWare 6 server, this figure should reside at about 10.
Volume Inventory
Tuning in NSS requires that you understand the data type, structure, and frequency of access. Before NetWare 6 SP2, this was mostly guesswork; however, in SP2 Novell added a Volume Inventory feature to the Novell Remote Manager tool.
Accessing the Volume Inventory is easy-just click on Volume Information and select "Volume Inventory Report." You will see a report similar to the one shown in Figure 5.
Volume Information report from the Novell Remote Manager utility.
Be aware that this report will take several minutes to run. It will cause only a moderate amount of CPU utilisation, but it will consume nearly all available disk I/O bandwidth. For that reason, NSS keeps the results of the last search and when this option is selected again, it prompts you to either create a new report or use the last report.
This report shows the following information by bytes in use and file count:
File types
Last modified times
Last accessed times
File size ranges
The Volume Information report gives you a detailed breakdown of the graphs below with hyperlinks. Not only does it tell you the constitution of the volume, you can actually go and check out the files that fall into this category. The files themselves are also hyperlinked so you can perform trustee/flag/delete/ rename operations, plus any lock information and so forth, straight off the link.
Knowing information such as the average file size can help you in working with some of the performance tuning indicators already discussed. It is a must before looking at the performance tuning parameters (such as /AllocAheadBlks) that are described in the next section.
Performance Tuning Parameters
Once you have established the need for performance tuning in NSS, the process of tuning can be a bit challenging. The traditional NetWare file system had very few parameters to tune; it was inflexible and had only one speed-fast. Tuning NSS is a matter of modifying a parameter or two, resetting the cache performance statistics, and then waiting and watching for performance gains or degradation.
For small file write operations, NSS may be slower than the traditional file system for several important reasons: it is a journaling file system and it uses Unicode instead of ASCII. When you couple that with the fact that most programs run on a NetWare server use small files, you will quickly realize that in some cases you may not be able to tune NSS to the performance of the traditional file system. However, you will be able to considerably improve its performance over the default settings.
In some situations and with some applications, the performance of NSS is significantly better than that of the traditional file system. As an example of this, Novell's internal testing has shown that GroupWise 6 can actually run faster on NSS 3.0 than on the traditional file system once it is tuned correctly.
It is important to realize that NSS was not designed to win benchmarking tests. It was designed by looking at actual data usage patterns and therefore, in a production environment, it will usually outperform any other journaled file systems that may benchmark above it. It should also be noted that NSS 3.x is significantly faster than NSS 2.x and usually faster than the traditional file system in the real world. Some basic testing may not show this difference.
Because NSS was designed with random access in mind, its performance may suffer against some other file systems for operations that require sequential b-tree access such as running the DOS DIR command. Novell considers it is more important that NSS run well in a large environment than it return to the DOS prompt in a shorter period of time.
Warning: Before you begin to adjust settings, you need to be aware that if you give NSS more memory than the server has, NSS will fail to load. If your SYS volume is NSS-based, your server will be inoperable until such time as you load the server with a NSS override to correct the situation.
If you need to restore a setting after a failed tuning attempt, use the server command line argument of -z"parameter". For example:
server -z"/namecachesize=xxx /cachebalance=yy /numbonds=zzzzz"
Alternatively, you can create a file named NSSSTART.CFG in the C:\NWSERVER folder and put the desired parameters into it as follows:
/namecachesize=xxx /cachebalance=yy /numbonds=zzzzz
General NSS Tuning Considerations
Here are some general considerations to bear in mind when tuning NSS.
Patches. Before you start any form of operation on NSS, always ensure that you are running the latest version of NSS. Novell's dedicated team of programmers are consistently looking for ways to improve the performance of NSS, which are typically added in incremental patches between major version releases.
No Other File System. Without a doubt, the most important consideration for NSS is this: don't split your resources between two file systems.
When FILESYS loads, RAM not allocated is given to legacy cache. When NSS loads, it will take (by default) 60% of that RAM for NSS. As the two file system caches are disparate, this will halve the performance of both caches compared to a single file system server. When applications load the amount of cache for both file systems, performance is reduced further.
If you must run multiple file systems, try to get to know the usage patterns of each system. This may enable you to balance RAM between them more effectively.
RAM. As with any file system cache, NSS performs better with more memory. You may find that the performance modifications you make to tune NSS require the purchase of more RAM for your server.
Do not add RAM over 4GB to improve cache performance. The Intel architecture has no DMA capabilities above 4GB; therefore, NSS deliberately does not use this memory.
The following tuning commands are entered at the NetWare server console prompt.
NSS /CacheUserMaxPercent=xx
This is the parameter for controlling the division of the file system cache between User and System data. The default percentage for User data cache is 70%.
The ideal percentage depends on the mix of operations occurring in the system. Generally speaking, if you have a server with few files and volume attributes and a generous amount of cache RAM, you can increase this parameter. Remember that whatever is not System cache is User cache.
NSS /CacheBalance=xx and NSS /MinBufferCacheSize=n
Every 30 seconds (by default) and/or when the server loads or unloads an NLM, NSS rebalances the NSS file system cache to the percentage specified by the /CacheBalance parameter. By default this is set to 60%; the valid range is 1-99 expressed as percentage. (You can change the default time interval with the /CacheBalanceTimer command. The range for this is 1-3600 seconds.)
The /MinBufferCacheSize represents the minimum amount of file system cache NSS will use. It will never balance the cache below the /MinBufferCacheSize. By default it is set to 512 * 4KB buffers, or 2MB. The valid range is 256-1,048,576.
On a server with only NSS volumes, you can usually adjust the /CacheBalance parameter upwards of 85% without any problem; some customers have reported that they can adjust it in excess of 95%. Monitor the Least Recently Used (LRU) Sitting Time on the NetWare server to ensure that you are not running the server too low on memory.
Note: LRU Sitting Time can be found in the MONITOR utility under "Disk Cache Utilization." Given that the server has been up for over 24 hours, this statistic should never drop below 24 hours. It is important to understand that LRU Sitting Time is a measure of non-cache memory, not file system cache memory.
No matter how file system cache is allocated in NetWare 6, it will show as "Cache Buffers" in MONITOR. This is because NetWare 6 can now request memory from NSS as it would from the traditional file system. As such, the /CacheBalance parameter is still relevant even if you just have NSS. The difference is that it can set the CacheBalance to a higher value than was possible in NetWare 5.x.
If you are concerned about the amount of CPU time being spent by the CacheBalance process, disable it and use the /MinBufferCacheSize command. This assumes you know the amount of RAM you want to specify for NSS file system cache. NSS makes no allowance for the misconfiguration of this setting! Novell recommends that you increase the setting over time and watch the LRU Sitting Time in MONITOR.
You can reduce either the /CacheBalance or the /MinBufferCacheSize. However, only the /CacheBalance can be disabled. File system cache will always consume memory up to its limit from the time you load NSS, regardless of which method you use.
NSS /MinOSBufferCacheSize=n
If you allocate too much memory to your NSS cache, you may run the server out of memory. You can safeguard against this possibility by setting the minimum amount of RAM reserved for the operating system via the /MinOSBufferCache- Size command. This value is set to 256 * 4KB buffers (1MB) by default. The valid range for n is 256-1,048,576.
NSS /CacheBalanceMaxBuffersPerSession=n
Usually when the /CacheBalance is recalculated, a small amount of RAM is either given or taken away from NSS Cache. This parameter controls the maximum cache balance transfer (in 4KB blocks) given to the NSS file system cache per cache balance. By default, this parameter is set to 1024 (4MB) per session. The valid range is 16 -1,048,576.
If your server has applications running on it that can consume significant amounts of RAM for a short period of time, you may want to increase this parameter. Otherwise your server may take several minutes to get back to the correct amount of cache dictated by the /CacheBalance parameter.
This parameter does not affect memory requests made of NSS by the OS. For example, if NSS is asked to give back 5MB of RAM, it will comply at the time of the request in one transfer.
NSS /NameCache=n
NSS will cache as many recently used Name Tree entries as specified by this parameter. By default, /NameCache is set to 2111; the valid range for n is 3-65,521. Each entry uses about 150 bytes of memory.
In medium to large servers with enough RAM, you should consider increasing this value to at least 40,000.
Note: Name cache will grow up to the specified limit as required. Unlike file system cache, it will not take the maximum amount of memory it needs from the start.
NSS /OpenFileHashShift=n
Once NSS has looked up the Name Cache to resolve the beast ZID to open a file, it must then go to the beast object to get file information (such as where it resides, trustees, and so on). This information is cached in the Open and Closed file hash table. NSS will check for it first before going to the b-tree on disk.
This parameter sets the hash table size for the beast hash that includes opened and closed files. It is important to understand that there is a relationship between the open and closed files being cached. Both open and closed files are stored in one hash table and are distinguished only by the hash table entries denoting which are closed files (and therefore, by logical exclusion, which are open).
As the total number of open files is added to the number of cached closed entries, it becomes apparent that NSS could develop long chains from a single hash bucket and performance will suffer significantly. If your server usually runs with many open files, you should increase this parameter. As a minimum, this parameter should be set to at least 25% of the size of your closed file cache.
Unlike the Name Tree where a cache miss results in only a 50% reduction in access time, missing this cache and going to disk will cause a significant performance penalty. If your server has the RAM, Novell recommends a 1:1 relationship for open+closed files to the OpenFileHashShift.
This parameter is adjusted by powers of two. By default it is set to 2^15; the valid range is 8-25. Each empty hash bucket consumes 4 bytes of RAM, and each entry in a hash bucket consumes 400-1000 bytes.
Here is a quick table to help you determine a suitable setting:
2^15 = 32,768 hash table entries 2^16 = 65,536 hash table entries 2^17 = 131,072 hash table entries 2^18 = 262,144 hash table entries 2^19 = 524,288 hash table entries 2^20 = 1,048,574 hash table entries 2^25 = 33,554,432 hash table entries
NSS /ClosedFileCacheSize=n
Obtaining beast (file) information from disk is an expensive process in terms of performance. If NSS can get the information from memory, it will enable file handles to be given quicker. The /ClosedFileCacheSize parameter keeps these beast objects in cache so NSS doesn't have to go to disk and unpack beast information again when it wants the same file.
In NSS 3.0, the default size is 50,000; the valid range is 16-1,000,000. Novell recommends setting this parameter to 100,000 or more if you have applications on your server that consistently cycle through the same set of files and you suspect that this cache is being flushed when combined with normal server operations. On average, each Closed File Cache entry consumes 0.4-1KB of RAM.
Note: The Closed File Cache will grow up to the specified limit as required. Unlike file system cache, it does not take the maximum amount of memory it needs from the start. It also adapts-as memory is consumed by other processes, it dynamically reduces the number of entries so the system does not become starved for memory.
NSS /NoDataShredding
NSS 3.0 gives users the ability to write over purged files with bit patterns up to seven times. This operation is called "data shredding." Unless you must use this feature for security reasons, it should be disabled, as data shredding consumes a great deal of disk I/O bandwidth. Data shredding can be disabled as a volume attribute from ConsoleOne, or via this command:
NSS /nodatashredding=volumename
Reducing or Increasing /AllocAheadBlks
The NSS /AllocAheadBlks parameter determines how many 4KB blocks NSS automatically allocates when it writes a new file (of any size) to disk. This parameter does not apply to writing to existing files. NSS dynamically decides how much data (up to the limit you specify in /AllocAheadBlks) to reserve when it writes, based upon the file size.
If you know that you are going to be writing smaller files consistently, you can usually reduce the /AllocAheadBlks value from the default of 15 (70KB) to 4-5 to improve performance. The valid range for this parameter is 0 - 63.
Some applications write smaller files by default. Novell GroupWise, NIMS/ NetMail, and BorderManager are examples of this behavior. For these applications, it is wasteful and expensive in terms of performance to spend the time to allocate extra blocks when NSS writes a file. You should set the /AllocAheadBlks to the average file size that gets created most frequently. Check the Volume Inventory in the Novell Remote Manager utility for a detailed breakdown of average file sizes.
Note: Depending on the behavior of the application in question, reducing this parameter may cause fragmentation in larger files. However, most modern desktop-based applications rewrite the entire file when they save it.
NSS /Compression
As storage technology improves with cheaper, larger media, the necessity of compression is called into question. Because compression slows down file access, you should not enable this feature unless you are sure you need it. By default, compression is not enabled.
Many administrators fall into the trap of thinking that compression will compress large files well. In actual fact, the compression routines used in NSS are taken directly from the traditional file system written in the 1990s, which ignore files above 256MB in size. In addition, compression limits the usefulness of such features as Pooling and Overbooking as you must reserve a area of disk for compression to work in. (For more information about these and other NSS features, see the online documentation at http://www.novell.com/documentation/ lg/nw6p/index.html?nss_enu/data/h5grftdf.html.)
As with the traditional file system, once compression is enabled on an NSS volume, you must recreate the volume to disable it. Compression can be enabled as a volume attribute option in ConsoleOne or via this command:
NSS /compression=volumename
NSS /NumWorkToDos=n
This specifies the number of work-to-do threads that NSS will use at any given time. By default this is set to 50; the valid range is 5-100. These threads are all background threads. For most situations, the default number of work-to-dos should be just fine. However, on heavily used servers with large disk farms, you can try increasing this number to 75 or greater.
NSS /MailboxSize
In NetWare 5.x, the mailbox size referred to the number of interrupt callback requests that NSS could queue up for the driver in question to respond to an acknowledged interrupt with a callback routine.
In NetWare 6, this parameter is no longer effective. The code is bypassed and any change made to this parameter will not make the slightest difference.
NSS /FileFlushTimer and /BufferFlushTimer
The /BufferFlushTimer parameter sets the maximum amount of time to keep a modified NSS file system cache buffer in memory before writing it. By default this is set to 1 second; the valid range is 1-3600.
The /FileFlushTimer parameter represents the maximum amount of time to keep a modified file in cache before flushing it to disk. By default this is set to 10 seconds; the valid range is 1-3600.
Increasing these settings will improve performance, but at the risk of data loss during a system failure. They directly relate to the amount of uncommitted data in NSS Cache.
Note: Increasing the /FileFlushTimer setting beyond a minute or two can lead to overflowing the ZLOG file and throttling system operations.
NSS /AuthCacheSize=n
The /AuthCacheSize reflects the number of cached authorisation entries. By default this is set to 1024 entries; the valid range is 16-50,000. For servers with large file systems and multiple trustees on different locations in the volume, you should consider increasing this amount to at least 20,000.
Note: NSS needs to increase this setting only if it has trustees assigned on different locations. For example, if you have 1000 trustees assigned for SYS:PUBLIC, NSS will only use 1 cache entry.
Conclusion
This AppNote has covered the analysis and tuning of NSS performance on a NetWare server.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.