GroupWise 5.5 Tuning, Optimization, and Sizing Recommendations
Articles and Tips: article
01 Oct 1999
Have you been wondering what the upper limits are for loading up GroupWise components on a parent system? This AppNote provides guidelines for the recommended number of domains per system, users per post office, and much more.
GroupWise performance depends a great deal on the workload you plan to place on it and the tasks you want it to perform. To make sure that GroupWise performs at peak efficiency given your particular implementation, you should follow the optimizing tips for these four areas of focus:
Document Management Service (DMS) sizing
This AppNote is adapted from "GW Sizing Recommendations (scalability)" (TID 2946201) which can be found at:
The term sizing refers to the upper limits of the size or number of specific GroupWise components that should exist within a parent component; for example, the number of users in a given post office.
This section lists system requirements and provides recommendations regarding GroupWise sizing. (You should consider the system requirements to be recommendations as well.) Use the recommendations with discretion based on your specific GroupWise implementation and the circumstances surrounding it.
This list of system requirements is given as a preface to the recommendations. You can use the information in this list as a guideline for planning the growth of an existing GroupWise system or the new installation of a GroupWise system.
GroupWise 5.5 with any patch level (such as 5.5.1 or 5.5.2)
Pentium Pro 200 or better
Client-to-Server LAN speed of 10 Mbps
Server-to-Server LAN speed of 100 Mbps
Used for e-mail only (users are not logging in to this server for applications, NDS authentication, or file/print services, and the server is not being used for NDS replication)
Set up your GroupWise system according to these recommendations.
One per server
One Message Transport Agent (MTA) per domain
Message Transport Agent (MTA):
One per domain
Run on same server as the domain
Run with IP links to other domains and post offices
Run on separate server from the post office (data store) or POA
Note: This recommendation is specific to those domains that act as high-volume mail traffic hubs. In most circumstances there is no problem with running a domain with an MTA on the same server as a post office with a post office agent (POA).
One per server
One POA per post office
Note: If you are running Document Management with a moderately to heavily used Library, Novell recommends that you have a second POA running on a separate server to perform the indexing tasks of the POA.
Post Office Agent (POA):
One per server
Run on same server on which the post office (data store) resides
TCP/IP (client/server) connection is used for client access
Based on the above-mentioned system requirements, the following sections detail the recommendations for sizing a GroupWise system. These recommendations have been divided up into three areas:
System Construction. The system construction is based upon recommendation for the number of domains and post offices, and their transport agents, in a GroupWise system. This has a direct affect on manageability and synchronization in the GroupWise system.
Domains per System
There is no limitation on the number of domains contained in a system. However, keep in mind that the more domains you have in a system, the more databases have to be synchronized. Novell recommends that the number of domains be low enough that databases can be synchronized without difficulty.
The number of domains contained in each system must be determined on an individual basis, and it will be different for each system you set up. Keeping in mind that more is not necessarily better, use these guidelines to help you determine if and when a domain is needed:
The primary domain should not have any post offices and should have a direct TCP/IP link to all secondary domains where possible. This ensures that administration traffic is isolated from message-delivery routes. It also ensures that administration changes are replicated as quickly as possible throughout the system. However, the primary domain should not be used as a routing domain.
Routing domains should be used at WAN-to-LAN connection points and service the remote domains that come through these points.
If a routing domain is used, it should exist on its own server and can service 60 or more links depending on LAN/WAN traffic and speed.
If the system contains multiple, high-traffic gateways, all gateways should be put on a separate "gateway" secondary domain. Additional domains for Internet traffic (primarily GroupWise Internet Agent and Web Access) can be placed at WAN/LAN hub locations to reduce the time and cost for message delivery and network bandwidth. Other high-traffic gateways they connect to can be placed in separate domains on separate boxes to boost the processing power of the system.
For Web Access and Remote Async connections, it is sometimes less expensive to have users access them locally instead of using a toll-free line. In this case, putting a domain in the remote location allows the gateway to run local to the users, eliminating or reducing the cost of long distance and toll-free numbers.
If communication lines are poor or lease lines are expensive, you may prefer to use an Async link. A secondary domain at the remote location would be required to allow for the async-to-async connection.
Two situations that may warrant another domain are (a) if you are using Dial-up Routing in a case where link scheduling could reduce costs, and (b) if you are using External Synchronization with other GroupWise 5.x systems to manage a high volume of administrative traffic.
Post Offices per Domain
As with the number of domains, the number of post offices per domain does not have a definite limit that applies in all situations. The number of post offices per domain is not as important as the number of messages the MTA for that domain has to route. For example, a domain with 50 post offices and 30 users per post office would be acceptable if the LAN/WAN speed were sufficient to route the volume of traffic generated by those users. As an example, one of the MTAs at Novell routes an average of 100,000 messages per day without problems. This MTA is on its own NetWare server and the server has no other processes running on it.
Consider these factors as well when deciding how many post offices to use in a single domain:
Processing power of the server the MTA is running on
Average number of administration messages per hour in the system
Busy search demand
Remote async requests (which put a greater strain on the system than direct users)
The number of gateways in the domain
The number of rules the users have active
The average user's activity on the GroupWise system
Number of POAs per Post Office
Novell recommends using one POA per post office. This POA should also be run on the same server that houses the data store. By running this configuration the POA is able to utilize built-in load balancing to determine which threads and/or requests get priority over others. When a second POA is loaded on the same server, one process cannot control the threads of the other process, and as a result the two processes compete for processing time.
If a second POA is needed for QuickFinder indexing or for faster processing, Novell recommends running the second POA on a different server. This server can be connected via the LAN (which should be on the same network segment as the first server), or a second network board can be installed with a crossover cable to either a NetWare server or an NT server where the second POA will run.
User Count. The user count is specific to the number of users in a given post office. The following subsections give criteria and recommendations for direct access (Client/Server or UNC) and for WebAccess-only users.
Number of Users per Post Office
The term users has various meanings in the groupware industry. Some vendors use this term to specify the total number of users regardless of their activity. Novell defines users as clients who are actively sending, reading, or otherwise using their mailboxes. This document refers to these users as "active users." You need to determine what maximum percentage of total users will be using mail services actively at any given time.
The term post office also has different meanings among different groupware packages. Historically, GroupWise has defined it to be "a grouping of users in a given data store location." As the groupware industry grows, it is becoming more common for post office to mean "a grouping of users in a given data store on a dedicated server" or, in other words, mailboxes per server. Although Novell doesn't require that this be the case, it is a preferred method of implementation.
Novell recommends that the number of active users per post office not exceed the range of 500 to 700. Assuming that on average only 60 to 70% of users will be logged in and using mail at any given time, GroupWise can easily support a post office of 1000 or more total users.
Note: These numbers are based on "active users" concurrently using GroupWise with a direct TCP/IP connection. The total number of users is only limited to how many of those users will be active at any given time. Novell foresees this number changing as technologies advance. In addition, the performance of the client, server, and agent can vary dramatically as settings for both the POA and the server are adjusted.
Of course, every system is different, and there are several factors to consider when deciding on an acceptable number for your situation:
LAN/WAN speed and topology. The slower the network and the more hops a packet has to make, the slower the performance will be. This gets worse as both the GroupWise client and GroupWise POA start flooding the network with TCP/IP packets.
Clean-up policies. Without standard clean-up policies implemented, there is no way to control the size of the GroupWise databases or the user's mailbox. The larger the databases get, the longer maintenance routines will take. In addition, as the number of messages the POA must query for Finds and/or indexing increases, the POA slows down. The larger the user's mailbox (database) gets, the longer it takes the client to display the items, resulting in a dissatisfied user.
The amount and size of attachments the average user is expected to have. In Novell Technical Services, the Post Office has less than 300 users due to the size of attachments each user has, these include large databases and coredumps mailed from customers. This example resulted in needing to reduce the number of users on a Post Office for disk space reasons.
Backup and restore. This can be an issue whether the problem is too many users or too much mail. This issue requires decisions to be made that are specific to each situation. Solutions will include limiting the number of users per post office, adding more disk space, implementing an e-mail policy, restricting the size of attachments, limiting the number of messages, and billing the user when limits are exceeded.
Number of Web Access Users per Post Office
The bottom-line recommendation for this is the same as above: 500 to 700 active users. The basic service to the client is handled through the POA the same as for a direct TCP/IP client (given that the Web Access Gateway is set up to do Client/Server). However, this recommendation can vary based upon several aspects: Web performance is acceptably slower (people expect the Internet to be slow), typically there are fewer active users at any given time, and due to slower speeds the average user will avoid sending large attachments.
With these aspects in mind, the number of Web Access users per post office can be greater than direct connect users. In the case of ISPs, the number can be even greater due to the limitation of modems that the ISP supports at a given time. For example, if an ISP has 500 modems, then 500 active connections are all that can be supported. Therefore, the total number of users per post office is unlimited.
Note: This section is added to provide recommendations for those installations who will provide Web-based access as their primary client access. Web Access is a gateway and doesn't have users specifically assigned to it. However, the gateway does provide Web-based access to a GroupWise mailbox which has to be defined in a specific post office within the GroupWise system.
Agent Memory. Below are the agent memory requirements. These requirements show the minimum memory usage the agents require. Having more than the minimum is suggested to ensure reasonable response times, especially during peak access times.
MTA Memory Requirements
These memory requirements are in addition to memory needed for the operating system:
Minimum required (small system with 3 to 5 direct links) 10 MB
For larger systems (more than 3 to 5 direct links), add the following per link to the 10 MB minimum required:
Light to moderate traffic (less than 50,000 messages per day) + .2 MB
Moderate to heavy traffic (greater than 50,000 messages per day) + .5 MB
Note: These numbers were gathered and compared against live systems, not lab environments. Additionally, they are the based upon averages. Novell recommends that these minimums be exceeded and not just met.
POA Memory Requirements
The memory requirements for a POA (Post Office Agent) will vary based upon the number of "active" users. The total number of users in the post office is irrelevant. Again, these numbers are based upon the above-mentioned assumptions.
Number of "Active Users"
Additional Memory Required
Document Management Service (DMS) Sizing
Document management is relatively new as a GroupWise component. GroupWise 5.5 was the first implementation with a valuable feature set. Given this information, sizing recommendations are based upon lab testing and few large customers.
Much like in the GroupWise Sizing section, the assumptions are listed first to establish a baseline. Then the recommendations are listed.
This is not a must-have list of software and hardware. However, these are the general hardware guidelines that seem to be consistent with successful implementation of DMS.
Any patch level of 5.5
Any version of GroupWise above 5.5
Pentium Pro 200 or better
250 MB memory or more
One post office per server
A second POA for indexing
Note: The second POA is not always necessary. However, in heavily used libraries it is recommended that a second POA on a separate server be set up to handle the additional load of indexing.
These recommendations are made with the above-mentioned assumptions in mind. Changes in the assumption may or may not have an impact on the DMS depending on how much the variance.
Number of Libraries per Post Office. The maximum number of libraries that can exist in a given post office is 256. This is a large number of libraries and probably should not be matched. A more realistic recommendation is to plan carefully according to the predicted number of total documents, how the documents are going to be divided, and who will need access.
Number of Libraries per System. The only limit here is the number of post office multiplied by 256 libraries (this is the maximum GroupWise will allow). Although this number is probably going to be very high, it is not the maximum that should be approached. The ideal is a carefully planned implementation that will satisfy the needs of those using it.
Number of Documents per Library. The library itself can hold up to one billion documents, although that is a logical and not a realistic limit. A more reasonable approach is to say that at any one time, one library should not hold more than one million documents. To leave room for system growth, plan for each library to have approximately 500,000 documents in it.
It is also important to realize that this is the number of live documents in the library and not the maximum number of documents in the system. These numbers do not take into consideration the size of the document store, just the number of documents. The DMS databases will grow based on the number rather than the size of documents.
Note: The recommended number of live documents is low because if the server holding these documents fails (due to a problem outside the control of GroupWise), there is a limit to the number of documents that are affected by the failure.
Optimizing the performance of the GroupWise Agents (MTA and POA) requires adjusting two separate groups of settings. The first is the GroupWise agent settings and the second is the optimization of the server on which the agent is running. The agent settings are covered in this section, while the server settings are covered in the "Server Configuration" section below.
Optimizing the MTA
When optimizing the MTA, the most important consideration deals with link configuration. In a large GroupWise system, improper and unplanned MTA links could result in severe LAN/WAN traffic problems. Below are several important objectives that have been identified relating to link configuration issues in the areas of the network and GroupWise administration. A recommended configuration is then suggested to meet these objectives.
In addition to the link changes, MTA performance relies on the performance of the server on which it is running. Fine tuning the server and configuring the MTA links as described below will accomplish the necessary MTA tuning.
Network-Related Objectives. Here are two objectives to consider when optimizing the network for an MTA:
Follow the WAN topology. It is desirable to configure links so that they follow the design of the WAN. By configuring links to hop across a minimal number of WAN segments, connectivity issues are minimized. Additionally, it is necessary to avoid bouncing traffic twice across the same segment and it is desirable to use the shortest routes available.
Control the number of GroupWise TCP connections and bandwidth usage across WAN links. Controlling the number of TCP connections will ensure faster file transmission and in general prevent GroupWise from hogging bandwidth.
Administration-Related Objectives. Here are some administrative objectives to consider for MTAs:
Minimize the number of links while maximizing number of users connected via three hops or less. It is desirable to keep the number of links to a manageable number, while still providing the fewest number of hops possible, preferably two or three hops or less between domains.
Isolate administration and replication traffic and minimize the latency of replication. It is desirable to configure the links so that administration changes do not interfere with message delivery and also so that administrative changes are replicated quickly throughout the entire system.
Minimize delivery time. It is desirable to minimize the time that messages take to be delivered.
Maximize response times. It is desirable to maximize the response times for remote download requests and busy searches across the network.
Recommended Configuration. To achieve the listed objectives, the following principles should be used when configuring links:
The primary domain is connected directly to all domains on WAN, including direct connection to domains servicing Async gateways. This ensures that administration traffic is isolated from message delivery routes. It also ensures that administration changes are replicated as quickly as possible throughout the system. The primary domain is not permitted to service any post offices or gateways. This serves the same purpose.
Each WAN site has a direct link to its closest regional hub domain, usually just one hop across a slow WAN link. This permits close control of bandwidth usage over slow WAN links and it also vastly reduces the number of total links requiring configuration in the system. All other domains in the system (except the primary domain) are linked indirectly to each WAN site via this closest regional hub domain, inbound as well as outbound.
The principal domain at each major site and regional hub domains and service domains such as Internet gateway domains are combined in a backbone group and have direct links to each other in mesh configuration across high speed WAN links. This balances maximizing the number of users connected in three hops or less and minimizes the number of links required for the entire system. Approximately 80% of the total users will be located on 20% of the domains in the system. This helps to minimize delivery time and to maximize response times.
All domains physically connected to the same LAN are linked in a mesh configuration. This reduces the workload on the hub domains and improves performance without impacting available bandwidth or significantly increasing the number of links in the system.
Optimizing the POA
The POA object has many settings that can be configured. Several of these can increase or decrease the performance of the POA. These settings are available so that administrators can change them based on their needs for optimal performance and stability.
Enable Automatic Database Recovery = On (default = On). This setting enables the POA to detect database problems and fix them in most cases. In the long run, this will improve performance because it prevents problems from getting really bad before they are detected and corrected. But, it does take resources to run the GWCHECK if and when problems are encountered. However, Novell recommends this be left On. The trade-off in stability is worth the possible loss in performance.
Enable Caching = On (default = On). Novell recommends that this setting be set to On to improve the performance of the POA. This allows the POA, at the software level, to handle caching of the databases it is working with. However if there are problems with database corruption, it would be a good idea to turn this setting off until the source of the problem is located and fixed.
Enable SNMP = Off (default = On). Turn this off only if SNMP is not being used to manage the POA. This feature requires quite a bit of I/O and processor traffic. If SNMP is not being used (through ManageWise or some other SNMP manager), turning this unneeded feature off could help performance.
TCP Handler Threads = 20 - 30 users per thread (default = 8). These are the TCP/IP threads that will handle the Client/Server requests. NCS has found that the appropriate setting depends on how active the users are. If the users are less active, only an average of a 5 to 10 items per day both sent and received, then 30 users per thread is sufficient. However, if the users are more active (more than 30 items are sent and received each day, in addition to finds and busy searches), then 20 users per thread is recommended.
It is important to adjust this number for each situation because each thread allocated take memory and resources. On the other hand, if there are not enough threads allocated, there will be pending requests which means slower performance for the end-users.
QuickFinder Indexing = On (default = on). The QuickFinder indices are created for both libraries and users databases. They are used to speed up the query results for a find as well as a document search. It is important to note that the QuickFinder takes up a lot of resources on the server and often will cause the utilization to peak until the indexing is complete. If a find (query) is made for a specific mailbox item and that item has not been indexed, it will be found but it will take longer. If a query for a document is made and that document has not been indexed, the document will not be found.
QuickFinder Interval = ? (default = 4 hrs). With GroupWise 5.5, an additional setting was added to allow this indexing to start on a specific offset from midnight. This gives the administrator more control to run the QuickFinder during a server's slowest time. There is no recommended number for this setting. It will vary based upon whether or not there is a library, how often users employ the find feature, and whether or not there is a POA created to handle QuickFinder indexing.
Max App Connections = 4 per user (default = 2048). Application connections are virtual connections. They are the workhorse for the IP traffic between client and POA. As new communication between client and POA is required, a new application connection will be spawned. After 5 seconds of no use, the connection will time out and terminate. An average user will use approximately 4 connections per session at any one time. Each connection takes up about 8 KB of memory. When application connections hit the maximum, the oldest connection is terminated to handle the request of the new ones. If that oldest connection was still in use, the client will request a new one, thus causing a vicious circle. If users complain of speed, this setting may be too low.
Max Physical Connections = 1 per user (default 512). A physical connection must be created in order to generate application connections. A user can have multiple physical connections. In general, one physical connection per user is sufficient because not all users are going to active at one time. If GroupWise hits the maximum physical connections, users will receive an error that they cannot connect to GroupWise at that time. Increasing the maximum for physical as well as application connections does not pre-allocate memory. The settings are there to protect GroupWise from accessing too many resources on the server.
CPU Utilization = 85% (default = 85%)
Delay Time = 100 millisec (default = 100 millisec). For these two related settings, the defaults are the recommended values. These settings are designed to keep peak performance on the server. If the processing load on the server is too heavy, the POA will start to delay the launch of new threads for 100 milliseconds. This allows the server to continue processing the current requests and yet not ignore other responsibilities. The POA is also designed to load balance its requests as the threshold is approached: Client/ Server threads become the highest priority. The POA will start to terminate other threads, such as GWCheck and QuickFinder, to free up resources for the client requests.
The optimization of the server itself is probably the most important focus for the GroupWise agents to run at their best. This section identifies specific settings that Novell has found to enhance and often fix issues dealing with GroupWise agents. These settings will help fine tune the server for optimal GroupWise agent performance.
Directory Cache Buffer Nonreferenced Delay = 30 sec (default = 5.5 sec). This setting will decrease processor overhead and I/O traffic. It determines how often the Directory Cache buffers are refreshed. Every refresh requires a new disk read and write to memory. The reason for this feature is in the case that a file did not get added to the buffer for whatever reason, it will be captured with the refresh routine.
By increasing the value to 30 seconds, you decrease how often the refresh takes place. By using this setting, there is little danger in losing data. As new files are added to a directory structure, it dynamically updates the buffer.
Minimum Directory Cache Buffers = 2-3 per connection (default = 20). By increasing this value, the buffer is already established and no additional resources are required to allocate more buffer space on the fly. This can eliminate processor and I/O bottlenecks.
Maximum Directory Cache Buffers = 4000 (default = 500). This setting protects the system from using too much memory for Directory Cache buffers.
Read Ahead Enabled (default = On). The Read Ahead feature significantly increases performance on NetWare servers. Read Ahead predicts what files are required next and loads them into memory for ready access.
Read Ahead LRU Sitting Time Threshold = 60 sec (default = 10 sec). This feature is related to the Read Ahead mechanism. The Read Ahead LRU Sitting Time Threshold says that if the LRU (Least Recently Used) buffer is below the specified time, do not use the Read Ahead feature.
LRU is an algorithm that is used for memory block / page replacement. An "LRU list" identifies the least recently used cache blocks (blocks that have been in cache the longest time without being accessed) and flags those for use first. It makes for a more efficient caching implementation.
The reason for the above setting is that if there is not enough memory to access data from available cache, Read Ahead will take up memory and processor time without increasing performance. If Read Ahead is not helpful, it makes sense to not use the resources.
This setting can be configured up to 1 hour. In general terms, if the LRU is 20 minutes or better, the system probably has sufficient memory. This setting could be effective anywhere from a minute to possibly 5 minutes.
Be aware that this disables Read Ahead which usually is not a recommended thing to do. If you find yourself using this option a lot, it is probably time to add more memory to the server.
Maximum File Locks = 20,000 (default = 10,000). Although GroupWise does much more record locking than file locking, it is wise to allocate additional file locks if there are a lot of users on the system. Be aware that this does require memory and therefore should not be overused.
Maximum Record Locks = 100,000 (default = 20,000). GroupWise performs a lot of record locks. If there are a lot of users on the system, it is wise to allocate enough record locks. Again, this requires additional memory and should not be overused.
Minimum Service Process = 100 (default = 10). Service processes are dynamic. By pre-allocating them, less overhead is required than allocating them on the fly. As long as there is sufficient memory, this number can be increased. A good rule of thumb is to monitor the server during peak times and set the Minimum Service Processes to whatever the current number of service processes is during those peak times.
Maximum Service Process = 1000 (default = 40). This also takes up resources. Monitor this setting using MONITOR.NLM. If the current number of processes begins to approach the maximum, increase the maximum service processes.
New Service Process Wait Time = .3 sec (default = 2.2 sec). This setting can drastically increase performance because when a Service Process is required, a new one can be created quickly. With the default setting of 2.2 seconds, the theory is that if the system waits long enough, a process will become free. If there is sufficient memory, there is no harm in creating a process instantaneous to the initial request.
Minimum Packet Receive Buffers = 2 -3 per user* (default = 50). Any request that is processed uses a Packet Receive Buffer. This includes all NCP requests, SAPs, RIPs, TCP packets, and so on. If the server is bombarded with requests and there are not enough packet receive buffers, the system will get bottlenecked and will start dropping requests. The result is loss of connection to users, loss of server to server connections, slowness, and so on.
Monitor the current number of packet receive buffers during peak times and make sure that the minimum is set to that number so that there are enough packet receive buffers at all times. Remember that this takes up memory, so you may need to add memory to the server.
Note: For a server hosting the Web Access gateway, this setting should be set to 2000.
Maximum Packet Receive Buffers = 4000 (default = 100). This protects the server against too many packet receive buffers allocating too much memory to processes.
New Packet Receive Buffer Wait Time = .1 Sec (default = 1 sec). If the server has sufficient memory, this setting can significantly increase productivity. As with service processes, the server will immediately spawn a new buffer without waiting to see if one becomes available first.
TCP Defend Land Attack = Off (default = On). This feature protects the TCP/IP stack against "land attacks." A land attack occurs when packets are sent to the server with the same source and destination. These packets get into a loop and can bring the server down. If the server in question has no access to the outside world, the chance of a packet doing this is extremely minimal. By turning this unneeded feature off, overhead is reduced and IP packets can be processed faster.
Other Server Performance Suggestions
Another way to improve performance on the server is to flag the WPCSIN and WPCSOUT directories (and their subdirectories) for Immediate Purge. These directories exist below each of the Post Offices, Domains, and Gateways in the GroupWise System. In addition, each MTA will have a MSLOCAL directory structure that should also be flagged as Immediate Purge.
Many of the recommended settings above will use additional server memory. For example, each additional buffer allocated takes up about 4 KB, and each service process requires about 16 KB. The best way to ensure that your server has sufficient memory is to watch the LRU count and the Available Cache Buffers. If the LRU count drops below 20 minutes and Cache Buffers fall below 40%, more memory is probably required.
This AppNote has covered various aspects of GroupWise tuning, optimizing, and sizing. For more information, see these related documents found on Novell's support site at http://support.novell.com.
"Performance, Tuning and Optimization" (TID 2943356) covers various areas of optimizing a NetWare server's overall performance. It presents several proactive and preventive maintenance measures to reduce the possibility of server downtime.
"POA Log File Definitions for TCP/IP" (TID 2939577) defines the statistics reported in the POA log file and provides ideas on how to interpret the log file information to optimize POA settings.
"Troubleshooting High Utilization for NW 4" (TID 2941108) contains documentation designed to help troubleshoot high utilization issues for NetWare 4.1x servers.
* Originally published in Novell AppNotes
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.