Supporting Mixed Language Environments with Novell NetWare

(Last modified: 18Mar2005)

This document (10097059) is provided subject to the disclaimer at the end of this document.

goal

Supporting Mixed Language Environments with Novell NetWare

fact

Novell NetWare 6.5 Support Pack 2

Novell NetWare 6.5 Support Pack 3

1.0 Overview

In NetWare® deployments where the client and server are using different code pages, some international characters or entire file and directory names might be converted incorrectly and subsequently become unusable by the client, other clients, and backup software. This problem has existed for a long time, but has become more widely spread as support for multiple language packs with Microsoft* Windows and other desktop products has increased.

To resolve this, we have updated several components in NetWare 6.5 Support Pack 2 and the Novell® ClientTM to support current Unicode implementations. This makes it possible to access multiple language files from an NCPTM client without changing the code page on either the NCP client or the NetWare server. The serve r components include the following:

NCPIP.NLM: This was changed to support larger request packet sizes because some of the newer Case 89 NCP clients can have larger request packet sizes.

Cross Platform and NJCL Libraries: In order to support Java*- based applications, the NJCL and cross platform libraries were updated to support the Case 89 NCPs. Because Java uses Unicode names already, this change allows the name to stay in Unicode and UTF-8 all the way to the file system.

NCP.NLM: A number of new NCP clients (Case 89) were added. These allow file and subdirectory names to be passed in UTF-8 rather than ANSI. This way, it doesn't matter what the server's local code page is because names don't need to be remapped before being stored in the file system. This allows the file system to support filenames from different code pages that are mutually exclusive---names that can't be mapped into the server's local code page. Note that these new NCP clients are supported only for NSS volumes.

Novell Storage Services: NSS was changed to support the new Case 89 NCP clients.

LIBC.NLM and Unicode: In NetWare 6.5, the Unicode* tables used by LIBC are current, but those used by unicode.nlm are about out of date. With Support Pack 2, unicode.nlm is changed to use the tables and APIs provided by LIBC. This adds support for a large number of new characters.

Storage Management System: Backup and other tools that uses non-Unicode APIs in the file system have a big problem: because the non-Unicode APIs require that all names be mapped into the server's code page, they can't handle names that are NOT in the server's local code page. When they are scanning a directory and encounter such a name, they get an error and don't look any further---thus missing any files that come later in the directory. This problem has existed since NetWare allowed CIFS and AFP clients to send filenames to the server in Unicode; the NSS team developed the FIXNAMEs switch on NSSCPT console command to force all filenames into the server's local code page to help address the problem, but the real fix is to have backup and other tools use Unicode APIs when accessing the file system. Storage Management ServicesTM (SMS) is changing in a way that the third-party software that rides above it won't need to change; SMS is keeping the Unicode changes localized to itself. Note that third party tools such as virus scanners are likely to have similar Unicode problems.

In addition to the changes being made in NetWare 6.5 Support Pack 2, there are some changes being made to code that is common among previous versions of NetWare.

An update to the Novell Client for Windows is available so that the client supports the new Case 89 NCP clients. This lets it take file and directory names which come to it (in Unicode) and send them to the file server without first mapping them to the client's local code page. This lets the Windows client support different code pages and fonts; the local file system as well as the CIFS protocol already support this feature. Note that there is a switch to turn on Case 89 support and i.t is turned off by default to preserve current behavior for customers that aren't seeing problems. This is designed to be backward compatible in that it will use the older Case 87 NCPs when talking to servers that don't support Case 89.

Although this new functionality will prove to be beneficial in most cases, there are some issues relating to upgrades and mixing code pages and Unicode configurations that you need to consider when using multiple languages in your NetWare 6.5 Support Pack 2 environment.
.

fix

2.0 Installing the Unicode Updates on NetWare 6.5 Support Pack 2.  Complete the following tasks:

• 2.1 Update the Novell Client Support Pack with the Unicode Update
• 2.2 Install and Configure the Updated Novell Client2.3 Configure the Novell Client to Use UTF-8 Encoding
• 2.4 Install the NetWare 6.5 Support Pack
• 2.5 Use the NSSCPT Utility to Correct Invalid Filenames and Directories

WARNING:  To limit problems when implementing this Unicode update, you should fix any file and directory names on the server before allowing users to access the server using the NSSCPT utility.

For more information, see 4.2 Resolving Code Page Problems.

2.1 Update the Novell Client Support Pack with the Unicode Update

Taking advantage of the Unicode* updates requires Novell® ClientTM 4.9 Support Pack 2 and the Unicode client update. The files in the Unicode client update can be copied on top of either Novell Client 4.9 Support Pack 2 or the Client Overlay; thereby, you have to make only one update to the workstation.

2.1.1 Apply Unicode Client Update to the Novell Client

To apply the Unicode client update to either the Novell Client Support Pack or the Overlay:

Obtain a copy of the Unicode client update and either the Novell Client Support Pack or the Novell Client Support Pack Overlay.

(Optional) Extract the files.

Copy the directory structure of the Unicode update beginning with winnt\i386 to the \winnt\i386 directory of either the Novell Client Support Pack or the Novell Client Support Pack Overlay.

2.2 Install and Configure the Updated Novell Client

To simplify the configuration of the Updated Novell Client, you can create a platform-specific configuration file that will configure the client with the settings you want when it is installed. For more information, see "Creating or Updating the Platform-Specific Configuration Files" in the Novell Client for Windows Adminstration Guide. This reference also discusses deployment methods using login scripts. You can also use the Automatic Client Upgrade to update the client on your workstations. For more information, see Installing Clients from the Network in the Novell Client for Windows Installation and Administration Guide. To install the updated client manually, choose one of the following options.

2.2.1 Install Option 1 - Support Pack Install

To install the updated files on a workstation with the Novell Client v4.9 for Windows NT/2000/XP already installed, run the setupsp.exe program located in the winnt\i386\ directory.

2.2.2 Install Option 2 - Installing the Novell Client with the Support Pack Integrated

To install the entire Novell Client 4.9 Support Pack 1 for Windows NT/2000/XP using the Support Pack Overlay, run the setupnw.exe program located in the \winnt\i386 directory.

2.3 Configure the Novell Client to Use UTF-8 Encoding

If you did not choose to configure the Novell Client in 2.2 Install and Configure the Updated Novell Client, then you will need to configure the client manually. By default, the Use UTF-8 Encoding and NCPTM option is turned off. If you are not experiencing any problems, you do not need to change this setting. If the correct Unicode character is not being written to the server, you should turn this on.

Right-click the Red N on the sstatus bar , then click Novell Client Properties.

Click the Advanced Settings tab.

Select the Use UTF-8 Encoding and NCP's parameter, then change the setting to On.

Click OK.

2.4 Install the NetWare 6.5 Support Pack

The Support Pack must be installed using NWCONFIG. You can run NWCONFIG from the server console, the Console Screens option in NetWare® Remote Manager, or RCONSOLE. When you have access to the server console, complete the following steps.

At a workstation DOS prompt, explode the file update by entering utf8.exe.

At the server console prompt, enter NWCONFIG.

Select Product Options > Install a Product Not Listed.

Depending on where the update is located, complete one of the following:

From the local volume sys:, press F3 and then specify the path, including the volume name, to spack.ips. For example, sys:\nw65sp2.From a different server on the network, press F3 and then specify the full path, including the server name, to utf8.ips. For example, server1\sys:\nw65sp2. NOTE:  This is not supported in this beta.

Press Enter.

Follow the on-screen prompts.

2.5 Use the NSSCPT Utility to Correct Invalid Filenames and Directories

The NSSCPT utility can scan filenames and directories for invalid characters and attempt to correct them or change all the filenames from one code page to another.

For more information, read 4.0 Code Page Interoperability Issues before running the utility.

3.0 Upgrading or Migrating to UTF-8 Support.  Complete the following tasks:

• 3.1 Update the Novell Client Support Pack with the Unicode Update
• 3.2 Install and Configure the Updated Novell Client
• 3.3 Configure the Novell Client to Use UTF-8 Encoding
• 3.4 Upgrade or Migrate to NetWare 6.5 Support Pack 2
• 3.5 Use the NSSCPT Utility to Correct Invalid Filenames and Directories

WARNING:  To limit problems when implementing this Unicode update, you should fix any file and directory names on the server before allowing users to access the server using the nsscpt utility. For more information, see 4.2 Resolving Code Page Problems.

3.1 Update the Novell Client Support Pack with the Unicode Update

Taking advantage of the Unicode* updates requires Novell® ClientTM 4.9 Support Pack 2 and the Unicode client update. The files in the Unicode client update can be copied on top of either Novell Client 4.9 Support Pack 2 or the Client Overlay; thereby, you have to make only one update to the workstation.

3.1.1 Apply Unicode Client Update to the Novell Client

To apply the Unicode client update to either the Novell Client Support Pack or the Overlay:

Obtain a copy of the Unicode client update and either the Novell Client Support Pack or the Novell Client Support Pack Overlay.

(Optional) Extract the files.

Copy the directory structure of the Unicode update beginning with winnt\i386 to the \winnt\i386 directory of either the Novell Client Support Pack or the Novell Client Support Pack Overlay.

3.2 Install and Configure the Updated Novell Client

To simplify the configuration of the Updated Novell Client, you can create a platform-specific configuration file that will configure the client with the settings you want when it is installed. For more information, see "Creating or Updating the Platform-Specific Configuration Files" in the Novell Client for Windows Adminstration Guide. This reference also discusses deployment methods using login scripts. You can also use the Automatic Client Upgrade to update the client on your workstations. For more information see Installing Clients from the Network in the Novell Client for Windows Installation and Administration Guide. To install the updated client manually, choose one of the following options.

3.2.1 Install Option 1 - Support Pack Install

To install the updated files on a workstation with the Novell Client v4.9 for Windows NT/2000/XP already installed, run the setupsp.exe program located in the winnt\i386\ directory.

3.2.2 Install Option 2 - Installing the Novell Client with the Support Pack Integrated

To install the entire Novell Client 4.9 Support Pack 1 for Windows* NT*/2000/XP using the Support Pack Overlay, run the setupnw.exe program located in the \winnt\i386 directory.

3.3 Configure the Novell Client to Use UTF-8 Encoding

If you did not choose to configure the Novell Client in 2.2 Install and Configure the Updated Novell Client, then you will need to configure the client manually. By default, the Use UTF-8 Encoding and NCP option is turned off. If you are not experiencing any problems, you do not need to change this setting. If the correct Unicode character is not being written to the server, you should turn this on.

Right click the Red N on the status bar, then click Novell Client Properties.

Click the Advanced Settings tab.

Select the Use UTF-8 Encoding and NCP's parameter, and then change the setting to On.

Click OK.

3.4 Upgrade or Migrate to NetWare 6.5 Support Pack 2

When upgrading to NetWare 6.5, use the NetWare 6.5 Support Pack 2 Overlay to perform the upgrade. See "Upgrading to NetWare 6.5" in the NetWare 6.5 Overview and Installation Guide for more information.

When migrating to NetWare 6.5, create your new NetWare 6.5 Support Pack 2 server. Then run the migration utility to migrate data from NetWare 6.0 to NetWare 6.5. See the NetWare Migration Wizard 7.1 Administration Guide for more information.

NOTE:  The Migration Wizard and Server Consolidation do not support the Novell Client with the Unicode update. You must run the utilities from a workstation running Novell Client 4.9 Support Pack 2 or earlier.

3.5 Use the NSSCPT Utility to Correct Invalid Filenames and Directories

The NSSCPT utility can scan filenames and directories for invalid characters and attempt to correct them or change all the filenames from one code page to another. For more information, read 4.0 Code Page Interoperability Issues before running the utility.

4.0 Code Page Interoperability Issues

Some code page interoperability issues and problems have been identified when upgrading from earlier versions of NetWare® and when switching server code pages and Unicode configuration to provide support for languages other than English. The symptoms of these issues and problems include the following:

Files might appear to be renamed and might include invalid characters in the filenameMany files might be missing or not displayed because one or more filenames appear to have changed or filenames include invalid characters

No problems exist when upgrading NetWare for files created using English or US code pages. Also, no problems have been identified for files created using only clients where Unicode was enabled. Unicode support was enabled for CIFS clients starting in NetWare 6.0 Support Pack 2 and NetWare 6.5. As stated earlier, it is also supported for NCPTM clients in NetWare 6.5 Support Pack 2.

4.1 Problem Scenarios

The following cases describe some of the more common scenarios where code page interoperability problems can occur. The first scenario covers problems caused by upgrades. The other scenarios cover problems caused by mixing and switching code pages and Unicode configurations. There are other scenarios not documented here where problems can also occur. An example is backing up files on a server using one code page and restoring them to a server using a different code page or Unicode.

Upgrades from Code Pages to UnicodeIf you upgraded to NetWare 6.0 Support Pack 2 or a later or NetWare 6.5 Support Pack 1 or a later and you have files on servers that were created prior to the upgrade using a language (code page) other than English, those filenames could appear to be changed from characters that existed in previous code pages or previous code page versions.

A specific example of this is when upgrading a NetWare 5.1 Support Pack 7 server configured to use a Japanese code page to NetWare 6.5 Support Pack 2. In this example, you might have to rebuild the NSS file system. Prior to rebuilding the NSS file system, you should run the NSS /POOLVERIFY=poolname server console command to determine if the file system requires a rebuild. See the Novell Storage Services Administration Guide for NetWare 6.5 for more information on the NSS file system.

This problem could also occur when upgrading from servers using other non-English code pages.

Servers and Clients Use Different Code PagesThis scenario involves having NCP or CIFS clients that are configured to use a different code page than the server, even though the server's code page has never changed.

Running the NSSCPT utility with -CPSWITCH might help resolve some problems created by this scenario. See 4.2 Resolving Code Page Problems for information on the NSSCPT utility.

Server Code Page ChangesThis scenario involves having NCP or CIFS clients that are configured to use code pages and, at some point, the server's code page changes. The fact that the server and client code pages do not match afterward is incidental.

Running the NSSCPT utility with the -FIXNAMES switch might help resolve some problems created by this scenario. See 4.2 Resolving Code Page Problems for information on the NSSCPT utility.

Separate Clients Use Code Pages and UnicodeThis scenario involves having one or more clients configured to use Unicode* and one or more clients configured to use code pages. In this case, you could have files created by a client configured to use Unicode that are not accessible by clients configured to use code pages. In this scenario, problems will continue until you upgrade your clients to all use Unicode.

Running the NSSCPT utility with the -FIXNAMES switch might help resolve some problems created by this scenario. See 4.2 Resolving Code Page Problems for information on the NSSCPT utility.

Older Backups in Multibyte Character Set Backups made before the Unicode change are stored in Multibyte Character set (MBCS) format, not Unicode; therefore, a conversion has already taken place before it went to tape. It is possible that certain characters are not mapped correctly in the backup itself. You have to restore the tape and run NSSCPT -CPSWITCH to correct the filenames.

With the new SMS solution, the data is stored in both MBCS and Unicode in the backup. The restore will use the Unicode streams (on NSS) which will preserve the characters.

4.2 Resolving Code Page ProblemsSome of the symptoms and problems described above can be resolved using the NSSCPT server console utility. This utility is new for NetWare 6.5 Support Pack 2 and must be used with either of the two switches described below:

-FIXNAMES causes the specified directory and its subdirectories to be scanned for filenames that are not recognizable by the current code page. It then corrects filenames by substituting and possibly adding recognizable characters. Filenames might contain more characters after using -FIXNAMES. FIXNAMES works only for the current server codepage. That code-page can be specified with the -SRC parameter, but it isn't necessary.-CPSWITCH can convert all the filenames in the file system from one codepage to another. If filenames differ between the original and destination code pages, then -CPSWITCH will rename the files to be recognizable by the destination code page.

NOTE:  In earlier versions, FIXNAMES and CPSWITCH were separate NLM utilities. The functionality of those utilities have been incorporated into NSSCPT. You should run NSSCPT with the corresponding switch instead of the NLM TMutilities.

Other parameters that are used with the NSSCPT console command include

-D (optional) instructs the NSSCPT command to display the name of each directory it works on as it parses the file system. This permits some monitoring of progress. Note that directories with very large numbers of files in them might remain displayed on the screen for some time.-SRC is used with the -CPSWITCH switch and is necessary if you want to specify a source code page other than the one the server is using. Any code page can be specified.-DST is required for the -CPSWITCH switch and is used to specify a destination code page. Any code page can be used with the DST parameter.

To use NSSCPT with the -FIXNAMES switch, enter the following at the server console:

NSSCPT path -D -FIXNAMES

Replace path with the directory path to be scanned. You must use DOS namespace format when specifying the path. The directory you specify and all subdirectories of that directory are scanned. The root directory of a volume can be specified in order to scan an entire volume.

For example, if you wanted to scan the entire sys: volume of a server, you would enter the following at the server console:

NSSCPT SYS:\ -D -FIXNAMES

In this example, NSSCPT scans the entire sys: volume correcting names that cannot be represented in the current server codepage and displaying each directory name as it progresses. You could leave out the -D switch to not display directory names during the scan process.

To use NSSCPT with the -CPSWITCH, enter the following at the server console:

NSSCPT SYS:\ -D -CPSWITCH -SRC 437 -DST 932

In this example, NSSCPT scans the entire sys: volume regenerating filenames, assuming the current names were generated from code page 437 and should have been interpreted using code page 932. Directory names are shown during operation, but can be made to not display by leaving out the -D switch. You can also leave out the -SRC switch and the code page number specified with it to have the command use the server's code page.

IMPORTANT:  All switches used with the NSSCPT command must currently be typed in UPPERCASE.

When you run nsscpt a log file named nsscpt.log is automatically generated and saved in the sys:\etc directory. nsscpt.log contains a description of the operations performed when you ran NSSCPT. For any filenames that have changed, this includes the source filename, the mode used to rename the file, and the new name of the file. The possible file rename modes are

LONG: - renamed long name.UNIX: - Renamed Unix name.LONG BY ZID: - Renamed long name using file system entry number as source.DOS: - Renamed DOS name.LONG GENERATED:- Renamed in long namespace but without any attempt to use any part of the original name (name will be fixnames.nnn, where nnn is a three digit number).

You must use a UTF-8 compliant editor or word processor to view or edit the nsscpt.log file. Windows WordPad can be used if desired..

5.0 Known Issues

5.1 Updated ConsoleOne

ConsoleOne® 1.3.6c is included with the Support Pack and should be used in place of older versions of ConsoleOne. This version of ConsoleOne supports only Windows* NT*/2000/XP. You should run ConsoleOne from the server. If you want to run it locally from a workstation's hard drive, you must copy the entire directory and subdirectories of sys:public\mgmt\consoleone\1.2 to a local drive and run drive:\1.2\bin\consoleone.exe.

5.2 Server Consolidation and Migration Wizard Utilities

The Migration Wizard and Server Consolidation do not support the Novell® Client TMwith the Unicode* update. You must run the utilities from a workstation running Novell Client 4.9 Support Pack 2 or earlier.

5.3 Running Unicode Implementations on a Traditional File System

To store international characters, you should move to NSS.

5.4 Japanese Characters Do Not Display on Korean-enabled Workstation'

Windows cannot display all of the Japanese characters on a Korean-enabled workstation. This is a Windows limitation.

5.5 Cross-File System Restores

NSS volumes store pathnames natively in Unicode while traditional volumes use MBCS. Therefore, backing up from an NSS volume and restoring to a traditional volume involves conversion from Unicode to MBCS. Incorrect conversions are likely to occur for some characters that do not have an equivalent representation in MBCS.

5.6 Data Backup in Non-English Environments

If data backed up from a NSS volume in non-English environments needs to be restored to a traditional volume, a conversion from UTF-8 to MBCS is required. This might result in certain specific characters in pathnames being altered.

This restored data needs verification especially if the target server runs in a different locale. Backup engines use MBCS interfaces and might display data differently from how they are represented on NSS volumes because of a conversion from UTF-8 to MBCS in non-English locales.

However, it is expected that data would be backed up and restored correctly, because TSAFS stores data natively in UTF-8 streams. In such circumstances, we recommend that the backup and a trial restore be manually verified before the backup is certified to be correct and consistent. In environments with MAC files stored on the NetWare server, files that have the character 0xDA might cause the scan to abort because this character is converted to a "/" during the scan by backup engines. If the backup engine did the scan in all available name spaces, then the files are backed up using their UTF-8 names.

6.0 Updating Servers in a Cluster EnvironmentTo upgrade servers in a cluster:

Remove a server from the cluster.Upgrade the server to the latest Support Pack.Reinsert the server into the cluster.Repeat the above tasks with the next server.To avoid any potential problems, you should upgrade all servers as soon as possible.

document

Document Title:	Supporting Mixed Language Environments with Novell NetWare
Document ID:	10097059
Solution ID:	NOVL101479
Creation Date:	18Mar2005
Modified Date:	18Mar2005
Novell Product Class:	Netware

disclaimer

The Origin of this information may be internal or external to Novell. Novell makes all reasonable efforts to verify this information. However, the information provided in this document is for your information only. Novell makes no explicit or implied claims to the validity of this information.
Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.