Invalid Certificate errors and how to regenerate the TED certificates

  • 3371732
  • 27-Sep-2006
  • 30-Apr-2012

Environment

Novell ZENworks Server Management 7 - ZSM7
Novell ZENworks Server Management 7 - ZSM7 Tiered Electronic Distribution (TED)
Novell Tiered Electronic Distribution (TED)
Novell ZENworks 6.5 Server Management - ZSM65

Situation

TED.log contained the following error:
ERROR: zsmserver1.dhh.kf.gov refused connection
Error in the [ZenworksInstallPath]\zfs-startup.log:

2005.03.17 07:44:42 [TED] Adding key store entry for ServerName.DNS.Context
2005.03.17 07:44:43 [TED] NDS and Local certificates do not match! Resolving all certificates is required.
Error in iManager under "ZENworks Server Management" / "Tiered Distribution View" / Select a Distribution / Select a Channel / Highlight an individual subscriber and the following text would pop near the mouse:
Error reported in the [ZenworksInstallPath]\PDS\TED\Sub\TED.LOG file on Child and Parent subscribers, [ZenworksInstallPath]\PDS\TED\Dist\TED.LOG for Distributors:

ON SUBSCRIBER SERVER RECEIVING THE DISTRIBUTION:

2005.03.16 02:58:41 [TED:Work Order In(DistributorDNSName.DNS.Suffix)] Get verifying signature for: DistributorDNSName.DNS.Suffix
2005.03.16 02:58:41 [TED:Work Order In(DistributorDNSName.DNS.Suffix)] Sig2 failure
2005.03.16 02:58:41 [TED:Event Processing] Stopped distribution from distributor DistributorDNSName.DNS.Suffix due to signature error. Make sure certificates have been properly resolved.
2005.03.16 02:58:43 [TED:Event Processing] Distribution Reply for DistributionName.Distributions.Dept.Site.Company, 1110900375421, message = 5 was sent successfully to DistributorDNSName.DNS.Suffix

ON THE DISTRIBUTOR OR PARENT SUBSCRIBER SERVER SENDING THE DISTRIBUTION:

2005.03.15 20:59:04 [TED] TreeName/DistributionName.Distributions.Dept.Site.Company 1110416400387: Distribution is rescheduled at Run time: Tuesday, March 15, 2005 9:01:00 PM GMT
2005.03.15 20:59:05 [TED] Error sending distribution to SubscriberDNSName.DNS.Suffix. Exception: Connection refused: Connection refused
2005.03.15 20:59:05 [TED] *** Exception: java.net.SocketException: Connection refused: Connection refused
2005.03.15 20:59:05 [TED] java.net.SocketException: Connection refused: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java, Compiled Code)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java, Compiled Code)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java, Compiled Code)
at java.net.Socket.(Socket.java, Compiled Code)
at java.net.Socket.(Socket.java:127)
at com.novell.application.zenworks.ted.net.WorkOrderClient.processUsingSocket(WorkOrderClient.java:241)
at com.novell.application.zenworks.ted.net.WorkOrderClient.run(WorkOrderClient.java:129)


Resolution

The solution is to regenerate the TED certificate to this subscriber. This is done by completing the following steps on the subscriber reporting the error (not on the distributor or any other server):

1. Rename/delete the existing [ZenworksInstallPath]\PDS\TED\security\private\.keystore file.
2. Stop the ZFS TED services (issuing an "exit" command in the Zenworks console screen or issuing a "ZFSSTOP" command at the server console prompt).
3. Inside ConsoleOne, browse to the subscriber object that references this server.
4. Right-click on that subscriber object and select"Resolve Certificates".
5. When this has completed, reload the ZFD TED services on the subscriber again (issue a "ZFS" command from the server console prompt).

The subscriber will read in the certificate file and recreate the [ZenworksInstallPath]\PDS\TED\security\private\.keystore file. It should no longer generate these errors.

NOTE:In some cases, you will also need to delete the [ZenworksInstallPath]\PDS\TED\TED.CFG on the affected subscriber and ANY Parent Subscribers in the routing hierarchy before reloading the ZFS TED services. In these cases, you must also unload ZFS TED from the Parent Subscribers and then reload ZFS on each server in order, starting at the top Parent (just below the Distributor) and working down to the affected Child Subscriber. There is no risk in deleting the TED.CFG, it will automatically be recreated the next time the Distributor refreshes. Some known cases where you would need to delete the TED.CFG would include:
1. When the Distributor object has been deleted and recreated via a new install.
2. If a Parent Subscriber was installed initially as a Distributor (a TED Server can only be either a Distributor or a subscriber, a Distributor can NOT be a Parent or Child subscriber to another Distributor).
If you have a large number of subscribers to regenerate the certificate on, it is possible to automate much of this process using Onsite Admin Pro and the TOOLBOX NetWare Server Console utility. To start with, you must first get a valid copy of the Distributor certificate. This can be done by:

1. Right-clicking on one of the Subscribers in ConsoleOne and selecting "Resolve Certificates".
2. Go to the [ZenworksInstallPath\PDS\TED\security directory and copy the DistServerName.DNS.Suffix.cer file to another location.

NOTE:the DistServerName.DNS.Suffix.cer is not specific to a particular subscriber, only to the distributor, so you can copy this file from a single subscriber to all subscribers this Distributor services.

Once you have a valid certificate file, you can copy this along with a .NCF file to unload ZFS, delete the existing .keystore file, and then reload ZFS. The following file was used in conjunction with TOOLBOX and Onsite Admin Pro to accomplish this:

ZenInstallVol:zenworks\pds\bin\zfsstop.nlm
delay 30
toolbox -nl
delay 3
del ZenInstallVol:\zenworks\pds\ted\security\private\keysto~1 /q /y
del ZenInstallVol:\zenworks\pds\ted\ted.cfg /q /y
delay 3
ZenInstallVol:zenworks\zfs.ncf
unload toolbox

FYI, toolbox -nl loads toolbox without local authentication and the /q /y del options are to force the delete to be quiet and to not prompt y/n on deletion.

At this point it is possible to use Onsite Admin Pro automate this process for multiple servers. You must first do a"Copy Multiple Files" job to copy both the DistServerName.DNS.Suffix.cer and the NCF file out to each server. You can then do an "Execute NCF File" job to execute the NCF file you just copied out.

NOTE:Onsite Admin Pro processes each of these sequentially, so it WILL have to wait for the 36 seconds of delay in the above file before going on to the next one. Also, you will likely want to put the .NCF file in the same directory as the new .cer file so you can copy them out together...otherwise you will need to initiate 2 copy multiple files jobs.

Verify that you have your hosts and hostname on the server contain the correct DNS information, short and long names need to be correct in these files.


Additional Information

The cause of each of these is that the subscriber and the distributor/parent subscriber it is receiving the distribution from do not have the same TED certificate information for the channel. The TED certificate information is stored in the [ZenworksInstallPath]\PDS\TED\security\.keystore file and is built based on the contents of the "zentedCertificate" attribute on the Distributor object. This information contains the Distributor server name, DNS name, and IP address (among other things) and if any of these change, it invalidates the certificate.

Formerly known as TID# 10097372