Novell is now a part of Micro Focus

Supporting PCI Hot Plug Technology in the Novell Architecture

Articles and Tips: article

01 Mar 1998


Imagine being able to swap a network adapter or disk controller without having to shut down your server. PCI Hot Plug technology makes it possible!

Introduction

Editor's Note: This AppNote is taken from a Technology Brief entitled "PCI Hot Plug Technology with Novell Architecture" originally published and copyrighted by Compaq, and revised and republished electronically with permission by Novell at http://www.novell.com/intranetware/hotplugpci.html. For more information about PCI Hot Plug technology in general, visit Compaq's web site at http://www.compaq.com.

Novell Inc. is a leader in the development of network operating systems. intraNetWare, developed by Novell, is the most widely used network operating system in the world. Because of Novell's expertise in this area, their network operating system technology is perfectly suited for integration with Compaq's PCI Hot Plug technology. Thus it is no surprise that intraNetWare is the first operating system to support PCI Hot Plug technology.

Compaq is a leader in the development and manufacture of computers, servers and options. As a world class leader, Compaq has a proven record of providing solutions that increase the availability or uptime of computer systems.

Because of the unique capabilities of Compaq and Novell, it is only appropriate for these companies to work closely to develop the software needed to implement the PCI Hot Plug technology throughout the business community.

PCI Hot Plug Technology

Novell is an acknowledged leader in providing state of art networking software. To cater to the customer requirement of higher system availability, Novell has partnered with a leading server vendor, Compaq, to provide support for PCI Hot Plug technology. PCI Hot Plug technology will add to the high-availability offerings presently available from Novell and Compaq. The capability to replace a PCI controller while a system is running is one aspect of high availability absent from current Novell solutions. Hot plug technology fulfills this requirement.

With current technology, a system must be powered down in order to replace PCI adapters in an industry standard server. For servers in enterprise environments, replacing a PCI network, storage, or other adapter while a system is operating offers substantial benefits. PCI Hot Plug technology allows for replacement of a failed adapter while the remaining adapters provide end-users with uninterrupted service. To provide even greater benefit, PCI Hot Plug technology can be combined with existing Compaq Redundant Network Interface Controller technology or upcoming array controller duplexing technology. This allows a redundant standby controller to respond to all requests while an administrator powers down the failed controller using PCI Hot Plug technology.

To provide this technology as an open industry standard, Novell and Compaq are working together in the following areas:

  • Developing standardized PCI Hot Plug technology by initiating and chairing the PCI Hot Plug Workgroup.

  • Working with leading adapter vendors to ensure broad acceptance and rapid implementation of the standard.

  • Working together to accelerate delivery of the PCI Hot Plug technology and to incorporate these PCI Hot Plug capabilities into Novell intraNetWare.

  • The Compaq Technology Brief entitled PCI Hot-Plug Technology (Doc. No. 398A/1196) outlines Compaq's overall strategy and direction regarding PCI Hot Plug technology. This document provides system hardware and Novell architecture overview information for the planned PCI Hot Plug implementations.

PCI Hot Plug Implementation

To deliver PCI Hot Plug technology to the market, Novell is working with key hardware and software partners, and industry-wide standards groups to implement the technology as an open rather than a proprietary standard. This effort will allow customers to move forward in implementing this technology with greater flexibility.

PCI Hot Plug Workgroup

A fundamental key to PCI Hot Plug development as an open standard is the PCI Special Interest Group (SIG). While Compaq initiated the PCI Hot Plug Workgroup within the SIG to establish a recognized industry standard for PCI Hot Plug technology, Novell is the first OS vendor to provide support for this industry standard. The PCI Hot Plug Workgroup includes other industry leaders such as Adaptec, Cirrus Logic, Digital Equipment Corporation, Hewlett-Packard, IBM, Intel, Microsoft, Pro-Log, SCO, and Texas Instruments.

The Novell Compaq Team

Novell is an expert in the development of network operating systems and standardized driver architectures. Compaq has expertise in the development of systems hardware, systems configuration software, device drivers, and the design of high-availability solutions. In December 1995, engineers from Novell and Compaq formed a team to design a standard software architecture to provide PCI Hot Plug support for Novell's intraNetWare and NetWare operating system.

Through Novell and Compaq's unique capabilities, benchmarking, and team efforts, the business community can reap great benefits from the quality products produced by this effort.

System Overview

PCI Hot Plug functionality requires both hardware and software components. A fully capable hot plug system includes the hot plug system hardware, operating system (OS) support, and device drivers as shown in Figure 1.

Figure 1: PCI Hot Plug technology.

The PCI Hot Plug hardware performs two main functions:

  1. Powers down a single adapter slot, allowing replacement of the adapter.

  2. Protects the system from the electrical effects of the replacement operation.

This hot plug solution differs from other possible solutions where either several slots are powered down or an entire PCI bus is powered down, thus requiring a system level shut down.

Novell's advanced OS and driver design allow a system administrator to replace and start up a new adapter while the system is running. The system hot plug hardware enables the use of industry-standard PCI adapters. However, adapter device drivers will need modification to utilize hot plug capabilities.

PCI Hot Plug technology will support three major processes: hot replacement, hot upgrade, and hot expansion.

  • Hot replacement is the process of removing a failed or suspect PCI adapter and inserting an identical adapter without any power disruption to the server.

  • Hot upgrade is the process of replacing an existing adapter with a revised card while the server is running.

  • Hot expansion is the process of adding a new adapter into an open slot while the server is operating.

The system hardware remains the same for all three hot plug capabilities. For Novell's intraNetWare, the first PCI Hot Plug feature supported is hot replacement. This feature was made available on intraNetWare in July 1997. Additional software support for features like hot upgrade and hot expansion on intraNetWare/NetWare 4.11 is now available with Novell's Support Pack 4.0 that was released on September 26, 1997.

System Hardware

Novell's OS and Compaq's PCI Hot Plug technology support hot plug functions without disrupting service to any other adapters in the system. The hot plug electronics, designed by Compaq and supported by Novell, consists of two separate electrical elements: the hot plug controller and the slot-specific power control.

Hot Plug Controller and Slot-Specific Power Control. Compaq designed its PCI Hot Plug Applications Specific Integrated Circuit controller to manage the following components:

  1. PCI Bus. The controller communicates with isolation devices on the PCI bus to electrically isolate a single PCI slot from the rest of the system. The isolation of this slot ensures replacement of an adapter without any interruption to the rest of the server.

  2. Power. The controller receives a command from the OS to power up or power down a single PCI slot. To perform this function, the controller uses the slot-specific power control. The slot-specific power control electronics allows proper power sequencing on the PCI bus and guarantees safe control of the power to the individual PCI adapter cards.

  3. Slot LED Indicators. The controller also governs the slot LEDs. A green LED indicates power; an amber LED indicates attention is required in one of the system's hot plug slots.

System Chassis Design. In addition to the electrical components required for hot plug, the overall system design requires mechanical adaptations for safety and ease of adapter installation and removal. Compaq's system hardware design protects both the user and adapter cards from the risks of electrical shorting.

Hardware and Software Control

In a typical scenario in which the user initiates an adapter replacement, the following steps occur to remove a single adapter while the system is running:

  1. The user notifies the OS of the intent to remove an adapter via the PCI Hot Plug user interface.

  2. After the OS stops all activity on the adapter, it notifies the hot plug controller to isolate the slot from the PCI bus and turns off the slot power.

  3. Once the slot power is off, the slot's green power LED will automatically turn off indicating it is safe to remove the adapter.

A power down sequence could also be initiated if an error occurs in the system. The Novell software and the Compaq hardware work together to monitor the system and detect errors on the hot plug adapter card. If an error occurs, the software performs the following functions:

  1. Notifies the hot plug controller to illuminate the amber attention LED on the suspect slot.

  2. Alerts the user of the error through the PCI Hot Plug user interface.

  3. Notifies the slot-specific power control to power down the slot. The user can initiate this function or it can occur automatically.

Adapters and Device Drivers

The vast majority of industry-standard PCI adapters can be used without any modifications. However, the device driver requires the addition of new features in order for the adapter to utilize hot plug capabilities.

Leading independent hardware suppliers committed to modifying their device drivers to be hot plug aware include Adaptec, BusLogic, Mylex, and SysKonnect. The standardization work with the OS suppliers and the PCI Hot Plug Workgroup is expected to encourage the proliferation of hot plug aware drivers for third-party devices. In addition, Compaq is developing hot plug capable drivers for its own leading PCI server adapters such as Compaq NetFlex controllers, Compaq Netelligent controllers, the SMART-2 Array Controller, and Compaq SCSI adapters.

PCI Hot Plug Compatibility

Concerns often arise regarding compatibility between new and existing technologies. PCI Hot Plug technology addresses these concerns by using standard PCI adapters.

In general, to achieve PCI Hot Plug functionality, a system requires hot plug hardware, an OS and device driver support. However, a system can also achieve PCI Hot Plug functionality by using new hot plug hardware, an OS, and a mix of its current and new hot plug device drivers.

This configuration brings hot plug functionality to the system's hot plug device drivers. Current device drivers will continue to operate as originally intended without hot plug functionality. This technology allows customers to upgrade their systems as needed and alleviates the need to replace their entire system.

Novell intraNetWare System Architecture

The PCI Hot Plug architecture leverages the inherit modularity of intraNetWare to minimize the changes required of third party adapter card software. The system relies on a new central component, the Novell Event Bus, which facilitates communications between the different software modules. The Event Bus is first implemented as a NetWare Loadable Module (NLM), allowing implementation of PCI Hot Plug on existing versions of intraNetWare.

Figure 2 illustrates the key components of the PCI Hot Plug architecture.

These components include:

  • Novell Event Bus (NEB)

  • OEM Specific System Bus Driver (SBD)

  • Novell Configuration Manager (NCM)

  • Novell Configuration Manager Console (NCMCON)

  • Device Drivers - ODI-compliant network adapters - NWPA-compliant storage adapters - Other Adapters

  • Installation Tools

Figure 2: Novell intraNetWare Hot Plug architecture.

The following sections describe the key components of the Novell intraNetWare system hot plug architecture in greater detail.

Novell Event Bus

The most important component of the PCI Hot Plug software architecture is the Novell Event Bus. This software bus facilitates communication between the administrator, the software modules, management agents, and device drivers, providing the foundation for the software architecture. The Event Bus will be implemented first as an NLM, allowing implementation of PCI Hot Plug on existing versions of intraNetWare. Future versions will integrate the Event Bus into the intraNetWare OS.

OEM Specific System Bus Driver (SBD)

Compaq, as well as other system hardware manufacturers, will provide a system bus driver specific to their PCI Hot Plug hardware implementation. The intraNetWare architecture allows multiple system bus driver modules to exist simultaneously. Again, this allows end users considerable flexibility in implementing hot plug capability in their environment.

The system bus driver abstracts the OEM's platform-specific implementation of PCI Hot Plug. This driver controls the system hardware necessary to perform hot plug functions and is responsible for the following:

  • Enabling or removing power from a hot plug slot upon request.

  • Configuring hardware and assigning resources like memory, PCI, and interrupts of PCI adapters in hot plug slots.

Novell Configuration Manager

For PCI Hot Plug servers, the Novell Configuration Manager manages the hot plug system and ensures proper system behavior. Its primary function is to sequence hot plug operations to ensure stability of the system. To achieve this function, the Configuration Manager properly shuts down the adapter's device driver before removing power from the slot. Another main role of the Configuration Manager is to map hardware resources to device driver software. (Future releases of intraNetWare will have an enhanced Configuration Manager to encompass other technologies in addition to PCI Hot Plug.)

Novell Configuration Manager Console

The Novell Configuration Manager Console is necessary to perform hot plug functions in intraNetWare systems. This NLM provides the interface for controlling all hot plug operations. In addition, it displays slot configuration details, including adapter driver information and status. (Novell will incorporate the Novell Configuration Manager Console as a standard component in future releases of NetWare.)

The Configuration Manager Console consists of two screens: the Main Menu and Slot Detail Information. The Main Menu displays a list of all PCI slots in the system. For each slot, the Main Menu displays the following information, as shown in Figure 3:

  • Slot number

  • Slot hot plug status

  • Product name or description of the adapter

  • Status of the slot/adapter (Remove, Replace, or Slot Detail Information)

Figure 3: Configuration Manager Console Main Menu screen.

Pressing <Enter< on a selected slot of the Main Menu will either produce a menu of Slot Options (Remove, Replace, or Slot Detail Information) or take the user directly to the Slot Detail screen. Online, context-sensitive help is available to guide the user through PCI Hot Plug actions.

The Slot Detail screen (Figure 4) displays information about a specific slot and the adapter in that slot. This information includes the following:

  • Slot number

  • Adapter name or description

  • Bus number and type associated with that slot

  • Slot/adapter hot plug status

  • Device and driver information associated with the adapter

Figure 4: Configuration Manager Console Slot detail screen.

Pressing <F2< activates the Slot Options menu, which allows the administrator to take actions such as removing or replacing the card in that particular slot.

Device Drivers

The minimal support required for hot plug functionality is the ability to shut down a single instance of the device driver. When removing an adapter from the system, notification and shut down of the associated device driver must take place through the user interface before removal of slot power. This prevents software errors from occurring.

ODI. Novell's Open Data-Link Interface (ODI) for networking support specifies the driver architecture for network devices. The power of Novell's ODI architecture stems from the fact that by modifying it lightly, the ODI specification encompasses PCI Hot Plug technology. Because of the inherent modularity of intraNetWare, the changes required in third party adapter device drivers are minimal.

The ODI architecture standardizes the support for network PCI Hot Plug drivers by providing a translation layer between the specific device drivers and the OS. This translation layer, called ODINEB, abstracts the ODI device drivers (Hardware Specific Modules) from the PCI Hot Plug system. Device drivers compliant with the modified ODI specification will be PCI Hot Plug compatible and will continue to work in non-hot plug environments.

To obtain PCI Hot Plug functionality, update LAN drivers to be compliant with the ODI 3.31 specification. To obtain Novell's version 3.31 specification, contact Novell Labs or visit Novell web site at http://developer.novell.com/devres/sas.

NWPA. The NetWare Peripheral Architecture (NWPA) specifies the driver architecture for storage devices. The NWPA specification, modified slightly, encompasses PCI Hot Plug technology. Because of the inherent modularity of intraNetWare, the changes required in third party adapter device drivers are minimal.

The NWPA architecture standardizes the support for storage PCI Hot Plug drivers by providing a translation layer between the specific storage device drivers and the OS. Device drivers compliant with the modified NWPA specification will be PCI Hot Plug compatible and will continue to work in non-hot plug environments, thus minimizing rework by third party vendors.

Drivers written to the DDFS specification, NWPA's predecessor, do not support hot plug technology. Therefore, convert storage drivers to NWPA to obtain hot plug functionality. To obtain Novell's version 2.32 specification contact Novell Labs or visit Novell web site at http://developer.novell.com/devres/sas.

Other Adapters. To enable other types of device drivers to support the PCI Hot Plug feature set, the Novell intraNetWare system architecture allows for NLMs to interact directly with the Event Bus. By sending and receiving PCI Hot Plug events, an NLM can perform all functions necessary for PCI Hot Plug.

Installation Tools

Use INSTALL.NLM to locate the required device drivers and properly configure a replaced adapter. Future enhancements to the installation tools will automatically determine the need for additional device drivers when adding adapters to the system.

Hardware and Software Availability

Compaq has already announced support for Hot Plug PCI with its future server releases. To obtain more information about hardware support, visit Compaq web site at http://www.compaq.com.For any of the three hot plug capabilities (hot replace, hot upgrade, and hot expansion), the system hardware will remain the same, but the OS and adapter device drivers will need to accommodate each situation. While hot replacement is already available with intraNetWare, hot upgrade and hot expansion on intraNetWare/NetWare 4.11 is included with Novell's Support Pack 4.0 that was released on September 26, 1997. Another expected enhancement includes the integration of Compaq Insight Manager to provide remote monitoring of hot plug slots.

Conclusion

With PCI Hot Plug hardware and software, replacement of an existing PCI card occurs without affecting other adapter cards or causing any downtime to the system as a whole. Novell and Compaq are working closely to develop hot plug capabilities for the Novell network operating system. Novell driver development kits supporting PCI Hot Plug have been made available to independent hardware vendors.

By leading the PCI Hot Plug Workgroup and partnering with key software companies such as Novell, Compaq is reinforcing its commitment to bring PCI Hot Plug to the market as an open industry standard. In turn it will allow the end user to reap the benefits of this technology without being tied down to proprietary systems.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates