Virtual IP Addresses in the NetWare 6.5 TCP/IP Stack
Articles and Tips: article
Senior Software Engineer
Novell, Inc.
kjambunathan@novell.com
Thanks to Brad Rupp for his valuable contributions to this AppNote.
01 Aug 2003
With the release of NetWare 6.5, Novell has enhanced the TCP/IP stack to support virtual IP addresses. This new feature is another high-availability offering that enables administrators to easily manage name-to-IP address associations of business services. It complements the existing load balancing and fault tolerance features of the TCP/IP stack and enhances the availability of servers that reside on multiple subnets. This AppNote introduces this feature, highlights its advantages with simple examples, and suggests scenarios where it could be put to best use.
|
Topics |
virtual IP address, TCP/IP stack, IP addresses, NetWare features |
|
Products |
NetWare 6.5 TCP/IP stack |
|
Audience |
network administrators |
|
Level |
beginning |
|
Prerequisite Skills |
familiarity with TCP/IP concepts |
|
Operating System |
NetWare 6.5 |
|
Tools |
none |
|
Sample Code |
no |
Introduction
A virtual IP address is an IP address that is bound to a virtual NIC that is driven by a new virtual driver named VNIC.LAN. As the name suggests, this virtual NIC is a purely virtual entity that has no physical hardware counterpart.
Conceptually, a virtual NIC can be thought of as a conventional TCP/IP Loopback Interface with added external visibility. Similarly, virtual IP addresses can be thought of as conventional Loopback addresses with the "127.0.0.0" IP network constraint relaxed.
You can think of a server with a virtual NIC and a virtual IP address as an interface to a virtual "internal IP network" which contains the server as the one and only host. This simple analogy is helpful in understanding virtual IP addresses and appreciating their advantages that will be detailed in this AppNote.
Regardless of their virtual nature, the virtual IP address and the virtual NIC essentially behave like physical IP addresses and physical NICs and are similarly configured through the command line, the INETCFG server-based utility, or the NetWare Remote Manager (NRM) Web-based utility.
Virtual IP Addresses: Basic Concepts
Definitions
Here are some definitions of terms relating to virtual IP addresses.
Virtual driver. The VNIC.LAN driver provided by Novell.
Virtual board (NIC). Any board configured over the virtual driver.
Virtual IP address. Any IP address that is bound to a virtual board.
Virtual IP network. An IP network of which the virtual IP address is a part. In practical terms, this is defined by the virtual IP address together with the IP network mask with which it is configured.
Host mask. The IP network mask consisting of all 1s - FF.FF.FF.FF (255.255.255.255).
Physical IP address. Any IP address that is not a virtual IP address. In practical terms, it is an IP address that is configured over a physical hardware NIC.
Physical IP network. An IP network of which a physical IP address is a part. In practical terms, a physical IP network identifies a logical IP network that is configured over a physical hardware wire.
Unique Characteristics of Virtual IP Addresses
Virtual IP addresses are unique in that they are bound to a virtual "ether" medium as opposed to any "physical" network medium such as Ethernet, Token Ring, and so on. In other words, the virtual IP address space is exclusive from the physical IP address space. As a result, virtual IP network numbers need to be different from any other physical IP network numbers. However, it should be understood that this mutual exclusivity of the IP address space for the physical and virtual networks doesn't preclude the possibility of configuring multiple virtual IP networks in a network domain.
Advantages of Virtual IP Addresses
In spite of their simplicity, virtual IP addresses offer two main advantages over their physical counterparts:
High availability
Unlimited mobility
These advantages mainly stem from the fact that virtual IP addresses are purely virtual and are not bound to a physical network wire. These two unique advantages are explored in detail in the following sections.
High Availability
If defined on a multi-homed server (one with more than one physical NIC), a virtual IP address is a highly-reachable IP address on the server when compared to any of the physical IP addresses, particularly in the event of server NIC failures. This assumes that the server is running a routing protocol and advertising its "internal" virtual IP network- which only it knows about and can reach-to other network nodes.
The limited reachability of physical IP addresses is due to two reasons:
TCP/IP protocols use link-based (network-based) addressing to identify network nodes. As a result, the routing protocols preferentially deliver a packet to the server through the network that the target IP address is part of.
Dynamic routing protocols are extremely resilient to intermediate link and router failures, but they do not adapt well to failures of links at the last hop that ultimately delivers a packet to the destination. This is because the last hop link is typically a "stub" link that does not carry any "routing heartbeats."
Therefore, if one of the physical cards in a server fails, the server-as well as any service that it hosts on the corresponding physical IP address-may become inaccessible. This could be the case in spite of the fact that the server is still up and running and can be reached through the other network card.
The virtual IP address feature circumvents the above problem by creating a virtual IP network different from any of the existing physical IP networks. As a result, any packet that is destined for the virtual IP address is forced to use a virtual link as its last hop link. Because it is purely virtual, this last hop link can be expected to always be up. Also, because all other real links are forcibly made to act as intermediate links, their failures are easily worked around by the dynamic routing protocols.
The above reasoning can be better understood with the help of a simple example. Consider a network configuration as shown in Figure 1.
Figure 1: A multi-homed server with all nodes running some dynamic routing protocol.
In this network, the server is a multi-homed server hosting a critical network service. For simplicity's sake, assume that all nodes are running some dynamic routing protocol.
If the client wants to communicate with the server with the IP address 1.1.1.1, it will try to reach the server through the nearest router, which is Router 1. If the 1.1.1.1 interface were to fail, Router 1 would continue to advertise reachability to the 1.0.0.0/FF.0.0.0 network and the client would continue to forward packets to Router 1. Being undeliverable, these packets would ultimately be dropped by Router 1. Therefore, in spite of the fact that the service is still up and running and can be reached through the other active interface, it is rendered unreachable. In this scenario, a recovery would involve the ability of the client application to retry the alternate IP address 2.2.2.1 returned by the Name Server.
Now consider the same scenario but with the server configured with a virtual IP address and the client communicating with the virtual IP address instead of one of the server's real IP addresses, as shown in Figure 2.
Figure 2: A multi-homed server configured with a virtual IP address.
In this configuration, if the 1.1.1.1 interface were to fail, the client would ultimately learn the new route through Router 2 and would correctly forward packets to Router 2 instead of Router 1. Thus, despite physical interface failures, a virtual IP address on a multihomed server acts as an always-reachable IP address of the server.
Generalizing the above discussion, it can be said that if a connection between two machines is established using a virtual IP address as the end-point address at either end, the connection will be resilient to interface failures at either end.
There are two important side-effects that directly follow from the highly- reachable nature of virtual IP addresses:
They completely and uniquely identify a multi-homed server. A multi-homed server with a virtual IP address no longer needs to carry multiple DNS entries for its name in the naming system.
They significantly enhance the LAN redundancy inherent in a multi-homed server. If one of the subnets to which a server interfaces were to fail completely or be taken out of service for maintenance, the routing protocols will re-route the packets addressed to the virtual IP address through one of the other active subnets.
Bear in mind that this resilience against interface failures provided by virtual IP addresses depends on the fault resilience provided by the dynamic routing protocols, as well as fault recovery features such as retransmissions built into the application logic.
Unlimited Mobility
Virtual IP addresses are highly mobile, as opposed to physical IP addresses which are limited in their mobility.
The degree of mobility is determined by the number of servers to which an IP address on a specific server could be moved, given a choice. In other words, if you choose a physical IP address as an IP address of a network resource, you are limiting the set of potential servers to which this resource could be transparently failed-over to a set of servers that are bound to the same wire (that is, on same subnet).
On the other hand, if you choose a virtual IP address, the set of servers to which the resource could be transparently moved is potentially unlimited. This is due to the sheer nature of virtual IP addresses; they are not bound to a physical wire and as a result carry their virtual network to wherever they are moved. Again, there is an implicit assumption here that the location of virtual IP address, wherever it be, is advertised through the owning server through some routing protocol.
The ability to move an IP address across different machines becomes particularly important when it is required to transparently move (or "fail over" in clustering parlance) a network resource that is identified by an IP address (which could be a shared volume or a mission-critical service) to another server.
This unlimited mobility of virtual IP addresses is a boon to network administrators, offering them more ease of manageability and greatly minimizing network reorganization overhead. For network administrators, the shuffling of services between different IP networks is the rule rather than the exception. The need often arises to move a machine hosting a particular service to some other IP network, or to move a service hosted on a particular machine to be re-hosted on some other machine connected to a different IP network. If the service is hosted on a physical IP address, accommodating these changes involves re-hosting the service on a different IP address pulled out from the new network, and appropriately changing the DNS entry for the service to point to the new IP address. However, if the service is hosted on a virtual IP address, the necessity of changing the DNS entries for the service is eliminated, thereby saving time on an additional overhead.
Other Added Features
Support for Host Mask
Virtual boards support the configuring of virtual IP addresses with a host mask.
Source Address Selection for Outbound Connections
As mentioned earlier, full resilience of connections to interface failures can be ensured only when the connections are established between machines using virtual IP addresses as end-point addresses. That means an application which initiates outbound connections to a virtual IP address should also preferably use a virtual IP address as its local end-point address.
This wouldn't be much of a problem if the application bound its local socket end-point address with a virtual IP address. But there are some legacy applications that bind their sockets to a wildcard address (such as 0.0.0.0). When these applications initiate an outbound connection to other machines, TCP/IP chooses the outbound interface's IP address as the local socket end-point address. In order for these legacy applications to also take advantage of the fault resilience provided by the virtual IP address feature, the default source address selection behavior of TCP/IP has been enhanced to accommodate the use of a virtual IP address as the source IP address. As a result, whenever a TCP or UDP application initiates an outbound connection with a wildcard source IP address, TCP/IP will choose the first bound virtual IP address as the source IP address for the connection.
This enhanced source address selection feature can be enabled or disabled globally as well as on a per-interface basis. This feature is enabled by default on all interfaces.
Reducing the Consumption of Additional IP Addresses
The only drawback in reaping the benefits of virtual IP addresses is the consumption of additional IP addresses. This constraint stems from the requirement that virtual IP network addresses must be different from all other real IP network addresses. While this constraint is not particularly severe in enterprises that use private addressing (where the IP address space is potentially large), it could become limiting in organizations that do not use private addresses.
In enterprises that use fixed-length subnetting together with a dynamic routing protocol like RIP-1, each virtual IP address could consume a large number of host IP addresses. One way to circumvent this problem is to configure a virtual IP address with a host mask of all 1s (that is, FF.FF.FF.FF), thereby consuming only one host IP address. Of course, the viability of this option depends on the ability of the RIP-1 routers on the network to recognize and honor the advertised host routes.
In autonomous systems that use variable-length subnet masking (VLSM) together with routing protocols like RIP-II or OSPF, the consumption of additional IP addresses is not a big problem. You could simply configure a virtual IP address with as large an IP network mask as possible (including a host mask of all 1s) and thereby limit the number of addresses consumed by the virtual IP address space.
Note: As of this writing, the propagation of virtual IP addresses to other network nodes is supported only through RIP. However, Novell is planning to add support for OSPF in the near future.
Typical Deployment Scenarios
Virtual IP addresses can potentially be used in any scenario that necessitates exploiting of their unique advantages discussed in this AppNote. This section details some typical deployment scenarios that are expected to be widely used.
Business Continuance Clusters
With the amount of mission-critical information being stored electronically today, businesses can no longer afford to lose access to their data. Access problems lead to expensive downtime, loss of revenue, or even failure of the business. To protect against this, many IT managers see the need to implement a disaster recovery solution. Using Novell Cluster Services, an IT department can build a highly- available disaster recovery solution using commodity hardware.
A Novell Business Continuance Cluster is a group of two or more independent, geographically-dispersed clusters. The data is replicated between the two clusters using SAN (Storage Area Network) hardware. Figure 3 shows the essentials of this clustering solution.
Figure 3: A Business Continuance Cluster.
As an example, consider a fictitious company called Acme Data Services. Because of the mission-critical nature of Acme's data, they have chosen to build a disaster recovery solution using a Novell Business Continuance Cluster. The primary cluster and data center is in New York, while the secondary cluster and data center is in New Jersey. The two data centers are connected via a SAN. Furthermore, clients have connectivity to all the nodes in both clusters. In the unfortunate event that some form of disaster destroys Acme's primary data center, services can quickly and easily be restarted on the cluster nodes in the secondary location.
In any network environment, one of the first obstacles that must be tackled is how clients locate and connect to the services. A Business Continuance Cluster can exacerbate this problem because services can migrate to nodes on a completely different network segment. While there are many potential solutions to this problem, such as DNS and SLP, none of them offers the simplicity and elegance of virtual IP addresses. With virtual IP addresses, the IP address of the service can follow the service from node to node in a single cluster, as well as from node to node in separate, distinct clusters. This makes the client reconnection problem trivial; the client only has to wait for the new route information to be propagated to the routers on the network. No manual steps are required, such as modifying a DNS server.
To use a virtual IP address in a Business Continuance Cluster, the use of a host mask is recommended. To understand why, consider the fact that each service in a clustered environment must have its own unique IP address-or, in this case, a unique virtual IP address. Furthermore, consider that each virtual IP address belongs to a virtual IP network whose route is being advertised by a single node within a cluster. Because Novell Cluster Services can migrate a service and its virtual IP address from one node to another, it follows that the virtual IP network must migrate to the same node as the service. If multiple virtual IP addresses belong to a given virtual IP network, one of two events must occur:
All services associated with the virtual IP addresses on a given virtual IP network must fail-over together.
The virtual IP addresses on a given virtual IP network must go unused, thereby wasting a portion of the available address space.
Neither of these situations is desirable. Fortunately, the use of host masks remedies both.
Once the appropriate virtual IP addresses and host masks have been determined, virtual IP addresses can be enabled in a Business Continuance Cluster via a three-step process.
First, the AUTOEXEC.NCF file on each node in both clusters must be modified to add the following lines, which load the virtual driver and create a virtual board named "VNIC". The new virtual board is then bound to the real IP address 1.1.1.1.
LOAD VNIC NAME=VNIC BIND IP VNIC Mast=255.255.255.0 Address=1.1.1.1
Second, the command to bind a virtual IP address for the service must be added to the cluster resource load script. The following is an example of a cluster resource load script for a standard NetWare volume called "Homes". This example uses host masks and assumes the virtual board has been named VNIC. Notice the addition of the "BIND IP VNIC Mask=255.255.255.255 Address=4.4.4.1" command, which binds the virtual IP address to the virtual board.
nss /poolactivate=HOMES mount HOMES VOLID=254 CLUSTER CVSBIND ADD BCC_HOMES_SERVER 4.4.4.1 NUDP ADD BCC_HOMES_SERVER 4.4.4.1 BIND IP VNIC Mask=255.255.255.255 Address=4.4.4.1
Finally, the command to unbind the virtual IP address must be added to the cluster resource unload script. The following is the matching cluster resource unload script for the same NetWare volume discussed above. Notice the addition of the "UNBIND IP VNIC Address=4.4.4.1" command, which unbinds the virtual IP address from the virtual board.
UNBIND IP VNIC Address=4.4.4.1 CLUSTER CVSBIND DEL BCC_HOMES_SERVER 4.4.4.1 NUDP DEL BCC_HOMES_SERVER 4.4.4.1 nss /pooldeactivate=HOMES /overridetype=question
Virtual Server Farms Behind L4 Switches
Over the last few years, enterprises have seen a large increase in network traffic seeking to access their mission-critical services. As a result, administrators are increasingly resorting to a Virtual Server Farm architecture in which a cluster of servers is configured behind a Layer 4 (L4) switch. The L4 switch distributes the incoming traffic across these servers, thereby providing better response times and increased scalability.
The popularity of these deployments has resulted in varied architectures for L4 switches. One such architecture is "MAC Bridging." In this architecture, the L4 switch and the individual servers in the farm share a common subnet and a "Virtual IP Address" over which the service is hosted. Even though this address is shared across both the L4 switch and the individual servers in the farm, every incoming IP packet addressed to the "Virtual IP Address" is intercepted only by the L4 switch. The switch then "bridges" the packet to a selected server in the farm by replacing the target MAC address (which is its own) with the MAC address of the selected server in the cluster.
Figure 4 shows a typical Virtual Server Farm configuration.
Figure 4: A typical Virtual Server Farm configuration.
Two key necessities in this architecture are:
Individual servers in the farm must recognize the bridged packet from the L4 switch as their own. Therefore, the virtual IP address needs to be bound on the servers.
Only the L4 switch should intercept incoming requests, not any of the individual servers. For this to work, the servers should not respond to any incoming ARP queries for the virtual IP address.
These necessities are easily met by configuring the "Virtual IP Address" of the farm as a virtual IP address on the virtual NIC.
Deploying Pre-production Servers in Enterprise Networks
With ever-changing business needs and increasingly fierce competition to provide better services, all enterprises must keep themselves abreast of emerging trends, technologies, and products. Therefore, network administrators are frequently required to procure and deploy new server software solutions. New solutions should be validated in a typical test environment before they are deployed in production environments.
However, it is a common observation that, no matter how sophisticated test environments are, they cannot exactly imitate real production environments that are less controlled and more dynamic. Thus the initial deployment of pre-production quality servers in a real production environment is an inescapable concern for network administrators. They must balance the two seemingly- conflicting requirements of exposing the pre-production server to the rigors of real-life production traffic, while at the same time provide a way to immediately revert back to the proven production environment in case of a failure.
In this type of scenario, it would be a boon to network administrators if there were a solution that automatically falls back upon the regular service in case of an unpredicted outage of the pre-production server and which automatically switches to the pre-production server once it is brought online. Such a solution is quite feasible-on a primitive level with minimal investment-by using the host mask binding feature supported with a virtual board.
All the network administrator has to do is configure the IP address of the production server on the virtual board of the pre-production server, but with a host mask. This configuration, together with the enabling of a dynamic routing protocol on the network, would ensure that the preferred destination for that IP address is the pre-production server. The routing topology would fall back upon the production server in case of an unpredicted outage of the test server.
Figure 5 shows a typical pre-production server configuration.
Figure 5: Deploying pre-production servers on an enterprise network.
Other Possible Uses for Virtual IP Addresses
In addition to the scenarios detailed above, virtual IP addresses could also be preferentially used for:
Remote control of a multi-homed server
End-point addresses of IP-over-IP tunnels
LAN-mapping address for point-to-point connections
Conclusion
The TCP/IP stack included with NetWare 6.5 supports virtual IP addresses. This new feature complements the existing load balancing and fault tolerance features of the TCP/IP stack and enhances the availability of servers that reside on multiple subnets. Virtual IP addresses enable administrators to easily manage name-to-IP address associations of business services.
The information provided in this AppNote is derived strictly from test scenarios; there may be deviations from these results in real user scenarios. Novell does not recommend deploying any new configurations directly in a production network. Configuration changes should always be verified in a simulated test network before being deployed in a production environment.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.