Novell is now a part of Micro Focus

Protecting Your Network from Hackers with Advanced BorderManager Packet Filtering

Articles and Tips: article

Martin Day
Consultant
Novell Consulting
mday@novell.com

01 Sep 2000


This AppNote addresses how to optimize the packet filters used by BorderManager to increase security. It provides practical information on how to competently work with packet filters and is specifically aimed at administrators charged with configuring and administering BorderManager firewalls.

Introduction

Packet filters form a fundamental component of the overall security of the BorderManager firewall. They need to be designed so as to allow the required functionality but at the same time prevent all undesired traffic from passing.

Recent distributed Denial of Service (DoS) attacks against high-profile Internet sites such as Yahoo! and Amazon have heightened awareness of security issues on the Internet. Not only could your organization be the target of such attacks but their distributed nature could mean that your organization is used as an unwitting accomplice to these attacks by hosting agents that can be remotely controlled and used to launch attacks against other sites.

Before an attacker can attempt to compromise a site it is necessary to search for open channels that are exploitable. There are many automated tools available that can be used for such "port mapping". These tools use a variety of techniques to fulfil this function and can provide reports detailing open channels. Attackers are then able to further investigate the open channels before launching specific attacks. If a firewall is able to avoid returning useful information to the automated scanning tools, then it reduces the chances of it (or the hosts that it protects) from being specifically targeted by an attacker.

The aim of this AppNote is to provide you with the knowledge necessary to optimize the packet filters used by BorderManager so as to increase the overall security of the product. This article is specifically aimed at administrators charged with configuring and administering BorderManager firewalls.

Note: Suggestions made within this AppNote should only be implemented by those with a full understanding of the issues involved. Badly configured firewalls are worse than having no firewalls at all as they lead to a false sense of security.

Additional information can be obtained from Novell's documentation and support sites: http://www.novell.com/documentation and http://support.novell.com. Furthermore, information regarding Nmap can be found at http://www.insecure.org/nmap.

Fundamentals of Packet Filtering

For the purposes of this discussion, only packet forwarding filters will be discussed as these are the filters that are most frequently adjusted to reflect a specific organisation's needs. Routing filters (e.g. RIP, OSPF, EGP) are not discussed.

Basics

Packet filters operate at the Data Link, Network, and Transport layers and so can only make decisions on which packets to pass based on limited information. For example, it is not possible to use packet filters to restrict traffic from specific users. The information that can be used to make decisions is restricted to:

  • Protocol type (e.g. TCP, UDP)

  • Source IP address

  • Destination IP address

  • Source port number

  • Destination port number

Consider the following situation: a client (or requestor) wishes to communicate through a firewall to another host that is running a service the client wishes to access (e.g. Telnet, HTTP etc). The client will send packets using a so-called "High" port. High ports are in the range 1024-65535. These packets need to indicate the service that they are destined for so that the destination host can deal appropriately with the packet. Common services utilise "well-known" ports. For example, HTTP uses port 80, Telnet uses port 23.

Thus, for communications to occur, the firewall needs to allow packets from the client with a high source port and a destination port of 80 in the case of HTTP, and port 23 in the case of Telnet.

The next step is to allow response packets back to the client so that the client is able to retrieve information. Considering the HTTP example, above, the web server will send packets from port 80 to the client. In order for the client to be able to determine what to do with the return packet it must match the port that was used in sending the original request packet.

Thus, the intervening firewall must allow response packets with a source port of 80 and a high destination port. This is illustrated in Figure 1.

Figure 1: HTTP traffic outbound and resulting response.

Static Packet Filtering

The earlier versions of BorderManager relied on static packet filters and the previous example is, in effect, referring to static packet filtering. In this approach the firewall does not keep track of the "state" of traffic passing through it. Thus, the firewall is not able to determine if an incoming packet is actually the first packet in an externally-initiated session or if it is a response to an internal client-initiated session. In order for responses to propagate through the firewall, a response filter exception must be permanently enabled. In the HTTP example, above, this means that high ports to an internal host(s) remain continually open. Obviously, the level of protection such filters provide is limited.

Firewalls that implement network address translation (NAT) go some way to offsetting the negative effects of these filters in the specific case where the client makes requests via a "dynamic" NAT interface. Via this mechanism, externally-initiated traffic that attempts to utilise the open high ports will fail as the dynamic NAT table will not contain an entry for this traffic and so will drop the packet.

It is not the intention of this AppNote to discuss NAT in any detail and interested readers should consult the documentation on the Novell web site (http://www.novell.com/documentation) for further information.

Dynamic Packet Filtering

Also known as "stateful" packet filtering, this approach provides a higher level of security than static packet filtering. Here, the firewall keeps track of the outgoing packets that it allows to pass and allows only corresponding response packets to return.

A single outgoing stateful filter will automatically allow the creation of a temporary (time-limited) inbound filter exception for the connection. This temporary exception will only allow packets from the host and port to which the outbound packet was sent. Thus, internal clients can access remote services without leaving permanent incoming channels open.

Using the HTTP example from the previous section, only a single filtering exception is required for a client wishing to communicate with an external web server. The stateful filter exception would have a high source port and a destination port of 80.

Usage of stateful filters, however does have a downside. Because the firewall must keep track of the "state" of connections, there is a performance overhead involved, thus stateful packet filtering is slower than static packet filtering.

Default BorderManager Packet Forwarding Filters

A standard installation of BorderManager 3.5 takes the generic approach of denying all traffic through the public interface and then allowing exceptions to permit the base functionality. The proxy services of the firewall together with VPN capability are considered base functionality.

In order for the default filters to provide this functionality as simply as possible, static filters are used. We will discuss replacements for these static filters (to enhance security) in a later section.

This section lists the default filters by packet type and briefly discuss each of them.

<ANY<

This filter exception is the only outbound exception. This filter type covers all IP whether it is TCP, UDP, ICMP etc. As an aside, this exception will allow outbound pings (responses to pings are another matter).

The key point here is that the source address is restricted to the public address of the BorderManager server itself.

For those installations that make use of secondary IP addresses, it should be noted that this filter that allows outgoing TCP/IP traffic will not apply to services using the secondary IP addresses.

dynamic/tcp

This filter exception is designed to allow TCP response packets to those packets generated by the BorderManager server itself. Thus, the source port can be any port and the destination port will be any high port (that the BorderManager server originally used to send packets).

In this case the destination address is that of the public interface of the BorderManager server itself. This is the same for the remaining default filters.

The two filter exceptions discussed so far will allow the majority of most typical communications through BorderManager. For example, TCP communications from the proxy for HTTP, FTP, NNTP, DNS (although most traffic will be UDP), SMTP etc.

dynamic/udp

This filter performs the same function as the dynamic/tcp filter but for UDP packets.

www-http

This filter exception causes a certain amount of confusion which isn't helped by the fact that the description (as displayed in the FILTCFG utility) exceeds the viewable area. At first glance it would be assumed that this filter is to enable the web proxy cache to function for web browsing. That is not the case. Careful examination of the exception reveals that it allows incoming traffic to a port 80 listener, that is it will enable incoming traffic to a web accelerator, if one is configured. Response packets from your internal web server are able to pass through the firewall courtesy of the <ANY< rule listed above.

Again, an area that causes problems for some administrators that wish to employ a web accelerator involves secondary IP addresses. These default filters will allow web accelerator functionality for an accelerator listening on the primary public address only. If accelerators are to be configured with port 80 listeners on secondary IP addresses, then additional filter exceptions must be configured.

Accel-Auth

This filter exception is an accompaniment for the www-http exception. Here, the destination port is 443, which is the standard port for SSL authentication to a web server. It is only necessary if your accelerated web host has a requirement to securely authenticate sessions.

ipx/tcp

This is the first of four inbound filter exceptions required for VPN functionality. In this case it is used with the site-to-site VPN configuration. The destination port is 213 (TCP).

VPN-AuthGW

Again for VPN functionality, but specifically for client-to-site authentication. This uses a destination port of 353 (TCP).

VPN-KeepAlive

This uses the same port as the preceding exception (353), but this time using UDP as the transport protocol. As its name suggests, it is used to allow keepalive packets to pass.

VPN-SKIP

SKIP (Simple Key Management for Internet Protocol) is the protocol used for the exchange of keys used in the VPN. It is an IP protocol but is completely different from TCP or UDP which is why it is referred to by an odd-looking protocol name (57).

Port Scanning and BorderManager - Nmap

This section demonstrates how the packet filters in a standard installation of BorderManager react to typical automated port scans.

There are many port scanning techniques and tools available. Nmap (http://www.insecure.org/nmap) is one such tool that implements many of the standard port mapping techniques and is used for the basis of the following discussion.

TCP Connect Scan

This approach sends a SYN (synchronise) packet to every port randomly using random source ports and tries to open a full TCP session. This involves the standard three-way TCP handshake:

  • Step 1: client sends a packet to the server with the SYN flag set.

  • Step 2: server responds to the client with the SYN and ACK flags set.

  • Step 3: client responds to the server with the ACK flag set.

Using random source ports and not scanning target ports in numerical order makes it more difficult for log analysis tools to spot the scan.

If the destination port is a high port, then the default filter exceptions on BorderManager will allow the packet to pass and so the server will respond to the packet with a RST ACK (reset acknowledge) as the port will not actually be open. The exceptions to this in a standard configuration are TCP ports 2000 and 3351 (used for Btrieve functionality) and in most cases port 1959 (proxy mini web server) that are actually open and so the server will respond with a SYN ACK.

Any other ports that allowed through the filters but may not actually be used (e.g. 80, 443, 213, and 353) will also result in sending a RST ACK.

Nmap interprets RST ACK responses as meaning the particular port is closed. If no response is received, then nmap interprets this as "filtered" i.e. a firewall is covering that port. Ports that respond with a SYN ACK are obviously interpreted as open.

Many installations of BorderManager are only required to provide a forward proxy, thus it would be possible to utilise stateful filters instead of the defaults that would then not provide any responses to these TCP connect scans.

When configuring such an exception, the destination interface will, of course, be specified as the "Public" interface but it is tempting to specify the source interface as "Any Interface" rather than specifying the private interface(s) explicitly. This means that an external scan (from the "Public" interface) to the public address (which is also listening on the "Public" interface) will pass through the filters. Although, you may not have the port(s) specified in the filter exception actually open, the server will still respond to the scan with a RST ACK thus providing the scanner with information.

TCP SYN Scanning

This is also known as "half-open" scanning and instead of trying to open a full connection the scanner responds to open ports with a RST flag. This has less chance of being logged.

Packet filtering logs will show the difference in these two scans. However, the end result is the same in both cases. Accessible, open ports are correctly interpreted as open. Non-filtered ports also respond to the scanner as before.

The same suggestion about stateful filters applies here.

Ping Sweep

This approach isn't truly a port scanning technique as ICMP (the protocol used by Ping) does not relate to ports. With default filters in place, there is no exception to allow incoming pings (or any ICMP packets for that matter). Thus, the scanner will receive no indication that a host is actually present.

However, nmap also allows the user to perform "TCP pings" in addition to the normal ICMP pings. Here, an ACK packet is sent to a particular port in the hope that the port will respond with a RST. Due to the prevalence of web servers, the default port used with such a TCP ping is port 80. From the previous section, it should be apparent that the incoming filter exception, www-http, will allow these packets through and in combination with the ANY outgoing rule from the BorderManager server itself, responses to these packets are also allowed out.

The above suggests that if the particular implementation of BorderManager is not making use of the HTTP accelerator, then the www-http exception should be removed.

As an aside, nmap has the option to precede all scans with an ICMP or TCP ping. If the ping fails to get a response then nmap will assume that the target is not there and so won't perform the rest of the scan. As the majority of scanning attempts will be made against a range of addresses rather than a specified address, it is likely that the ping option will have been selected to avoid wasting time. Thus, preventing responses to such pings provides a simple approach to avoid in-depth scanning from many potential hackers.

UDP Port Scan

When scanning UDP ports, some hosts won't respond with an acknowledgement if the port is open or with an "ICMP_PORT_UNREACH" error if the port is closed. (UDP is a connectionless protocol, unlike TCP.) However, some do, hence this scanning method which aims to get a picture of closed ports and so, by a process of elimination, determine open ports.

This method is generally slow, especially against hosts that limit the ICMP error message rate (e.g. Linux). With BorderManager (and specifically NetWare) the scan is extremely slow as no responses are sent (regardless of filters). Thus nmap will eventually conclude that all ports are open i.e. the utility effectively fails with lots of false-positives.

However, if the BorderManager firewall is mapping a secondary IP address to an internal host (perhaps for SMTP to a mail server, for example) then this scanning method may give valid results against the secondary IP address. Hence, this is a very good reason to restrict filter exceptions to allow communication to this internal host to the absolute minimum. For example if SMTP is required to the internal host, then onlyallow port 25 -- don't simply allow all traffic in and out from the IP address of the internal host.

FIN Stealth

This approach sends FIN packets that may result in the target host responding with RST packets from closed ports. Open ports tend to ignore the FIN packet.

With default filters, BorderManager will send RST packets in response to requests to high ports as they are allowed through the filters (via the incoming exception "dynamic/tcp"). Any of these high ports that are actually open won't respond and so nmap will rightly determine those ports as open. Attempts to ports that are blocked by filters are dropped and so nmap will wrongly determine these ports to be open.

Again, this approach results in a lot of false-positives. The best approach to negate any usefulness of this method is to use outgoing stateful filtering (if possible) instead of relying on an incoming static filter exception such as the default dynamic/tcp.

Bounce Scan

With this approach, the attacker makes use of the FTP Proxy function. By connecting to an FTP server, it may be possible to have the FTP server send a file elsewhere. Fortunately, the FTP proxy feature is largely disabled on FTP servers and so this approach won't work.

This scanning attack relies on setting up an FTP control session to an FTP server (ideally behind the firewall protecting the network that the attacker wishes to scan) and then having that FTP server send data to a defined port. The target host should give an error if the port is closed but should report a success message if the port is open.

Obviously, data sent from external FTP servers with such an approach would be blocked by packet filters. However, should BorderManager be shielding an externally-accessible FTP server then that server should be placed on a DMZ (de-militarised zone) but more importantly, the FTP proxy feature should be disabled.

Fragmentation Scanning

This is a modification of other scanning techniques where the probe packets are fragmented. It can be used with SYN or FIN methods.

With BorderManager, fragmented packets are automatically dropped before the packet has to be processed by the filter rules (and so isn't logged).

General Tips and Tricks

This section contains various tips that should prove useful when configuring packet filters. They are in no particular order.

  • When specifying source and destination interfaces try to avoid using "All Interfaces" and use specified interfaces instead. For example, a filter exception from All Interfaces to Public is usually intended to allow internal hosts connectivity out through the public interface. However, using "All Interfaces" instead of the specified private interfaces means that the filter actually also applies to external users sending packets from the public interface to addresses on the public interface of the BorderManager server. This could potentially be a security hole if the corresponding ports are open and source/destination addresses have not been specified.

  • Specifying the interface explicitly as for the previous point makes the filter exceptions more clearly readable. For example, if you are creating an exception for external users to reach accelerators configured to listen on the Public interface then specify both source and destinations interface to be Public. Examination of the filter reveals its exact purpose.

  • The default filters only apply to the specific, primary, address of the interface to which the filters were applied. If you subsequently add secondary IP addresses to this interface then additional rules to allow the required services to function will be required.

  • When enabling additional functionality through BorderManager it is always better to use application proxies rather than packet filters whenever possible as this is more secure.

  • With secondary IP addresses enabled on a Public Interface that also has Network Address Translation (NAT) enabled careful consideration should be given to the entry that must be made in the static NAT table to allow access to and from that address. A common example is when enabling an HTTP accelerator on a secondary IP address. As the accelerator is actually listening for requests on the same interface as that on which NAT is enabled, both public and private addresses in the static NAT table should be set to the secondary IP address in question.

  • Whenever filters are changed, they should be backed up. The corresponding file is SYS:ETC\FILTERS.CFG. Copy this file off the server and rename it to something meaningful, for example, "Default filters plus smtp 1.cfg". The sequence number at the end is a good idea to keep track of the order of the cumulative changes. Should subsequent changes be made which cause problems/security issues, then it is a simple matter to find the last useable version of the filters and then copy them back to SYS:ETC\. Before doing so it is recommended to take the following steps:

    1. Unload IPFLT, IPXFLT, and FILTSRV.

    2. Delete the current, problematic, SYS:ETC\FILTERS.CFG.

    3. Reinitialize system and confirm (via filtcfg.nlm) that no filters are now set.

    4. Copy over the required filters file to SYS:ETC\.

    5. Rename the file to FILTERS.CFG

    6. Unload IPFLT, IPXFLT, and FILTSRV.

    7. Reinitialize system and confirm that correct filters are now in place.

  • Following on from the previous point, don't make multiple changes to the filters at the same time. Make one change at a time, ensure functionality is as expected, and then back up the new filters.cfg file before making further changes.

  • When new services are introduced that require new protocols to pass through the BorderManager server it is tempting to allow all traffic to pass to and from the hosts concerned so as to ensure that the service will function. This approach should be avoided at all costs as it seriously affects overall network security. The best approach is to find out the minimum ports that must be opened (either from the vendor of the service or by packet tracing) and only allow those ports to be opened.

  • Whenever filter exceptions are created, ensure that a sensible remark is added. What may seem like an obvious filter exception at the time of implementation may seem confusing six months later, especially when many filter exceptions have been added.

  • If pairs of static filters are used to allow internal hosts to communicate to external hosts, then ensure that the ACK bit filter option is enabled on the response exception. This will at least provide some additional security.

  • Although, it may be a good idea to remove default filter exceptions that aren't required in a particular implementation, it is not such a good idea to modify the default filters. For example, modifying a filter without changing the remark may mean that later on when the filters are being checked, the casual auditor may simply read the remark without closely examining exactly what the filter does. Thus, security holes may be missed.

  • In general terms, using stateful filter exceptions for internal hosts to communicate externally provides higher security than static filter exceptions. However, for incoming traffic there's no real security benefit from using stateful filter exceptions. In these circumstances, choice of filter type is ultimately a decision based on speed versus manageability (of a fewer number of exceptions).

  • Remote console access to a firewall is generally not a good idea as anyone who obtains the relevant password can make remote modifications to the firewall that greatly influences its security, and hence the security of the entire network. This is of vital importance when the Pure IP remote console utility, RCONJ, is used. By default, RCONJ listens on TCP port 2034 that is a high port. Default filters will allow external communication to this port on the BorderManager server itself via the dynamic/tcp (for incoming) and the any filter exception (for outgoing responses). Thus a user on the Internet simply has to guess the remote password (which in many cases is a very simple word) to have remote console access to the server. Hence, never, enable RCONJ on a BorderManager firewall.

  • TID 10018659 explains in detail how to approach troubleshooting of packet filters. It also includes relevant set parameters that contribute to the security of the firewall (such as "Set Filter Packets With IP Header Options").

  • When troubleshooting packet filtering issues, adopt a structured approach. Typically ask the following questions:

    • What hosts will initiate the communications and where are they?

    • What ports must be opened in both directions?

    • Will security be improved by specifying source and destination host IP addresses explicitly? Or will that come at too high a price as far as management is concerned? For example, if 230 hosts on a subnet are to be allowed access via a particular filter exception then it makes sense to specify the subnet (or all addresses, if hosts on multiple subnets are involved) rather than individual IP addresses.

    • Is a NAT interface involved?

    • Are the appropriate static NAT table entries applied?

    • Is it necessary to reach services actually running on the BorderManager server through an interface configured with dynamic NAT? If so, SET NAT DYNAMIC MODE TO PASS THRU = ON.

    • Where are the actual port listeners awaiting client requests located? Are they on the same interface from which the request initiates (e.g. for HTTP accelerators) or is the listener on an internal host. Configure both filters and NAT appropriately.

    • Have a clear understanding of any routing issues. For example, consider an external user attempting to connect directly to an internal host. The target host must either have a static route configured back to the requestor or it must have a default route (configured or learnt via a routing protocol) appropriately set.

Conclusion

From the preceding sections, you may be tempted to remove your default filters and start employing stateful filters exclusively. Bear in mind, however, that the default filters have been designed to provide as much functionality as possible out of the box without opening any major vulnerabilities. The default filters need to allow legitimate externally-initiated traffic to pass, such as access to site-to-site VPN, client-to-site VPN, HTTP acceleration, and SSL authentication. Any reconfiguration of filters needs to take these requirements into consideration.

As a general guideline, the more services that are provided through the firewall have a detrimental effect on the overall security of the firewall - if you allow direct communications to an internal host through the firewall then you should ensure that services offered by that host cannot be compromised. This is best achieved by ensuring all security patches relevant to the host are applied. Of course, rather than providing direct access to any hosts via the firewall, use of application proxies are far more secure.

Finally, it cannot be stressed too highly that if you are unsure of the repercussions of changing default filters --Don't do it! Seek assistance.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates