Novell is now a part of Micro Focus

Using Public-Domain Site Blocking Lists with Novell BorderManager

Articles and Tips: article

Marcus Williamson
Consultant
Connectotel Ltd.
marcus.williamson@myrealbox.com

01 Feb 2003


This AppNote discusses the issues and costs associated with users accessing inappropriate Internet content from the workplace. It then describes how network administrators can use free public-domain site blocking lists in conjunction with Novell BorderManager and third-party tools to enforce network-wide Internet access policies.

Because this AppNote was written in the U.K., it retains the British spelling used by the author.


Topics

Internet site blocking, proxy servers, network security

Products

NetWare, Novell BorderManager 3.7

Audience

network administrators

Level

intermediate

Prerequisite Skills

familiarity with NetWare and BorderManager

Operating System

NetWare 4.11 and above

Tools

Connectotel AdWall, FileWall, LinkWall

Sample Code

no

Introduction

Internet access from every desktop has become ubiquitous within most organisations. Whilst the provision of Internet access for all offers many benefits, it also presents challenges to the network administrator who must manage that Internet access. The administrator must ensure that such access is provided within the organisation's acceptable use policy, in accordance with legal requirements, and in keeping with the appropriate financial budgets.

This AppNote discusses how network administrators can make use of free public-domain site blocking lists in conjunction with Novell BorderManager on NetWare networks to enforce network-wide Internet access policies. It begins with a discussion of the policy, legal, and cost-related issues, as well as the various types of unwanted content. It then covers some of the site lists that are available in the public domain, and concludes with instructions for installing and configuring these lists.

Policy, Legal, and Cost Issues

This section provides a discussion of issues surrounding employee Internet usage, in terms of corporate policies, legal concerns, and costs.

Policy Issues

Every organisation should have a defined acceptable use policy (AUP) which should be made available, understood, and signed by all users who have Internet access. An AUP should make clear to users what constitutes acceptable use and indicate what types of sites and content are not appropriate for use within the business environment. For example, these may include sites relating to pornography, racism, sports and leisure, and even free e-mail.

In the event of Internet abuse, a documented disciplinary procedure should detail what actions will be taken by management. The exact procedure adopted will depend on the company culture and on the local legislative regime in the country where the abuse occurs.

An organisation's management may decide on one of two main approaches for managing Internet content, which we shall term here "proactive blocking" and "reactive monitoring."

Proactive Blocking. This approach for managing Internet access involves making policy decisions on the types of Internet content which will be permitted, and not permitted, for use within an organisation.

In this method, enforcement of the policy decisions is usually carried out using site blocking software, which may be installed on each network client or on a proxy server. A proxy server is a system which provides access to the Internet on behalf of a user, thus acting as the user's "proxy." The network administrator defines rules dictating who can access which sites, by individual name, by groups of users, and by collection of sites.

Site blocking software can usually be configured to use blacklists or whitelists, with a blacklist containing lists of sites to be blocked, whereas a whitelist contains sites which are deemed to be universally "suitable."

Reactive Monitoring. This approach for managing Internet access involves no active blocking of Internet sites. Instead, network administrators use logs retrospectively to determine which Internet sites have been accessed, when, and by whom. In the event of Internet abuse, the logs can be used to determine which user was responsible for the abuse.

Note that in many jurisdictions, routine monitoring of Internet access logs is not acceptable under human rights legislation, even when this consists of personal Internet use within business hours. In these cases it is necessary to obtain specific legal authorisation, usually in the form of a court order, to obtain access to information about a user's Internet access.

Hybrid Approach. In this approach, elements of both proactive blocking and reactive monitoring may be used, so that sites are blocked via predefined rules but logs are also maintained for subsequent analysis. It is this approach which is most commonly adopted, as it provides two levels of monitoring.

Legal Issues

Some countries, such as China and Saudi Arabia, filter Internet access at the point of connection into the country via "super proxies." This allows the governments to control what citizens may and may not see via their Internet connection. Such blocking has been criticised by most other administrations. For example, in April 2002 the European Uniton (EU) unanimously rejected the blocking of pornographic sites at the Internet borders of the EU. In doing so, the European Parliament said, ". . . such drastic measures do not solve the problem of sites outside the EU nor that of sites which are legal for adults but could be harmful to children or young people" (European Parliament vote on 11 April 2002: http://www.nordicom.gu.se/unesco/notices.html).

So, in most countries, it is left to the network administrator to ensure that unsuitable sites are not available to Internet users. However, there are also legal requirements in some countries to ensure that access to unsuitable sites is not available to minors.

For example, in the United States, the Children's Internet Protection Act (CIPA) entered into force in April 2001. This mandates site blocking for K-12 (primary) schools and public libraries, and denies access to certain federal funds for organisations not meeting this requirement. It is therefore essential for administrators in this type of environment to have a blocking solution in place.

Another source for potential legal issues is through sexual harrassment via the access and display of pornographic images within the workplace. In the US, sexual harrassment is an unlawful form of discrimination according to the Civil Rights Act of 1964, and companies can be sued for exposing employees to such harrassment. For example, in 1999 three employees of a Cincinatti-based company brought a suit against their employer for exposing them to pornography downloaded from the Internet (Orange County Register, 7 May 1999: http://www.xstop.com/pressroom/articles/05_07_99_forced_viewing.html).

Cost Issues

The cost impacts of unwanted Internet activity can be divided into a number of areas as follows:

  • Bandwidth costs

  • Support costs

  • Time costs

  • Legal costs

Bandwidth Costs. Whilst access to Internet sites containing HTML and graphics files will not necessarily impose a strain on an organisation's bandwidth, the use of streaming video and audio certainly will. It is common to find Internet access being used for listening to Internet radio, or to access video streams which may not be relevant to business operations. In these cases, the costs are in terms of using bandwidth on both the "Internet" and "LAN" segments of the network.

The LAN side of the equation will not usually be such a concern, but Internet bandwidth costs can be considerable. While exact prices vary from region to region, a minimum estimate of around $3000 per annum per megabit/second of bandwidth is not unreasonable. At that price, the unnecessary use of streaming media will soon make its impact felt in cost terms.

Support Costs. Support costs can increase as a result of the installation of games and non-standard screensavers and wallpaper files, which take up memory and disk space on users' computers. One approach to tackling the use of non-standard screensavers and wallpaper is to implement desktop policy packages using software such as Novell's ZENworks for Desktops. However, a still better approach is to block this type of material at the source using software to deny access to "games" sites.

Misleading advertising is a further headache for the administrator. Messages such as the one shown in Figure 1 can appear on Web sites, leading users to believe that there is a problem with their computer.

Figure 1: Advertising graphics can mislead users into believing something is wrong with their computer.

Blocking this type of site can be done using software which looks for graphics from known advertising sites and prevents them from reaching the user.

Time Costs. According to The UCLA Internet Report 2001 - "Surveying the Digital Future," the percentage of users who access non-business sites from the workplace has increased from 50.7% to 60.7% between 2000 and 2001. This probably reflects the increasing availability of Internet access within the workplace. Additional statistics are given in this extensive report, which can be found at http://www.ccp.ucla.edu/pdf/UCLA-Internet-Report-2001.pdf.

Inevitably, many users will want to make use of the Internet for personal use from the work environment. However, it is up to management to decide whether this should be allowed, or whether restrictions should be placed on certain sites or services, such as the use of free personal e-mail, for example. If management decides that such services are not to be made available from within the workplace, the network administrator will need to develop a strategy and select suitable tools for blocking access to these sites.

Legal Costs. As indicated above, organisations may leave themselves open to legal action, with resulting legal costs and fines, if they provide access to "unsuitable" material from within the workplace or educational institution. Legal cost considerations should also take into account the possible damage to an organisation's reputation, which could be caused by negative publicity resulting from high profile legal cases.

Another legal cost to consider is when an employee is terminated for violating the AUP, comprehensive and accurate logging protects the employer against a wrongful dismissal suit.

Unwanted Content Types

Unwanted content can be divided broadly into one of the following three types:

  • Advertising

  • Unwanted file types

  • Undesirable sites

Many solutions are available to help you deal with unwanted content. Examples include SurfControl and N2H2, in addition to manual entry of sites to block. In this AppNote, we'll use as examples products from Connectotel.

Advertising

Advertising is an unwanted type of Internet content for the following reasons:

  • It is a distraction for users.

  • Advertising images take up valuable bandwidth on the local network.

  • The wording of ads may display spurious warning messages, misleading users into believing there is a problem with their computer and generating unnecessary support calls, as described above.

  • Clicking on advertisements displayed in the browser may take users to non-business related or undesirable sites.

  • Displaying advertising graphics within a Web page allows advertisers to monitor user's surfing patterns.

Advertising content can be managed using a solution such as Connectotel AdWall, which allows the administrator to define a list of sites from which advertising content will be blocked. Such lists are available, in a number of formats, from many locations on the Internet. Advertising which would have been present in a page is then replaced with a "broken graphic" icon.

Unwanted File Types

As indicated previously, unwanted file types can increase costs through increased use of bandwidth on the LAN and Internet side of the network. An organisation may choose to block access to files by file type. For example, the network administrator may place a block on the downloading of all .MP3 music files, or deny access to all Real Audio .RAM files, which point to streaming audio content.

This type of content can be managed using a solution such as Connectotel FileWall, which allows the administrator to define which types of files will be prevented from passing via the proxy server. In the event that users try to download one of the defined file types via Novell BorderManager with FileWall, they will see a "403 Forbidden" error screen, as shown in Figure 2.

Figure 2: Screen displayed when a site is blocked via LinkWall.

Undesirable Sites

An administrator can choose to block access to undesirable sites using a solution such as Connectotel LinkWall. In this case, the administrator creates a file called LINKWALL.LST which contains a list of sites, and/or names of files containing lists of sites. For example, the list might include the following entries:

www.yahoo.com
ww.hotmail.com
$include sys:blacklists/porn/urls
$include sys:blacklists/porn/domains

Note: With LinkWall, you can reference other list files through the use of the $include command, as shown in the example LINKWALL.LST file above.

When users try to access one of the sites contained in the list, they will see a "403 Forbidden" screen similar to the one shown in Figure 2 above.

The next section provides more information on the various site lists that are available in the public domain.

Public Domain Site Blocking Lists

Public domain site blocking lists have been compiled by organisations and individuals who wish to provide a free alternative to the lists supplied by commercial companies. The lists are compiled either by an individual or small team, using public input to a website, or via a robot which has been programmed to search for certain types of sites.

Information about such a robot can be found at: http://ftp.teledanmark.no/pub/www/proxy/squidGuard/contrib/squidGuardRobot/

Using public domain site blocking lists provides the following advantages over subscription-based solutions:

  • Cost- There is no initial cost, or ongoing cost, for the use of the list.

  • Openness-The contents of the list are completely open in ASCII text format, so that anyone can see what sites are in the list. This is important when considering freedom of access to Internet resources, which may be being blocked inappropriately by a third-party list.

  • Flexibility-The system administrator may choose to use the contents of one or more lists without incurring additional cost. The administrator may change the use of a list at any time by configuring the software appropriately.

A number of public domain lists are available for use in site blocking efforts within an organisation. This section describes three of the most commonly used ones.

squidGuard Blacklist

The squidGuard Blacklist is a public-domain collection of site lists which can be used with the Squid proxy (under UNIX) and now also with Novell BorderManager. This list is generated by a software "robot" which searches the Internet looking for URLs which fit into one of squidGuard's predefined categories.

The categories of sites currently maintained within the squidGuard Blacklist are:

  • ads

  • aggressive

  • audio-video

  • drugs

  • gambling

  • hacking

  • mail

  • porn

  • proxy

  • violence

  • warez (pirated and illegally "cracked" software)

Each list contains a series of URLs or domain names in the following format:

sitename1
sitename2
sitename3
. . .

Again, these files can be referenced with LinkWall through use of the $include command, as mentioned in the previous section.

To obtain more information about squidGuard Blacklist, visit http://www.squidguard.org/blacklist.

DMOZ

The DMOZ open directory project (ODP) is a public domain project to provide an human-edited index of the Web. DMOZ allows downloading of dumps from the DMOZ ODP database. The dumps contain details of Web sites in a format known as Resources Description Format (RDF), which is described at http://dmoz.org/help/getdata.html.

The following Web page lists tools which can extract data from RDF files: http://dmoz.org/Computers/Internet/Searching/Directories/Open_Directory_ Project/Use_of_ODP_Data/Upload_Tools.

"Hosts" Files

Using a replacement "hosts" file is a commonly-used and inexpensive method of blocking advertising sites. In this approach, a file is created which contains a local host address, normally 127.0.0.1 (also known as localhost), followed by a host name. The method works by redirecting host name lookups, which would normally resolve via DNS to a real IP address, to the localhost address. The result is that advertising graphics do not then appear in the Web page. Examples of files in "hosts" format for blocking of advertising are widely available on the Internet.

By default, the "hosts" file is located in the following folders:

  • C:\WINDOWS (Windows 95/98)

  • C:\WINDOWS\SYSTEM32\ETC\HOSTS (Windows NT/2000/XP)

A good description of this method of blocking advertising can be found at http://www.accs-net.com/hosts/benefits_restrictions.html.

The disadvantage of this approach in a networked environment is that the modified hosts file must be copied to every computer. A better solution is to maintain one central copy of this file and have it shared by all users of a proxy server. Solutions such as Connectotel AdWall can read and interpret files in "hosts" format, allowing blocking of advertising via a proxy server.

Another method of blocking ads, using Novell BorderManager to redirect requests from ad servers to a Web server on your network, is described in "Blocking Browser Ads with Novell BorderManager" in the January 2003 issue of Novell AppNotes, available at http://support.novell.com/techcenter/articles/ana20030102.html.

Installing and Configuring Public-Domain Site Blocking Files

To use any of the public domain lists described above, you will need the following software:

  • NetWare 4.11 or above (the network operating system)

  • Novell BorderManager 3.5 or above (the proxy server)

  • Connectotel FileWall, AdWall, or LinkWall (the site blocking software)

  • SquidGuard Blacklist, DMOZ, or a "hosts" file (the site blocking list)

Note: Other proxy servers can be used besides BorderManager. For example, Connectotel Proxy Engine (CProxy) is a free HTTP proxy which runs on any version of NetWare 4.11 and above. It provides proxying of HTTP requests from the browser to the Web, NDS/eDirectory username and password basic authentication, logging of all Web access in HTTP common log format, and support for additional products such as Connectotel AdWall, FileWall, and LinkWall. CProxy does not currently provide higher-end services such as caching, proxying of SSL/HTTPS, or proxying of protocols other than HTTP. If you require these features, Connectotel recommends the use of Novell BorderManager.

The example which follows assumes the use of the squidGuard Blacklist with Novell BorderManager and Connectotel LinkWall. The instructions assume that NetWare and BorderManager are already installed and running on your network.

Installing the LinkWall Software

To install the LinkWall software, download the LINKWALL.EXE file from http://www.connectotel.com/linkwall.

Run the LINKWALL.EXE program and follow the prompts within the setup program. As a result, the following files will be installed on the NetWare server:


File
Location

LINKWALL.NLM

SYS:ETC\LINKWALL

LINKWALL.NCF

SYS:SYSTEM

LINKWALL.DLL

SYS:PUBLIC\WIN32\SNAPINS

LINKWALL.LST

SYS:ETC\LINKWALL

LINKWALL.ACL

SYS:ETC\BORDER\ENGLISH

LINKWALL.KEY

SYS:ETC\LINKWALL

LINKWALL.DOC

SYS:ETC\LINKWALL

Configuring the Software

To configure the Proxy Server and LinkWall software, follow these steps.

  1. Start the NetWare Administrator (NWAdmin) utility and choose the Server object on which the Proxy and LinkWall software are running.

  2. Choose the BorderManager Access Rules tab. The dialogue shown in Figure 3 will be displayed.

    Figure 3: The BorderManager Access Rules page.

  3. Choose the option to insert a rule. The dialogue shown in Figure 4 is displayed.

    Figure 4: The Access Rule Definition dialogue.

  4. Set the Action to "Deny" and Access Type to "URL".

  5. Click on the browse button in the "Destination" area and then choose "Select from Connectotel LinkWall list", as shown in Figure 5.

    Figure 5: Choosing to "Select from Connectotel LinkWall list".

  6. Having chosen that option, select the tick-box so that "Turn LinkWall On" is enabled, as shown in Figure 6.

    Figure 6: Enabling LinkWall.

  7. You should also ensure that the "Enforce Access Rules" tick-box has been selected within the BorderManager Setup tab, as shown in Figure 7.

    Figure 7: Selecting "Enforce Access Rules" on the BorderManager Setup page.

Installing the squidGuard Blacklist

To install the squidGuard Blacklist, use the following procedure:

  1. Download the file blacklists.tar.gz from the following URL:

    http://ftp.teledanmark.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz

  2. Extract the blacklists.tar.gz file (gnuzip format) to produce the blacklists.tar file.

  3. Extract the blacklists.tar file (archive format) to a folder named sys:blacklists to produce the "blacklists" folder structure.

  4. Edit the SYS:ETC\LINKWALL\LINKWALL.LST file to include the names of the groups which you want to block.

    For example, these lines will block sites containing pornographic material and material relating to hacking:

    $include sys:blacklists\porn\domains
$include sys:blacklists\porn\urls
$include sys:blacklists\hacking\domains
$include sys:blacklists\hacking\urls

Activating LinkWall

The final stage is to activate the LinkWall software on the NetWare server. To do this, type "LINKWALL <Enter>" at the file server console. This will display a screen indicating that LinkWall has been loaded and showing how many URLs have been read.

Staying Current

Any site blocking solution will only work well for as long as the site list files are kept up-to-date. In this respect, site blocking files are similar to virus signature files, which must also be kept regularly updated.

The methods below show how to maintain the squidGuard Blacklist. Similar techniques can be used for maintaining any other public-domain site list.

Manual Updates. Follow the procedure outlined above for downloading, unzipping, and untaring newer versions of the squidGuard Blacklist as they become available.

Automatic Updates. You can automate the above procedure by using server-based tools. An NCF file (NetWare command file) shown below illustrates the use of commands such as the following:

  • HTTPGET - to retrieve a file from a site using HTTP (available free from Connectotel)

  • UNCOMPR - to extract the contents of a GZ file (available free from Connectotel)

  • TAR - to extract the contents of the resulting TAR file (included with NetWare)

  • DELAY - to introduce a waiting period between commands in NCF files (included with NetWare)

The ZLIB zip/unzip library, used by UNCOMPR, is also included with NetWare.

SGB.NCF

rem
rem Load ZLIB library
rem
load zlib
rem
rem Get file
rem
load httpget ftp.teledanmark.no /pub/www/proxy/squidGuard/contrib/
blacklists.tar.gz sys:blacklists/blacklists.tar.gz
rem
load delay 30
rem
rem Uncompress ZIP file
load uncompr sys:blacklists/blacklists.tar.gz
rem
load delay 10
rem
load tar -xvf sys:blacklists/blacklists.tar
rem
load delay 10
unload linkwall
load delay 10
linkwall
echo Finished!

The above SGB.NCF file will perform the following actions in sequence:

  • Get the latest blacklists.tar.gz file.

  • Extract blacklists.tar.gz to produce blacklists.tar.

  • Extract blacklists.tar to produce the "blacklists" folder structure.

  • Unload the LinkWall software.

  • Reload the LinkWall software.

The amount of delay between each stage of this process can be adjusted, depending on the speed of server and of the Internet links being used.

Conclusion

This AppNote has shown how public domain site blocking lists can be used with Novell BorderManager, in conjunction with Connectotel's AdWall, FileWall, and LinkWall. Using these techniques, administrators can ensure that the policy, legal, and cost issues described herein can be avoided.

Additional References

For further reading, see "Understanding BorderManager HTTP Proxy Logs" in the January 2002 issue of Novell AppNotes, available at http://support.novell.com/techcenter/articles/ana20020102.html.

For more information about Novell BorderManager, see the product Web page at http://www.novell.com/products/bordermanager.

For more information about Connectotel products, see http://www.connectotel.com.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates