Novell is now a part of Micro Focus

Using WebSpy Analyzer and WebSpy Live with Novell BorderManager Log Files

Articles and Tips: article

Meena Guruprasad
Senior Software Engineer
Novell, Inc.
mguttikar@novell.com

Thanks to Julia Andrys of WebSpy for her assistance with this AppNote.

01 Mar 2003


This AppNote provides information on how to use WebSpy's Analyzer Standard and WebSpy Live to simplify the process of working with Novell BorderManager HTTP proxy log files. It explains the use of the tools to analyze and extract meaningful data from these log files.


Topics

Novell BorderManager log files, Internet usage monitoring tools

Products

Novell BorderManager 3.x, WebSpy Analyzer Standard 3.0, WebSpy Live 1.0

Audience

network administrators

Level

intermediate

Prerequisite Skills

familiarity with Novell BorderManager

Operating System

NetWare 4.x and higher

Tools

none

Sample Code

no

Introduction

The BorderManager HTTP Proxy log files can be difficult to read and understand. Although the files are in text format, they are not well formatted. Another drawback is that the log files provide only historical data, not live data in real time. The NetWare Administrator (NWADMN32) utility does not export the common and extended log files of BorderManager.

For better understanding, the log files should be read through an application or through a program which provides mechanisms to understand the information in a more meaningful manner.

This AppNote describes two tools to analyze and extract meaningful data from the BorderManager log files. They are WebSpy Analyzer Standard and WebSpy Live.

Types of HTTP Log Files

Novell BorderManager has three types of HTTP log files:

  • Common

  • Extended

  • Indexed

The WebSpy tools act on the common and extended log formats of the BorderManager proxy. The common and extended logs files are in text format and are placed in the /etc/proxy/log/http/common and /etc/proxy/log/http/extended folders on the server.

BorderManager log files provide the following functionality:

  • Determining the patterns of bandwidth utilization

  • Identifying attempts to access undesirable resources

  • Monitoring of Internet usage

  • Providing insight into an individual user's mode of Internet usage

  • Specifying the IP address from which the Internet is being accessed

  • Reporting of file size and many other details either by site or by authenticated user

Just as with the common log files, extended log files also consist of fields which record outbound access to the Web through the proxy. These files are created only if you check the common and/or extended logging checkboxes for the HTTP proxy in NWADMN32. The log files are created with names in the following style: YYMMDD-A.

Note: More complete information on the log files can be found in "Understanding Novell BorderManager's HTTP Proxy Logs" by Marcus Williamson (published in the January 2002 issue of Novell AppNotes at http://support.novell.com/techcenter/articles/ana20020102.html) and in the electronic book A Beginner's Guide to BorderManager 3.x by Craig S. Johnson (available for purchase at http://www.caledonia.net/bmadmin.html).

WebSpy Analyzer Standard

WebSpy Analyzer Standard 3.0 is a tool for Internet usage analysis and reporting. It enables you to create comprehensive and customizable reports based on Internet usage in an organization by analyzing data from the information contained in the common and extended log formats of Novell BorderManager. It is a highly configurable tool that allows you to load data from proxy server log files.

Analyzer Standard helps you achieve common business and IT goals by:

  • Improving employee productivity and morale

  • Increasing bandwidth efficiency

  • Reducing security threats

  • Reducing legal liability

Analyzer Standard has retained all the features that have made it one of the most popular log analysis tools available. These include:

  • The ability to analyze data on all your Internet usage in one package

  • Custom report generation to Microsoft Word or HTML so you get the exact information you require

  • Added ability to create powerful and comprehensive three-tier reports

  • Customizable profiles for creating profiles to suit your organization

  • Drill down capability for in-depth interrogation of data

  • Find functionality in both Summaries and Aliases to quickly and easily find specific data

  • Right-click to add a user directly to a Department in Summaries

  • Task scheduling for "set and forget" data analysis

  • Simple installation and setup

  • Intuitive user interface with a Web-like look and feel

  • Works with existing network resources such as Windows NT User Groups

  • Smooth integration with your current network

Analyzer Standard 3.0 has been developed with many new features and improvements on the previous version (Analyzer 2.1). These include:

  • New user interface, which includes task pads for easy use and navigation of the application

  • Improved Import Wizard with enhanced time filtering on imported data

  • New Excel (.CSV) report format, for exporting reports to a spreadsheet

  • Import/Export of .CSV files for aliases and profiles, enabling more efficient editing of large files

  • New HTML (.WSR) report format, collating all HTML files into one archive file - HTML files can be extracted from WSR files for publishing reports on a Web page or intranet

  • New improved HTML help, providing detailed information and assistance with all Analyzer Standard functionality

  • Improved Report Wizard and generated reports. When generating a report, you can e-mail it, view it, prefix it with the date (to help ensure other files don't get overwritten), and copy the report to a location. If the report is a HTML report, you can extract the HTML files from the .WSR file to a specified location.

  • Improved Scheduled Tasks, due to improvements made in importing and report generation

  • Faster Microsoft Word reports, with improved formatting

For in-depth information on how to use WebSpy Analyzer Standard with BorderManager proxy logs, refer to http://www.webspy.com.

Installing WebSpy Analyzer Standard 3.0

To install WebSpy Analyzer Standard 3.0, double-click the file you downloaded, or insert your CD. If you are using an evaluation version of Analyzer Standard, you will be prompted for a password to commence the installation. In this case you would enter the password "monitoring".

Complete the steps in the Install Shield to install Analyzer Standard onto your computer.

Getting Started with WebSpy Analyzer

Once you have installed Analyzer Standard, complete the following steps to load your log files into the application and commence your data analysis.

  1. To import your log files, go to "Import log files" from the management task pad on the Inputs screen (see Figure 1).

    Figure 1: The WebSpy Analyzer Inputs screen.

  2. Click the "Import log files" link to launch the Import Wizard (see Figure 2). This wizard guides you through the process of choosing from where to import the log files, which files to import, and what data is required from the log files. Click Next to start using the wizard.

    Figure 2: The Import Wizard welcome screen.

  3. To add files to import, go to the Log files page (see Figure 3) and click the "Add files" option.

    Figure 3: The Log Files screen.

  4. From the Open dialog box that is displayed (see Figure 4), select the file you want to import.

    Figure 4: Selecting a log file to import.

    To import an extended log file, select the extended log file from the Open dialog box. Click Next to continue.

  5. In the Detection Options screen (see Figure 5), choose the first option to detect the format of all the log files, then click Next to proceed.

    Figure 5: Selecting the format of log files to detect.

  6. Analyzer Standard will attempt to detect the format of the log files according to your selection. If the detection does not work, you must manually specify the log file format using the drop-down list (see Figure 6). For common log files, the format to select is "Novell Bordermanager." For the extended log file format, select "Novell Bordermanager Extended" from the drop-down list. Click Next to continue.

    Figure 6: Manually selecting the log file format.

  7. In the next screen (see Figure 7), you choose what type of filtering you want done on the data that is imported.

    Figure 7: The Filter screen.

    By default, all data (excluding failed hits) will be imported. You can select to include failed hits if you want. This screen also asks if you want to further filter the data. If you check "Yes," you can filter by Date and Time, Protocol (Web or Mail), Departments, and Profiles (Adult, Shopping, and so on). Click Next to continue.

  8. On completing the Import Wizard, you will see a summary screen similar to the one shown in Figure 8. Click Finish to start importing the data.

    Figure 8: Completing the Import Wizard screen.

Other Important Features of WebSpy Analyzer Standard 3.0

This section describes some other important features of WebSpy Analyzer Standard 3.0 that you may want to investigate.

Data Drill Down. WebSpy Analyzer Standard will analyze your loaded log data files and display the results on the Summaries Overview. From the Overview, you can drill down to different summary pages displaying your data broken down into different categories and levels of detail.

The data is organized into the following categories:

  • Departments - separate groups made up of your organization's members

  • Users - the people using your organization's Internet resources

  • Profiles - distinct categories of types of Web pages (for example, news

  • Sites - the sites that were accessed

  • Files - the file types that were accessed

  • Protocols - the protocols that were accessed

  • Months - the month of the year

  • Weeks - the calendar week number of the year

  • Dates - the date the resource was accessed

  • Days - the day of the week

  • Hours - the hour of the day that the resource was accessed

Figure 9 shows an example of a Summaries overview report.

Figure 9: A Summaries overview report.

Customizable Reports. Analyzer Standard can create reports as either Web documents (HTML or Web pages) or as Microsoft Word documents. Further, the Web documents can be created with frames that provide a table of contents on the left, or as a flat document that you scroll through. These reports can then be distributed to relevant parties either via e-mail or by an automated Web publishing process.

There are a number of available report types:

  • The Comprehensive Organization Report contains an overview of your organization's Internet activity, summaries for each category of data, and overviews of the top items in each category.

  • The Multilevel Comparison Report is a comparative statistical analysis between any two number of categories.

  • The User Sessions Report enables you to assess how long your users spent actively using the Internet.

  • The Custom Report Template allows you to configure your own report.

  • The Productivity Report gives you the ability to input your own data on costs and timings to assess how productively the Internet is being used.

The partial report shown in Figure 10 contains an overview of an organization's Internet usage. This includes a summary for the top five items in each category and an overview of the most active departments and users. The categories are: users, profiles, sites, files, months, weeks, dates, days, and hours.

Figure 10: Example of a Web-based Organization Report.

Figure 11 shows another section of this report. (The report is actually a single Web page, but it has been broken up into several screenshots for illustration.) This section shows an overview of the organization's Internet usage, giving details on the total number of hits, total size downloaded, number of active users, number of sites accessed, number of active days, the active department, most active user, and so on.

Figure 11: Organization section of the Web-based report.

Figure 12 shows another section of the report. This one displays the top five users of the Internet in the organization.

Figure 12: Top 5 Users section of the report.

For more information about these reports, refer to the documentation for Analyzer Standard at http://www.webspy.com.

Profiles. Profiles are useful to help determine what kinds of Web pages are being accessed. You may find it useful to configure your own Profiles to reflect the different browsing patterns of different departments in your organization. Additionally, you can use the "My Organization" profile to build up a list of approved or acceptable sites, and remove any other profiles.

Profiles are collections of keywords that are checked against the site name and resource name of a hit to see if they match. There are two types of keywords: included and excluded. If a site or resource name contains an included keyword of a profile, it will be placed in that profile. If a site or resource name contains an excluded keyword, it will not be placed in that profile even if the site or resource name contains an included keyword.

You can add, delete, or edit profiles and keywords to suit your organization's Internet usage patterns.

The Miscellaneous profile contains all the hits that have not been assigned to one of the other profiles.

Aliases. Aliases are used during log file importing to translate names or IP addresses into a more useful form. This means that a user name will be used instead of the IP address, a Web site's name instead of the IP address, and a type of file rather than the file extension.

Scheduled Tasks. The Scheduled Tasks feature enables you to create actions for Analyzer Standard to perform at a time specified by you. With tasks, you can instruct Analyzer Standard to import log files for you and create reports.

Each task is made up of actions, of which there are three possible types:

  • Import actions instruct Analyzer Standard which log files to import.

  • Report actions tell Analyzer Standard what type of report to create.

  • Schedule actions tell Analyzer Standard when to perform the tasks.

You can have multiple import and report actions. While it is possible to have more than one schedule action, this is not recommended except for advanced users.

WebSpy Live

WebSpy Live 1.0 monitors your proxy server's current log files to provide you with a real-time picture of what the people using your network are doing. It enables organizations to access data on Internet use by triggering customized alerts based on content, downloaded file size, or browsing time. Using this tool, you can monitor how much time users are spending on the Web and see where they are going instantly. This makes it easier for you to give immediate attention to problem areas and promote a proactive response to help eliminate Internet misuse in school or workplace environments.

With WebSpy Live you can do the following:

  • See who is browsing the Web at any moment

  • Be notified as soon as unacceptable browsing occurs

  • Identify any users who spend too much time browsing, while they are still browsing

WebSpy Live is the perfect complement to Analyzer Standard, and it works with both the common and extended log files created by BorderManager.

Key Features of WebSpy Live

Here are brief descriptions of some of the key features of WebSpy Live:

  • Customizable Triggers. WebSpy Live allows users to customize the parameters and browsing limitations of users via triggers based on profiles, size, file extension, and length of time spent browsing. When an alert is triggered, the administrator receives the information via a pop-up dialog, similar to an instant messaging system.

  • User Session Summaries. Enable users to access data on the day's activity. Broken down by user or department, session summaries allow a quick assessment of productivity and browsing.

  • Active/Inactive User Lists. View active and inactive users in real time to establish traffic patterns and decipher data on large networks, where multiple users might access a single workstation.

The real-time data provided by WebSpy Live has an advantage over other tools such as WebTrends and BRDSTATS because they only provide analysis on historical data.

As in Analyzer Standard, Aliases transform cryptic network names or IP addresses into more useful names. WebSpy Live also allows you to filter out traffic that you are not interested in, enabling you to monitor only what you want to.

WebSpy Live uses the same data input, aliases, and profiling as WebSpy Analyzer Standard. For further information on how to use the program with BorderManager logs, refer to http://www.webspy.com.

Components of WebSpy Live

WebSpy Live has three main components, as described below.

Live Status. This is an unobtrusive list where you can see which of your users are currently browsing the Internet and which alerts have been generated (see Figure 13). Different color icons indicate the priority of the alert. You can select any alert or user to see a summary of the behavior that generated the alert.

Figure 13: The WebSpy Live Status window.

This resizable small window can be kept open on your desktop to provide a quick reference to any incoming alerts. If you need to, you can minimize the window to a system tray icon.

Live Summary. The Summary window provides details about each user's session activity. It shows which sites are currently being visited, if the user is idle, the number of sessions, and the time and sites visited during each session. You can browse the sites to check if they comply with organizational Internet acceptable usage policies.

Live Configuration. This component allows you to manage your trigger settings, locate your log files, customize profiles (Web site categorization keywords), resolve user aliases, and organize your company into departments from an Outlook-style interface.

Inputting Your BorderManager Logs

To add an input, perform the following steps.

  1. Open Live Configuration by right-clicking the Live icon in your computer's system tray (next to the clock) and selecting Configuration from the menu.

  2. Open Inputs by clicking on the Inputs sidebar icon or by selecting Views > Inputs from the main menu.

  3. Click the Add button to launch the Input Wizard (see Figure 14).

    Figure 14: The Input Wizard welcome screen.

    The Input Wizard will guide you through the process of choosing the location and format of the log files you want to monitor. Click Next to continue.

  4. In the Folder screen (see Figure 15), specify the location log file you want to monitor.

    Figure 15: The Folder selection screen.

    The File mask option allows you to select a particular file extension, or you can choose a wildcard. For the Formats option, select "Novell Bordermanager" for common HTTP log files or "Novell Bordermanager Extended" for extended log files. Click Next to continue.

  5. Next you see the Advanced Settings screen (see Figure 16) where you can customize settings to improve monitoring in certain circumstances. The defaults are suitable for most environments. Click Next to continue.

    Figure 16: The Advanced Settings screen.

  6. In the Filter screen (see Figure 17), you can choose to do filtering on the logged data as it is imported. Once you've made any desired settings, click Next to finish.

    Figure 17: The Filter screen.

Using the Trigger Wizard

WebSpy Live produces an alert when someone browses the Internet inappropriately. You can specify inappropriate browsing conditions in a trigger. When a trigger is breached, WebSpy Live will display the alert in the Status List. For example, Figure 18 shows alerts being displayed for "Excessive Size (>10 MB)", "Inappropriate Content", and "Personal Browsing."

The Trigger Wizard enables you to set up specific triggers based on:

  • Profile

  • Size

  • File extension

  • Time

All active alerts and users are displayed in Live Status. You can double-click on any alert to see the details of the activity that triggered the alert in a Details dialog. Then you can choose to act on that alert by sending an e-mail, or to dismiss the alert. To dismiss an alert from Live Status, right-click on the alert and select "Dismiss" from the pop-up menu.

From the Details dialog, you can also open any listed sites in your default Internet browser by right-clicking on the site's name and selecting "Browse to:" from the pop-up menu. You can even reassign the site to a different profile, if you want.

Alerts displayed in the Live Status window.

If you want to contact a user about their browsing, you can click the Email button on the Details dialog, and WebSpy Live will launch a new e-mail containing the activity details displayed in the Details dialog. You can edit the e-mail if necessary before sending it to the user.

Figure 19 shows an example of the information sent for user Admin.

Figure 19: Default information to be sent to user Admin.

Figure 20 shows an example of the type of information that would be sent by default to a user named Meena.

Figure 20: Default information to be sent to user Meena.

Improving WebSpy Live's Performance

To improve the performance of WebSpy Live, you should keep only your most recent log files in the folders being monitored. If you archive the older logs in another folder so that WebSpy Live only monitors the most recent log file, you will find that WebSpy Live works faster and more efficiently.

You can also modify options on the Advanced Settings page of the Input Wizard to minimize CPU usage. You will need to do this on a per-input basis. You can use aliases, departments, and profiles from WebSpy Analyzer 2.0, and profiles from WebSpy Analyzer 1.0.

There are lots of options you can specify to make Live easier for you to use. For example, you can play sounds whenever an alert is raised, automatically start Live whenever your computer starts, and keep Live Status on top of any open windows. You can change any of these settings from Live Options, which you can access from Live Configuration, Live Summary, and Live Status, or from the Live system tray icon.

Comparison of WebSpy and Other Analysis Tools

Other Internet usage analysis tools, such as BRDSTATS and WebTrends, are also available for use with BorderManager log files. The historical analysis that these tools provide is similar to that of WebSpy Analyzer Standard. They differ in the ways they can be customized to view the extracted data. Apart from the data WebTrends generates, it provides the administrator with data or activity per month, days, dates, week, and hours. BRDSTATS gives information on Internet usage in a plain text file. The user interface and additional features like customizable data drill down appear more user friendly in WebSpy Analyzer Standard.

WebSpy Live is unique in its ability to provide information on current Internet usage in real time. This, along with its ability to send alerts when a trigger is activated, make it an excellent tool for monitoring Internet usage in any organization.

Conclusion

Given that the Novell BorderManager log formats are not easy to read, the WebSpy programs discussed in this AppNote provide a means to extract meaningful data from the logs. WebSpy Analyzer and WebSpy Live give the administrator the facility to monitor the Internet usage of any organization through the proxy log files.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates