Bring Your Own Laptop
Articles and Tips: article
01 Apr 2005
Create Your Own Analysis Lab Using the Included Laura's Lab Kit v6 DVD!
Stand back! I'm so happy I could spit! Really! After years and years of presenting courses at BrainShare, we are finally offering a hands-on analysis lab. Certainly I could stand up in front of you and show you trace files of horrific network communications. I could razzle and dazzle you with spectacular filters that could pinpoint a piece of lint crossing port 14 on the switch twenty rooms away where the weary BrainShare IT team hides. Although that's been fun in the past, this year, my BrainShare breakout focuses on the tactile learner-the one who has to have their hands on the keyboard at all times-you know who you are!
This isn't your typical hands-on lab either. All attendees work on their own laptops that can be preconfigured with Ethereal before the session. (For more information on how to configure your laptop for this session, see Preparing Your Laptop for the BYOL: Ethereal Session.) But the best part about it is that you can run this lab by yourself even if you don't attend BrainShare just by following the instructions in this article and the others on the Laura's Lab Kit v6 (LLK6) DVD (1.25Gb) online. The trace files used in this session are also on the DVD. LLK6 also includes the remaining step-by-step instructions for the labs, follow-up lab sessions and several self-paced lab sessions as well.
With an eye on troubleshooting and security, you can follow step-by-step instructions for opening, filtering and analyzing a series of trace files. We'll be using Ethereal for this session because it is an open source analyzer available for a variety of platforms. (See the analysis instructions in Step-by-Step--Lab 1: Spyware/Adware Process to tackle this lab on your own. Refer to your LLK6 DVD for the remainder of the lab instructions and trace files.)
Lab Task List: Troubleshooting and Security
Ethereal is a great tool for capturing traffic, but it lacks any interpretive elements such as the "expert systems" included with Sniffer and EtherPeek NX. Because of this limitation, we need to wade through the packets to find the problems. The trace file names are listed after each title. These trace files are on LLK6.
Lab 1 Spyware/Adware Process (evilprogram.dmp)
It's estimated that more than 20 percent of personal computers contain spyware or adware. This is the toilet paper stuck on the shoe of your computer! In conjunction with my BrainShare session entitled Spyware, Adware and Keyloggers: Spotting and Eradicating Crappy Traffic (and Yes, I make up my own titles), this lab focuses on a host that is battling the "Stopguard" nightmare. Initially we see the host dutifully checking in with its virus detection manufacturer. So far so good. Unfortunately, we see the compromising process that is grabbing hold of the victim clearly once we create and apply an Ethereal display filter to remove the virus update traffic. In a backwards sort of way, many times we need to weed out the good traffic to see the junk left behind. (See Step-by-Step--Lab 1: Spyware/Adware Process.)
Lab 2 The "Network is Slow" (slowneterrors.dmp)
It's the dreaded "the network is slow" complaint! Users NEVER call when the network is fast, eh? In this lab we examine the traffic captured on a workstation that exhibits sluggish behavior. This lab teaches a key lesson in analysis-time is a prioritizing factor. Don't get stuck trying to remedy ugly traffic that takes 269 microseconds (that's 269 millionths of a second)! In this lab we break down the traffic in several sections containing faults and identify the culprit based on the actual time wasted. Traffic can be very deceptive and I've encountered more than one analyst who focused on the ants swarming around his feet while the elephants came charging up his backside.
Note: Instructions for Labs 2 through 6, Ethereal and the referenced trace files are on LLK6 DVD.
Lab 3 Hello! Hello! Goodbye! Goodbye! (uglytrace.dmp)
Sometimes a trace jumps out at you and slaps you across the face with its bizarre contents. In this lab, we examine a trace filled with duplicate packets. Or are they? Analysts often use port scanning to tap into a switch and copy traffic down a monitor port and there have been some problems that have caused duplicate packets to arrive at the analyzer. Is that the problem? Or do we have an impatient and overzealous application that has a nervous twitch and can't abide by the TCP timeout rules? Ahhh...the secrets of sequence and ID numbers and time-to-live fields solve this problem.
Lab 4 Go Away Kid-You Bother Me! (madclient.dmp)
When a user complains of an application lock-up (of course they are certain this is due to the network just being too slow!), your trace provides answers the application developers forgot to display. The TCP handshake and several latency calculations provide evidence that the network is innocent! And even though Ethereal doesn't recognize the application, we can look inside the packets to identify not only the name of the application and its static port number, but we can also see why the user isn't getting the service they request.
Lab 5 Just a Scan, Ma'am (justascan.dmp)
When a trace turns up evidence of a port scan, you need to determine if the offender is spoofing its IP address and MAC address. You also need to identify possible vulnerabilities that this scan uncovers and trace them back to the offender before they use them against the target. This methodical scan hides a more sinister process; however, in this lab you will learn when a scan isn't just a scan. This lab reminds us of the importance of acknowledging time values in our traces.
Lab 6 Arp! Arp! Arp! (nosmtp.dmp)
After a user complains that they can receive e-mail, but they can't send it, we examine the trace to look for evidence of a communication problem. In the trace we can see the user pinging another host and logging in to an FTP server on the network. The round-trip latency times look good. Don't blame the network this time, buddy! Further review of the trace (and a good knowledge of how TCP/IP communications work) reveals the most likely cause of the problem.
These labs point out many of the unique personalities evident in a network. They give you a solid foundation for interpreting trace files and teach you methods to filter out the good, the bad and the ugly packets. It's now time to try one of these labs out. Follow along with Step-by-Step--Lab 1: Spyware/Adware Process to open the trace file on the LLK6 and locate the method that a slimy adware program is using to remotely load processes on a target system.
Spyware/Adware Process / The beauty of including this BYOL Lab in this issue of the magazine (and on the LLK6) is that we're bringing a little bit of BrainShare to you by explaining how you can run the lab by yourself even if you don't attend BrainShare. As mentioned, all the step-by-step lab instructions, applications and related trace files can be found online.
In this lab exercise we are examining the communications of a client who complains that her system suddenly launches her browser and takes her to a Web site where she is confronted with a barrage of anti-spyware and anti-adware advertisements. (The irony begins.) Her system feels as if it is possessed (except that the monitor is not spinning around ala Linda Blair).
In this trace file, evilprogram.dmp, I have built and applied a capture filter to pick up all traffic to and from her computer's hardware address-I really don't want to be distracted by the broadcasts and multicasts that flutter about in the background. Figure 1 depicts the capture filter definition used in this trace file.
To follow along in this lab you must have Ethereal installed on your system. Ethereal version 0.10.9 is included on the LLK6 DVD along with WinPcap 3.0 (the Windows Packet Capture Driver that Ethereal requires).
Note: You may find some slight interface differences if you have an earlier or later version of Ethereal.
Step 1: Create an Ethereal Labs directory on your hard drive and copy the trace files from the LLK6 over to that directory. (You could certainly just reference these files on the DVD during the exercise, but I'm sure you want those files with you at all times-you never know when you'll get the urge to peek at some packets!)
Step 2: Launch Ethereal.
Step 3: Select File > Open on the Ethereal menu bar. Select your local drive off the drive list and double-click on the Ethereal Labs directory you created in Step 1. Double-click on the evilprogram.dmp trace file. The trace file opens and displays the three-panel view (Summary panel, Detail Decode panel and the Hex Dump panel).
Step 4: Now let's see just what happened to this poor, pathetic system. First, we see the client's DHCP request (packet #1) being sent from 0.0.0.0 to 255.255.255.255. She'll need to get that IP address before she can make any connections with other hosts. We are missing the DHCP response (a DHCP ACK packet) in this trace because of our capture filter-the DHCP ACK packet is sent from the DHCP server or DHCP relay agent to the broadcast MAC address. Our filter is only looking for traffic explicitly addressed to our host's MAC address. We must assume that our host received the DHCP ACK, however, since she begins the process of testing out the IP address for duplicate addresses on the network (packets 2, 4 and 6).
Something feels a bit funny here right away in this trace-we can see someone trying to obtain a TCP connection to Linda's system before the duplicate IP address test has even completed (packets 3 and 5). Sorry, buddy-she can't respond to you until she completes the duplicate IP address test. Why is someone knocking at her door already? On what port are they trying to obtain a connection? Aren't clients the ones who typically request connections of servers? This really doesn't feel right so let's look further into the trace.
Scroll through the trace until you reach packet 68. Check out all those connection attempts! Other hosts are banging on her door (TCP SYN packets) to obtain connections on a variety of ports. She continually responds defiantly with a series of TCP RST (reset) packets.
Step 5: Look at packets 68 and 69-this makes us feel a bit better-she performs a DNS query to obtain the IP address of McAfee's VirusScan update server. She wants to check for updates to her virus definitions. Whew! We always want to see some virus update processes take place right away when clients boot up. In packet 69 she receives the IP address of the McAfee update server (18.104.22.168).
Packets 70 through 93 is the little update server "dance" that McAfee VirusScan clients go through. She makes a TCP connection (packets 70-72) and then promptly states that she's finished (packets 73-75)-but finished with what? This little process repeats itself a total of four times-but why? Who knows? This is just one of the typical mysteries of update processes: they all have their little rituals to perform.
Scroll through to packet 93 which appears to be the end of this little dance.
Step 6: Now the fat lady's going to sing! Look at packets 94 and 95. She is now ready to get the updated virus definitions. First she must find the us.mcafee.com server. (This allows update servers to be regionalized so a host in London may do a DNS query for uk.mcafee.com instead.) The DNS server indicates that the us.mcafee.com server is located at IP address 22.214.171.124. Sound familiar?
Dutifully, she connects again to the McAfee server (packets 96-98) and asks for the news (packet 99). Scroll through the trace a bit. You see her asking for application instructions (starting at packet 105) and alerts (starting at packet 118). She posts some virus definition information on the McAfee server as well (starting at packet 128). Look inside the hex portion of packets 128 and 139 to see what she is reporting-she sends the running dll version information up to the McAfee server.
This traffic to and from the McAfee server doesn't worry us so it's time to filter it out so we can look for the unusual communications.
Step 7: We are going to build and apply a filter to remove all traffic to and from the McAfee update server at 126.96.36.199.
Click in the filter window above the trace file headings. Enter !ip.addr==188.8.131.52. The background of this window changes from red to green as you type in this filter. That is Ethereal's wonderful syntax error checking mechanism in action! (They only need this because the display filter syntax is a common cause of confusion and errors.)
Your filter should look like the filter shown in Figure 2. Click Apply. Ethereal hides all traffic to and from 184.108.40.206. The packets are still in the trace, but they are not in our way anymore. If you want to see them again, simply click Clear.
Step 8: Oh, look! Packets 147 and 148 indicate that we've got someone trying to send Linda a message using Microsoft's Messenger Service. Because we disable that ugly service the moment we unbox any new computer, Linda responds with an ICMP Destination Unreachable/Port Unreachable message-get lost, buddy! In the hex window you can read the message coming along:
DANGEROUS SPYWARE VIRUS FOUND
DELETE THIS SPYWARE VIRUS IMMEDIATELY
You can read the entire popup message in packet 147. What a scam-I'm surprised they didn't say to delete command.com to clean it up! No one should have Messenger popups assaulting them through the day.
Step 9: Scroll through from packet 208 to 349. Ugly, ugly, ugly. Look at all those folks trying to connect to Linda's system. These folks are poking around on various ports and she consistently sends back TCP RST packets to refuse the connections. Good girl!
Can we filter these packets out? Hmmm...let's see. These connection attempts are coming from a variety of IP addresses. They use a variety of source and destination port numbers as well. We could filter these all out, but it would take more time than we want to spend on it. Why don't we find out a bit more about these connections.
Step 10: Click on Statistics > Conversations. We can see there are three Ethernet connections (mostly between Linda and the router, Cadent_22:89:c2). Click on the TCP tab in the Conversations window. This tab lists the total number of TCP connections and connection attempts in the trace as 90 (TCP:90). Yipes! She's just booting up! Click on the Packets column heading to sort it in ascending order, as shown in Figure 3.
Now you probably know that successful TCP connections require at least three packets (the three-way handshake):
client to server (SYN)
server to client (SYN ACK)
client to server (ACK)
How could we have some TCP connections listed with one and two packets? If we hopped onto the network for a quick trace, we may have just caught the end of a TCP connection or the final packets in a connection teardown. In this case, however, we are seeing many successful TCP connections. The TCP conversations listed with only one packet can be seen at the beginning of the trace (packets 3 and 5). The TCP conversations listed with two packets are also unsuccessful because our host simply sent back RSTs in response to the initial SYN packets. In this window we do see numerous different IP addresses TCP port numbers as the source. Interestingly, you can also see that some folks tried the handshake process more than once-they are listed with four or six packets in the connection process (two or three sets of SYN and RST).
Right mouse click on a conversation to quickly build and apply a display filter based on the source and destination IP addresses and port numbers.
Note: This will wipe out your previous filter defined in Step 7. After you play around with the filtering capabilities from within the Conversation window, you might want to clear out any filters applied and perform Step 7 again.
Click the Close button to close down the Conversations window when you are done with it.
Step 11: Now let's go back to packet 350. Uh oh! We have someone making a connection to our client on port 1025. Typically in our client-server environments, our network users' systems are not the servers; they are the ones sending the TCP SYN packets up to the Web and file servers. We become suspicious because we aren't familiar with any application that this client should be running on port 1025.
Look closely at packets 357-359. We have a DCE (Distributed Computing Environment) RPC (Remote Procedure Call) session underway. DCERPC enables a system to call a procedure on a remote host. A DCE host daemon (dced) must be running on the target in order for a session to be established from a remote machine. This is not a good thing. No one should be running processes on this host from a remote location.
Inside the hex dump of packet 359 we find the word MEOW twice-probably not a coincidence. We wonder what this process is telling the client to do. We don't have to wait long to find out.
Step 12: The next conversation our client initiates is a DNS lookup for updates.virtumonde.com. We don't recognize this domain name. (It doesn't sound like any software package installed on the client that we believe might want updating.) It's time to do a bit of research. A quick Google search of Virtumonde finds that it is a nasty adware program that supports pop-up ads and browser hijacking. Bingo! In packet 382, the DNS server provided our client with two IP addressees-220.127.116.11 and 18.104.22.168. In packets 383-385 our client makes a connection to the update.virtumonde.com server.
Step 13: Let's filter out traffic to and from this server to see what our client does. In the filter window, type in ip.addr==22.214.171.124 and click Apply. Yipes! Look at packet 386! Our client is downloading an executable file called bkinst.exe over this strange connection. This is making my stomach turn. We can see that we are downloading a Windows application by peering into the hex portion of packet 388 as shown in Figure 4. We see the recognizable "This program cannot be run in DOS mode" message in the program.
Step 14: Click Clear to remove our filter so we can see what is happening next in the trace. In packet 400 the client performs another DNS query-this time she is looking for Virtumonde.com. She finds and makes an HTTP connection to 209.123.150. Instead of downloading something from this second server, however, she sends something up to it using an HTTP POST command (packet 405). In response, the Virtumonde server sends our clients some new configuration information in packet 412:
AdCategoy: Some ConnectionPerDay: 2 PopUpPerDay: 24 SetIDWas: Unreleased StatisticsUploadDelay: 1 StealFocus: a (perhaps "active?")
We can only guess what the purpose of some of the line items are because these guys don't make their code and commands known to the general public. These spyware/adware applications are the scum hanging off the shoe of the network-really!
Step 15: Scroll through the rest of the trace to look for other suspect communications.
You can learn how these spyware/adware packages operate by looking at the trace file of an affected client. Sometimes you can even figure out techniques to thwart them throughout the enterprise (such as blocking specific DNS lookups or connections to specific IP addresses). If you'd like to be walked through this process, check out the Self-Paced Labs section of LLKv6.
Check Out LLK6 (Included with this Issue)
This BYOL enables you to see and perform additional analysis sessions like the one featured in this article. On the LLK6, I'll guide you through the step-by-step interpretation process which can often be the most daunting task for a protocol analyst. Check out each of these labs on the LLK6. I've also included AVI files that demonstrate each lab. You can do the labs yourself or just follow along as I do the labs for you. Simply click on the "Self-Paced Labs" button on the main LLK6 window.
For additional self-paced labs and trace file interpretations, check out the Interpreting Trace Files section of "Laura Chappell's Master Library," a compilation of CDs and DVDs containing network and security analysis resources, training and trace files that is available online at www.packet-level.com. The Master Library includes more than 50 packet-by-packet descriptions of traffic including DHCP, HTTP, attacks, Voice Over IP (VoIP) and 802.11 communications.
Inside Laura's Lab Kit v6 (LLK6)
Amazing! This is our sixth year producing the Laura's Lab Kit. This year's version carries the theme of Russia and IMHO, it's our most interesting and comprehensive resource for tools, training and trace files. The following lists some of the tools and training included on it:
Achilles: a Man-in-the-Middle and proxy Web server
Ad-aware: spyware/adware detection and removal
AIM Sniffer: (demo) AOL Instant Messenger session recorder and viewer
AiroPeek NX: (demo) wireless protocol analyzer
Cain & Abel: full version of this software billed as a "password recovery tool" (but it does much more)
Camtasia: (demo) a graphic/audio capture tool used to build the self-paced labs
Captain Nemo (See Figure 6): (demo) multiplatform file manager (Novell, Linux and Windows file systems)
Davory: (demo) file recovery system
Ethereal: full version (0.10.9) open source protocol analyzer
EtherPeek NX: (demo) Ethernet protocol analyzer
Everest: full version of this system auditing tool (formerly Aida32)
Forensic Toolkit: fully-functional dongle-less forensic tool version
Forensic Toolkit Imager: (demo) disk imager
HexWorkshop: (demo) one of the best hex editors around
HTTP Sniffer: (demo) HTTP session recorder and viewer
Hurricane Search: (demo) project-based forensic search tool
Invisible Secrets: (demo) my favorite steganography tool*
MSN Sniffer: (demo) MSN session recorder and viewer
NetScanTools Pro: (demo) "must-have" multifunction tool
Packet Builder: full version of this traffic creator and generator tool (complements Ethereal)
ProDiscover for Windows: (demo) forensic investigation tool for Windows
Registry Viewer: (demo) product for registry forensics
RFC Viewer: (demo) automatic indexer for RFCs (great!)
SnagIt: (demo) best screen capture utility around
Sniffer Portable: (demo) high-end protocol analyzer (includes an expert system)
Specter: (demo) nice, clean honeypot for Windows
TCPView: full version of this TCP connection and UDP endpoint enumerator
Trace (X-Ways): (demo) forensic tool that recycles hidden Explorer cache and Recycle Bin files
WinHex: (demo) forensic tool to examine hidden partitions, RAM contents and more
WinPcap: full version (3.0) of the Windows packet capture driver from winpcap.polito.it
LLK6 Training and Resources**
Self-paced course: TCP/IP Troubleshooting (includes IP, UDP, TCP, ICMP and ARP)
Self-paced course: Introduction to Wireless LANs (includes antenna fundamentals,
decibel math, site survey, troubleshooting and WLAN security)
Certified Wireless Network Administrator (CWNA) Official Study Guide
Interpreting Trace Files Lab: Just a Scan, Ma'am
Interpreting Trace Files Lab : ARP! ARP! ARP!
Interpreting Trace Files Lab : Go Away Kid-You Bother Me!
Interpreting Trace Files Lab : Hello! Hello! Goodbye! Goodbye!
Interpreting Trace Files Lab : The "Network is Slow"
Interpreting Trace Files Lab : Spyware/Adware Process
LLK6 Resources List
IP Protocol Poster (2' x 3' layout in PDF format)
ICMP Protocol Poster (2' x 3' layout in PDF format)
TCP/IP Protocol Stack Reference Card
Linux Reference Card
Laura's Ethereal Filter Set (capture and display filters)
ASCII & Keycaps Chart Reference Card
LLK6 Trace File List
DHCP bootup, relay process
127.0.0.1 traffic on the wire
Connection-oriented application setup, fault
ARP: gratuitous and address lookup processes
DNS: slow and successful
Ethernet frame types
Man-in-the-Middle poisoning process
FTP connections, failed passwords, faults, get and put files
ICMP convert channel communications
Ping and traceroute with unique tool signatures
OS fingerprinting and vulnerability scanning
SMTP relay tests and faults
TCP connections, faults, scans, echoes and attacks
* Following the tradition of the previous five lab kits, LLK6 contains several stego-images. Use Invisible Secrets to unhide the secrets. Password: hide; encryption method: blowfish.
** Some of the LLK6 contents are locked. Register your Laura's Lab Kit online for free at www.packet-level.com/registerllk to receive the free unlock key.
In addition, there are more than 50 articles on analysis, troubleshooting and security. Enjoy!
Did You Know?
Here are some little-known facts about Laura's Lab Kit over the years:
Each version of LLK has had a secret embedded in an image using Invisible Secrets; some versions have numerous hidden secrets.
More than 200 people successfully finished the Treasure Hunt in Version 4.
Only 87 people successfully finished the Egg Hunt in Version 5.
The number 19 has special significance. This number might be hidden in images and directories.
All covers (except one) spotlight the last place Laura was visiting before production of the DVD. Can you name the five locations used in LLK2 through LLK6?
Laura's default steganography password is "hide" (all lower case, no quotes).
Laura's default encryption method is blowfish which was written by Bruce Schneier of Counterpane.
In the Scotland version, the bagpiper is at Edinburgh castle.
The koala shown in the Australia version is at Beerwah Zoo (Steve Irwin's zoo).
The shark IS a dead-end in the LLKv5 Egg Hunt. But the picture on the wall in the vet's office links to Laura's humorous PPT collection online.
* Originally published in Novell Connection Magazine
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.