Opening Large Capture Files with WireShark (aka Ethereal)

Sometimes it might be needed to open ”’big”’ network traffic capture files (a.k.a. .cap) . This challenge is commonly encountered since this sort of logging it is usually very verbose. Leaving a packet sniffer (such as tcpdump) overnight logging all the packets that go through a network interface card might generate several gigs of data.

WireShark (formerly known as Ethereal) is an excellent open source packet sniffer with a nice user interface (GUI) and available for many different platforms.

According to WireShark’s documentation:

Wireshark uses memory to store packet meta data (e.g. conversation and fragmentation related data) and to display this info on the screen.

How much memory actually used is depending on:

  • the number of packets captured (well, depending on the capture duration and the network load on the line)
  • the kind of packets captured (small/large packets, some packet types will lead to much more memory usage than others)
  • the preference settings, e.g. the “TCP desegmentation” setting

In my experience (and with the capture files and Preference settings I’m usually working with), I need about ten times of memory than the actual capture file size. But again, this will largely depend on the things noted above.

Fortunately, WireShark includes a easy-to-use tool to manipulate capture files: editcap.

Editcap is able to do things like processing a huge capture file and extract the packet information that correspond to a certain period of time. This functionality is perfect if, let’s say, you have a 24 hours capture file and you know that problems were reported at a certain time. You could then extract a bunch of packets around that time, export this into a new file and open it using WireShark.

Editcap Usage

Usage: editcap [options] ...   [ [-] ... ]

A single packet or a range of packets can be selected.

Packet selection:
  -r                     keep the selected packets, default is to delete them
  -A         don't output packets whose timestamp is before the
                         given time (format as YYYY-MM-DD hh:mm:ss)
  -B          don't output packets whose timestamp is after the
                         given time (format as YYYY-MM-DD hh:mm:ss)
  -d                     remove duplicate packets

Packet manipulation:
  -s            truncate each packet to max.  bytes of data
  -C            chop each packet at the end by  bytes
  -t    adjust the timestamp of each packet,
                          is in relative seconds (e.g. -0.5)
  -E  set the probability (between 0.0 and 1.0 incl.)
                         that a particular packet byte will be randomly changed

Output File(s):
  -c   split the packet output to different files,
                         with a maximum of  each
  -F       set the output file type, default is libpcap
                         an empty "-F" option will list the file types
  -T         set the output file encapsulation type,
                         default is the same as the input file
                         an empty "-T" option will list the encapsulation types

Miscellaneous:
  -h                     display this help and exit
  -v                     verbose output

Editcap Example
This example extracts all the packet information corresponding to 4th July 2008, 6am to 7am and writes it into a new file, i.e.:

  • Source file: 2008-07-04_capture.cap (813 MB)
  • Destination file: 6to7.cap (48.5 MB)
editcap -A "2008-07-04 06:00:00" -B "2008-07-04 07:00:00" 2008-07-03_capture.cap 6to7.cap

Showing MySQL Permissions

Usage:

mysql> show grants for 'USERNAME'@'SERVER';

Example:

mysql> show grants for 'root'@'localhost';

+---------------------------------------------------------------------------------------------------------------+
| Grants for root@localhost                                                                                     |
+---------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY PASSWORD '2e71fce803407b4a' WITH GRANT OPTION |
+---------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

See Also

Manipulating VMWare Virtual Disks with Virtual Disk Manager

From Using VMware Virtual Disk Manager:

VMware Virtual Disk Manager is a utility in VMware Server that lets you create, manage, and modify virtual disk files from the command line or within scripts.

One key feature is the ability to enlarge a virtual disk so its maximum capacity is larger than it was when you created it. This way, if you find you need more disk space on a given virtual disk, but do not want to add another virtual disk or use ghosting software to transfer the data on a virtual disk to a larger virtual disk, you can simply change the maximum size of the disk. This is something you cannot do with physical hard drives.

Another feature allows you to change whether or not all virtual disk space is preallocated or growable, and whether or not the virtual disk is stored in a single file or split into 2GB files. For example, you might find that you preallocated all the disk space for a virtual disk, but need to reclaim some hard disk space on the host. You can convert the preallocated virtual disk into a growable disk and remove the original virtual disk file. The new virtual disk is large enough to contain all the data on the original virtual disk. The virtual disk grows in size as you add data to it, as if you never preallocated the disk space when you created the virtual disk.
You can use the virtual disk manager to:

  • Automate the management of virtual disks with scripts.
  • Create virtual disks that are not associated with a particular virtual machine, to be used for templates, for example.
  • Switch the virtual disk type from preallocated to growable, or vice versa. When changing the disk type to growable, some space on the virtual disk is reclaimed. You can shrink the virtual disk to reclaim even more disk space.
  • Expand the size of a virtual disk so it is larger than the size specified when you created it.
  • Defragment virtual disks.
  • Prepare and shrink virtual disks without powering on the virtual machine (Windows hosts only).
  • Rename and move virtual disks.

You cannot use the virtual disk manager to create physical (raw) disks. You cannot shrink physical disks at all.

See Also

References

HAProxy hot-reconfiguration

As of version 1.2.8, a new soft-reconfiguration mechanism has been introduced.
It is now possible to “pause” all the proxies by sending a SIGTTOU signal to
the processes. This will disable the listening socket without breaking existing
connections. After that, sending a SIGTTIN signal to those processes enables
the listening sockets again. This is very useful to try to load a new
configuration or even a new version of haproxy without breaking existing
connections. If the load succeeds, then simply send a SIGUSR1 which will make
the previous proxies exit immediately once their sessions are closed ; and if
the load fails, then simply send a SIGTTIN to restore the service immediately.
Please note that the ‘grace’ parameter is ignored for SIGTTOU, as well as for
SIGUSR1 when the process was in the pause mode. Please also note that it would
be useful to save the pidfile before starting a new instance.

The ‘-st’ and ‘-sf’ command line options are used to inform previously running
processes that a configuration is being reloaded. They will receive the SIGTTOU
signal to ask them to temporarily stop listening to the ports so that the new
process can grab them. If anything wrong happens, the new process will send
them a SIGTTIN to tell them to re-listen to the ports and continue their normal
work. Otherwise, it will either ask them to finish (-sf) their work then softly
exit, or immediately terminate (-st), breaking existing sessions. Citation.

Procedure

The command to be issued to restart HAProxy gracefully would be:

haproxy -f configfile -sf

Example (added the PID location):

haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

References

Fixing VMWare vmxnet driver networking issues under Debian Linux

It seems that using particular combinations of VMWare Server and Linux Kernel version(s) while installing VMWare Tools under Linux guest machines, may render the virtual machine’s networking down.

This page provides a “hacky” workaround to solve this situation. There might be other deeper and more proper solutions out there but I came up with this one because it is very simple to put in place and not harmful at all.

So, let’s imagine this scenario:

  • VMWare Server 1.0.2 (other versions might apply as well, not tested thought)
  • Debian Linux 4.0 Etch guest, running 2.6.18-6-686 kernel (other versions might apply as well, not tested thought)
  • VMWare Tools 1.0.2-39867 (other versions might apply as well, not tested thought)

Then, after the VMWare Tools get installed, the screen shows something like this:

The configuration of VMware Tools 1.0.2 build-39867 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an X session.

To use the vmxnet driver, restart networking using the following commands:
/etc/init.d/networking stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/networking start

Enjoy,

--the VMware team

Right now the networking does not work. If you try to see what is going on, you should see something like this:

ifconfig

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1556 (1.5 KiB)  TX bytes:1556 (1.5 KiB)

That’s it. No network interfaces. If you go for VMWare Tools’ installer suggested steps, the networking should work again:

/etc/init.d/networking stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/networking start

E.g.:

ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0C:29:23:95:ED
          inet addr:10.123.16.83  Bcast:10.123.16.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe23:95ed/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4425 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7426 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:316655 (309.2 KiB)  TX bytes:494628 (483.0 KiB)
          Interrupt:169 Base address:0x1424

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1556 (1.5 KiB)  TX bytes:1556 (1.5 KiB)

The issue here is that this settings will be lost upon restart. I guess it is a matter of having the proper modules loaded properly but I have not been able to find the proper configuration files combination to get this reliably enough.

Proposed solution/workaround

Basically, create a simple script that does what VMWare Tools’ intaller suggests and have it invoked upon system restart.

The script should contain the following:

#! /bin/bash
#
# vmxnet driver loader - loads the vmware network driver.
#
#

PATH=/sbin:/bin:/usr/sbin:/usr/bin

/etc/init.d/networking stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/networking start

This file should be created under /etc/init.d folder, with 755 permissions (i.e. chmod 755 filename). The, to have it invoken upon restart you could do this:

update-rc.d vmxnet-loader defaults 10
 Adding system startup for /etc/init.d/vmxnet-loader ...
   /etc/rc0.d/K10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc1.d/K10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc6.d/K10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc2.d/S10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc3.d/S10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc4.d/S10vmxnet-loader -> ../init.d/vmxnet-loader
   /etc/rc5.d/S10vmxnet-loader -> ../init.d/vmxnet-loader

I suggest having the initscript invoked with a lower sequence code (i.e. 10) so the networking gets activated before other services which may use and/or need it.

That’s it. Now your virtual machine’s networking should be fine upon restarts.

See Also

HAProxy 1.3.15.2 installation under Debian Etch (compiling from sources)

HAProxy is an excellent load balancer which performs extremely well. This page explains how to install HAProxy 1.3.15.2 since this is one of the recommended versions on the ”HAProxy Mailing List”:

Server response time discrepancy

Also, it has been recommended on that thread to use one of the following kernels:

  • 2.6.22
  • 2.6.25
  • 2.6.18

Installation Steps

First, install some required tools/packages:

apt-get update
apt-get install build-essential make libpcre3 libpcre3-dev

If you want to stick to one of the recommended kernels, at the time this how-to was written, Debian Etch standard apt-get repositories include the kernel 2.6.18, which could be installed (optional):

apt-get install linux-kernel-headers

Then you should reboot after, to start using this new kernel.

If you want to find out which kernel you are using, you may want to run this:

uname -rs

The output should be something like this:

Linux 2.6.18-6-686

Now, you should configure syslog daemon to listen following this document:

Configuring syslog to receive messages from the network (aka listen)

Check the HAProxy’s README file, and make sure:

To build haproxy, you will need :

  • GNU make. Neither Solaris nor OpenBSD’s make work with this makefile. However, specific Makefiles for BSD and OSX are provided.
  • GCC between 2.91 and 4.3. Others may work, but not tested.
  • GNU ld

Proceed with the compilation as follows (note that I have used TARGET, CPU and USE_PCRE. These options need to be double checked on the readme file, it is very clear):

cd /opt/
wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.15.2.tar.gz
tar zxvf haproxy-1.3.15.2.tar.gz

#
# Double check your options on the readme file first!!!!
# http://sysbible.org/att/HAProxy-1.3.15_README.txt
#
cd /opt/haproxy-1.3.15.2
make TARGET=linux26 CPU=i686 USE_PCRE=1
make install

ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy

Now you should be ready to go:

haproxy

HA-Proxy version 1.3.15.2 2008/06/21
Copyright 2000-2008 Willy Tarreau

Usage : haproxy -f  [ -vdVD ] [ -n  ] [ -N  ]
        [ -p
 ] [ -m  ]
        -v displays version ; -vv shows known build options.
        -d enters debug mode ; -db only disables background mode.
        -V enters verbose mode (disables quiet mode)
        -D goes daemon ; implies -q
        -q quiet mode : don't display messages
        -c check mode : only check config file and exit
        -n sets the maximum total # of connections (2000)
        -m limits the usable amount of memory (in MB)
        -N sets the default, per-proxy maximum # of connections (2000)
        -p writes pids of all children to this file
        -de disables epoll() usage even when available
        -ds disables speculative epoll() usage even when available
        -dp disables poll() usage even when available
        -sf/-st [pid ]* finishes/terminates old pids. Must be last arguments.
haproxy -vv

HA-Proxy version 1.3.15.2 2008/06/21
Copyright 2000-2008 Willy Tarreau

Build options :
  TARGET  = linux26
  CPU     = i686
  CC      = gcc
  CFLAGS  = -O2 -march=i686 -g
  OPTIONS = USE_PCRE=1

Granting MySQL Permissions

Usage

mysql> grant PERMISSIONS on DBNAME.* TO 'USERNAME'[@'SERVER'] 
    -> identified by 'PASSWORD' [with grant option];
mysql> flush privileges;

Examples

mysql> GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX on exampledb.* 
    -> TO 'exampleuser'@'10.10.10.10' IDENTIFIED BY 'secret';
Query OK, 0 rows affected (0.07 sec)
mysql> GRANT ALL PRIVILEGES on exampledb.* TO 'exampleuser'@'10.10.10.10' 
    -> IDENTIFIED BY 'secret' WITH GRANT OPTION;
Query OK, 0 rows affected (0.06 sec)

Notes

  • WITH GRANT OPTION should only be added when really needed; this privilege allows a user to grant to others (more info).
  • The location from where a user can connect (i.e. ‘username’@’server’) can be also a network if specified asĀ  ‘username’@’10.120.%.%’

See Also

Name based virtual hosting

It is possible (using HTTP/1.1) to have several websites served on the same IP address and port and still differentiate them based on the host name.

This should be done at the web server configuration level.

Microsoft IIS

IIS uses the so called “Host Headers”. It is straightforward to set up, you may want to look into these articles:0

Apache

More complex and versatile, you should check: Apache’s Name-based Virtual Host Support.

Forcing Perl to install CPAN packages via HTTP (i.e. avoiding FTP)

If you are behind a firewall and your FTP connectivity with the external world is just restricted you might get frustrated with Perl’s automatic way of installing packages (via CPAN) because it uses FTP protocol by default.

Solution

Edit your CPAN settings file (probably /etc/perl/CPAN/Config.pm) and change the line:

'urllist' => [],

it should look like this:

'urllist' => [q[http://www.perl.com/CPAN]],

After that you should be able to install automatically CPAN modules using http protocol instead of ftp.

See Also