Network Lunchbox project - JasonAntman.com

1. About the Project
2. Background
3. Features
4. Getting/Using It
5. Source Control Access
6. How I Built It
   1. Basic Install (debootstrap)
   2. Read-Only Root Filesystem
7. ToDo
8. Contact Me

My Network Lunchbox project (originally I was going to name it palantir) aims to provide a ready-to-use image for Soekris net4801 (and possibly 4501) to make them usable as general-purpose network troubleshooting devices. Specifically, the intent is enable administrators of large, complex or geographically dispersed networks to be able to install a Soekris device with one interface on a network reachable from their workstation and use the other two interfaces as entry points into unreachable or problematic networks.

At work, we were having some issues with viruses that run DHCP servers. Without getting into the nitty-gritty details, it was screwing up DHCP on one of the few hundred subnets that we run. But, since the affected subnets are in buildings spead across a few square miles, it isn't always feasible to send someone out with a laptop to do a packet capture (especially if it's a transient problem). Due to the network architecture (specifically, DHCP helper on the routers and VLANs constrained to access routers) we couldn't jump on the affected subnets from the office and dump the traffic. So, Network Lunchbox was born.

The current procedure for us is as follows:

1. If we don't have a spare Lunchbox, write the image to a CF card and put it in a net4801.
2. Have a support staff member (or one of our admins) deliver the unit to the problem location, connecting eth0 to a subnet/VLAN without any problems, and eth1 (and eth2, if need be) to a subnet that's experiencing some sort of problem (DHCP issues, rogue DHCP server, anomalous traffic, etc.).
3. Wait for the box to phone home to one of our servers via HTTP.
4. SSH to the box and run tcpdump, dhclient, nmap, and anything else we need - as if our workstations were directly on the bad subnet.
5. Perhaps, via nifty use of ssh -c, start a packet capture and save the output directly on our workstation.

• Debian 5.0 Lenny distribution based - no need to compile anything, just use apt-get to install any necessary software.
• Fullly functional Debian - multiple users (not single user mode), cron, etc.
• OpenSSH server and client, SFTP support.
• tcpdump, nmap, traceroute, ping.
• No fancy custom scripts, just trimmed down vanilla Debian.
• By default, eth0 comes up via DHCP and other interfaces stay down until someone specifcally does something to them (safe to put on "bad" subnets).
• Full PHP and Perl for scripting.
• phone home script sends IP address, hostname, MAC address, all interface information and dumped CDP information to specified URL at a given interval. All data is encrypted.
• Custom modified to run with a read-only root filesystem to allow use on a CF card.

To download and use the Lunchbox, there are a few options:

1. Download an image of the 1.0GB Compact Flash card (lunchbox0.1.img.bz2) and dd it to a CF card. The 1GB image includes a root filesystem mounted read only with 446MB free, and a /var filesystem mounted rw with 145MB free.
2. Download the archive files of the root (lunchbox0.1-hda1.tar.bz2) and /var (lunchbox0.1-hda2.tar.bz2) filesystems. You'll need to install a boot loader on the CF card, partition it, etc.

The server-side components for the phoneHome script are in subversion in the phonehome-server/ directory.

The image and filesystem archive files have the following naming convention:

• (ver) - Network Lunchbox version (as seen at lunchbox.jasoantman.com or in /lunchbox/version.php)
• (SVNrev) - The revision number of the SVN repository that this image matches up to.

Examples for Lunchbox v0.1, SVN revision 2:

• lunchbox0.1-hda1_r2.tar.bz2 - hda1 (/) filesystem archive
• lunchbox0.1-hda2_r2.tar.bz2 - hda2 (/var) filesystem archive
• lunchbox0.1_r2.img.bz2 - CF card image

Note on the full CF card image: This obviously would have fit on a 512Mb CF card. However, I opted to make the image for a full 1GB card mainly because: 1) CF cards are cheap these days and 2) since I wanted this to be an all-purpose box, I'd rather have lots of empty space than run out.

Source code for Network Lunchbox is now hosted on github, at github.com/jantman/NetworkLunchbox.

The lunchbox/ directory holds everything that's specific to this project - the phonehome script and stuff like that. The rest of the repository holds the files that I modified from the debian defaults..

How I Built it:

Unfortunately, I don't have the best notes on the build process. However, I'll write down what I remember, and fill in more in the future. The one important thing to know is that, though we're going to be using Debian on i386, we need not use Debian as our build host, or even use i386 as our build host architecture. Also, the subversion repository mentioned above has copies of most of the Debian files that I modified, if you don't want to download and extract the root filesystem archive.

1. Start with my workstation, OpenSuSE 11.1 x86_64 host.
2. For the first part of things, I more or less followed the Linux.com tutorial.
3. Download the Debian 5.0 Lenny debootstrap package.
4. Install the openSuSE alien package, convert debootstrap to an RPM.
5. Put an empty/unused/unneeded 1.0GB CompactFlash card in a USB CF reader, attach to workstation.
6. Partition a CF card the way I want it (~750MB ext2 partition, bootable, for /, the rest of it ext2 partition for /var). Format both.
7. Mount the CF card / partition at /mnt/debian and the /var partition at /mnt/debian/var.
8. debootstrap --arch i386 lenny /mnt/debian --> --http://ftp.debian.org/debian.
9. Follow the above tutorial - mount proc and dev, chroot to the new install.
10. Edit /etc/fstab. For now, leave the root partition mounted as rw. Remember that though we'll see our CF card mounted as /dev/sdX on our build host, on the actual Soekris it will be /dev/hda.
11. apt-get install anything you really need.
12. Setup /etc/hostname.
13. Setup /etc/network/interfaces.
14. Add a user (lunchbox:lunchbox, for the image) and set root password (changeme).
15. Install the linux-image-2.6.26-2-486 kernel image. Just make sure you use linux-image-*-486.
16. apt-get install grub. Now the REALLY IMPORTANT part (this could gronk your workstation). You need to install grub on the CF card, not your hard drive. On my workstation, this meant grub-install hd3. Once again, make sure you put grub on the right disk.
17. Edit /boot/grub/device.map so the only line is: (hd0) /dev/hda.
18. Edit /boot/grub/menu.lst to be correct (at a minimum, setup for serial and noacpi).
19. Uncomment the line in /etc/inittab to get a serial getty.
20. Follow the below tutorial (and stuff online for anything that I may have missed) for making a read-only root filesystem.
21. Umount the CF card partitions, put CF card in Soekris, check that it boots. If it does, and you can login, all is well.
22. Set the date via NTP

This takes some work. I honestly didn't write down all of the steps, as it took quite some trying to get rid of errors.

1. Edit /etc/fstab and add "ro" to the mount options for /.
2. mkdir /var/etc
3. rm /etc/mtab && ln -s /proc/mounts /etc/mtab
4. mv /etc/adjtime /var/etc/ && ln -s /var/etc/adjtime /etc/adjtime
5. mv /etc/motd /var/run/motd && ln -s /var/run/motd /etc/motd
6. mv /etc/resolv.conf /var/etc/resolv.conf && ln -s /var/etc/resolv.conf /etc/resolv.conf
7. modify /etc/rsyslog.conf and remove anything not needed
8. Modify bootmisc.sh and checkroot.sh in /etc/init.d/
9. Modify /sbin/dhclient-script
10. mkdir /var/etc/default
11. mkdir /var/root && mv /root/.ssh/known_hosts /var/root/ && ln -s /var/root/known_hosts /root/.ssh/known_hosts

• Get the whole filesystem (or at least everything I modified) in subversion.
• Totally strip down /etc/rsyslog.conf to minimize writes (possibly remote syslog everything??
• Fix some kernel module issues - pcspkr, acpi

You can check out my blog or just drop me an E-Mail. Or you can find more contact options here.