BlogFish

Just a quick note for people running OpenBSD with memory-based filesystems like described in my article.

If also get double for filesystems (e.g. /var ) after booting:

# mount
/dev/wd0a on / type ffs (...)
mfs:26303 on /var type mfs (...)
mfs:30728 on /dev type mfs (...)
mfs:6263 on /var type mfs (...)

Make sure to include the noauto option for the “double” filesystem:

# cat /etc/fstab
/dev/wd0a / ffs ro 1 1
swap /var mfs rw,-P=/proto/var,-s=32768,noexec,nosuid,nodev,noauto 0 0
swap /dev mfs rw,-P=/proto/dev,-s=1200,-i=128,noexec,nosuid 0 0

The problem is that /var was also mounted from /etc/rc. Got this from misc@openbsd.org.

Further a new version of Flashdist for OpenBSD 3.7 and Current is out.

After discussing some issues with Thomas, we have now an improved solution for OpenBSD 3.7 on WRAP.

First, we combined the two memory-based filesystems on /var and /tmp into one and symlinked /tmp to /var/tmp.
Then we increased the mfs-filesystem to 16 MB (or even 32 MB) because the default 8 MB are nearly full after the default install.

So the /etc/fstab now looks like this (16MB):

/dev/wd0a / ffs ro 1 1
swap /var mfs rw,-P=/proto/var,-s=32768,noexec,nosuid,nodev 0 0
swap /dev mfs rw,-P=/proto/dev,-s=1200,-i=128,noexec,nosuid 0 0

Then there was the problem of keeping the logs, mail and other important stuff of /var over reboots. As /var is mounted on a memory-filesystem, its content is vanished if the machine reboots. On a new boot it is populated from /proto/var, a cpoy of the “real” /var after installing. My solution was calling rsync from /etc/rc.shutdown:

/usr/local/bin/rsync -vaz /var/ /proto/var/

This syncs the actual /var with /proto/var before shutdown or a reboot so that when /var is populated from /proto/var on a boot, everything is sync and current. No logs, mails or the content of /var/db is lost. rsync can be installed easily from a package.

pkg_add ftp://my/path/to/packages/i386/rsync-2.6.3.tgz

I had to complete first OpenBSD 3.7 CD on a local HTTP mirror. Worked fine. Do not wonder if the rsync call is reporting such errors:

rsync: mknod ”/proto/var/cron/tabs/.sock” failed: Invalid argument (22)
...
rsync: mknod ”/proto/var/empty/dev/log” failed: Invalid argument (22)
...
rsync error: some files could not be transferred (code 23) at /usr/obj/i386/rsync-2.6.3/rsync-2.6.3/main.c(702)

That means that rsync could not create the sockets for cron and bind. Nothing to worry about because as far as I can tell they are created on boot on /var and this is the only place where they are needed.

Another problem was that OpenBSD only detected 64 MB RAM instead of 128 MB. The solution is to update the Bios to at least 1.07. We installed 1.08 through xmodem upload. This was a hard fight for me because I could not get minicom to enter the Bios or tip starting the xmodem upload. With tip I was able to enter the Bios but I could not start an upload with lrzsz as tip was no longer reacting to the escape characters. I tried minicom but with minicom I could not enter the Bios. After OpenBSD booted, everything worked through minicom, but pushing “s” in order to enter the WRAP Bios resulted in no reaction. After some hours of fun with minicom/c-kermit/tip/cu/lrzsz Thomas finally got it.

The trick was to boot the WRAP first and then connect through minicom. Only so we were able to enter the Bios and upload the new 1.08 Bios with lrzsz from within minicom. We had always connected with minicom first and than booted the WRAP because you had to be fast in order to push “s” during memory counting. For more information on the Bios update see this page about Bios update on the very similar Soekris NET4801.

After this hurdle my WRAP uses 128 MB RAM under OpenBSD and can netboot with PXE.

Further Thomas has a new webpage dedicated to OpenBSD 3.7 and WRAP. There you can also find some information on how to control the LEDs.

UPDATE:

You can get the Bios from here: <a href=”http:// www.pcengines.ch/wbios108.zip”> www.pcengines.ch/wbios108.zip

UPDATE 2:

See my follow-up article for a small issue with double mounted filesystems.

I’ve been playing around with OpenBSD 3.7 and my WRAP. I wanted to do a normal, full install and not a stripped down version. If you are looking for this, check out OpenSoekris or Flashdist. This writing should also apply to Soekris boxes with CF cards.

In order to install OpenBSD on the CompactFlash card I plugged the card with a card reader to a PC and booted the OpenBSD installation media. I choose to install on the sd0 disk (da0 on FreeBSD). So far so good. Now comes the tricky part. I had to edit the drive settings in disklabel with the “e” command in order to set the cylinders/heads/sectors to the numbers that the WRAP BIOS saw. You have only to do this if the geometry that the WRAP reports is different from the geometry that the PC reports. After this hurdle, the rest is easy.

When prompted if one want to redirect to serial console or com0 enter yes and choose a baud rate of 38400. The installed will then insert these lines in /etc/boot.config:

stty com0 38400
set tty com0

When you are finished with the installation you get a shell. Check that the boot.config settings are present:

# cat /mnt/etc/boot.config

If they are not present insert them:

# echo “stty com0 38400” > /mnt/etc/boot.config

Next you have to edit the fstab because the installer used sd0 as the CF device but the WRAP will use wd0:

# echo ”/dev/wd0a / ffs rw 1 1” > /mnt/etc/fstab

You one can halt the machine, get the CF card from the card reader and insert it into the WRAP. Before you boot the WRAP, connect to its serial console from another computer.
On OpenBSD do:

# tip -38400 tty00

On FreeBSD the device is sio0. When you boot the WRAP you should see the BIOS and boot screen over the serial console. Until I got the geometry problem solved I only saw an error stating: No O/S. In this case check the shown C/H/S values with those you use during install.

Hopefully OpenBSD will boot normally and from now we’ve got a “normal” OpenBSD PC.

But there are some issues left. CF cards are not made for many read-write cycles and a normal install will harm your CF card over time. So lets fix that. First one could mount the root filesystem noatime. This means than the inode is not updated on reads which will save write cycles.

# echo ”/dev/wd0a / ffs rw,noatime 1 1” > /etc/fstab

The best would be if one could mount root read-only but some directories like /tmp or /var have to be read-write. So will use a memory based filesystem for /tmp. This ensures that it is writable and that no read or write operations will touch the CF card for /tmp. Be sure to use ”>>” instead of ”>” or the content of /etc/fstab will be lost.

# echo “swap /tmp mfs rw,nodev,nosuid,noexec 0 0” >> /etc/fstab

This does not solve the problem with /var because /tmp holds only temporary data and /var contains some directories and files that should be present on boot or normal operation. The solution is to use a memory based filesystem but populate it with a copied /var directory on creation.

# mkdir /proto
# cp -rp /var /proto/var
# echo “swap /var mfs rw,-P=/proto/var,noexec,nosuid,nodev 0 0” >> /etc/fstab

The last thing left is /dev. Again we need a memory based filesystem populated with the content of the former /dev but we cannot just copy /dev. In /dev device nodes reside so we have to create them with the MAKEDEV script.

# mkdir /proto/dev
# cp /dev/MAKEDEV /proto/dev
# cd /proto/dev && ./MAKEDEV all
# echo “swap /dev mfs rw,-P=/proto/dev,-s=1200,-i=128,noexec,nosuid 0 0” >> /etc/fstab

Now the root filesystem can be mounted read-only. The resulting /etc/fstab should look like this:

# more /etc/fstab
/dev/wd0a / ffs ro 1 1
swap /tmp mfs rw,nodev,nosuid,noexec 0 0
swap /var mfs rw,-P=/proto/var,noexec,nosuid,nodev 0 0
swap /dev mfs rw,-P=/proto/dev,-s=1200,-i=128,noexec,nosuid 0 0

Reboot the WRAP. OpenBSD should now use the CF card only for reading and use memory for writing. Whenever you want to change something on the CF like for editing a file in /etc, just remount / read-write, make your change and remount / read-only.

Remind you that using memory based filesystem has the disadvantage that nothing survives a reboot. Make sure to log to a logging host over the network if you want to keep logs or other information.

Thanks to Derick Siddoway for help on some issues.

UPDATE:
Check out the follow-up article.

Yesterday Thomas and I received our WRAP boxes. They are cheaper than the Soekris net4801 but lack the IDE connector, the PCI slot, the USB interface, and have only one serial port. But for our purposes they suffice.

We are planning to do some testing with CARP+PF like described here. We will hopefully move on to create a redundant OpenBSD Hardware Firewall consisting of two or more WRAP/Soekris/VIA C3/?? boxes configured by a web-interface (maybe Ruby on Rails :-). The whole thing would be placed in a 1U Rack.