Most of the OpenBSD systems I am in charge of are deployed in data centres, powered by UPSs which provide them with electrical power during periods of public grid power outages. But there is also a number of OpenBSD systems I administer, which are deployed in much less favourable conditions; where frequent power outages last longer than UPS batteries do, or where there are no UPSs at all (such as branch office routers in godforsaken places where having electricity and Internet access at all is considered a lucky circumstance). These latter systems are likely to have high rate of unclean shutdowns caused by prolonged or unexpected power outages, which in turn increase the probability of their inability to boot without human intervention. This article describes steps to make OpenBSD system more resilient to unexpected power outages by minimising the possibility of inconsistent file systems after unclean shutdowns, which is achieved by mounting all disk partitions in read-only mode. Filesystems which have to be writable - /var, dev and /tmp - are mounted as writable memory file systems.

Described setup does not work correctly on OpenBSD 6.4, due to possible bug in unveil(2), described on bugs@, which prevents using tcpdump while / (actually /etc, to be more specific) is mounted read-only. Let's hope OpenBSD will (sys)patch this for 6.4 soon. Until then, 6.3 is still way to go with described read-only setup.

Hardware

I used PC Engines apu3b4 router board because of the following characteristics:

  • It features 3 GbE ports which is ideal for my branch offices - two ports are used for two different ISPs, while the third one is used for local LAN
  • Its CPU - 1 GHz quad Jaguar core with 64 bit and AES-NI support - does good job encrypting IPSEC flows
  • It has 4Gb of RAM, which is more than enough for basic routing and firewalling tasks, but also for memory fily systems

As for local SD card storage, I used SanDisk Extreme PRO. It's relatively pricey, but my experiments with lower quality SD cards resulted in prolonged branch office downtime and 500 kilometer trip, which in the end proved much more expensive than initial savings.

My laptop doesn't have serial port, so I used Moxa UPort 1110 USB to Serial adapter to connect to router board.

Ethernet port closest to the serial port - em0 - needs to be connected to network, and its IP address, configured during initial setup, needs to have full Internet access.

Bootable USB flash drive was created with dd from install62.fs image, and inserted into any of the two USB slots on apu3b4.

Software

For the purpose of this tutorial, I used GNU Screen on Linux Mint, mostly because it provided me with easy way to create video tutorial with recordMyDesktop. Contrary to Linux Mint 18.3, which needs third-party driver from Github (Moxa's official driver for Linux is for older kernels and it won't work on contemporary Linuxes), Moxa UPort 1110 USB to serial adapter works with OpenBSD 6.2 out of the box. GNU Screen is also available as a package, so the only piece of software I'm missing in order to create next update to this tutorial in OpenBSD is for screen recording. If you are familiar with such software, please drop me a line.

I also use rsync on the router for the purpose of periodical syncing of /var memory file system to disk, which is the only third-party package in the setup which does not come with base OpenBSD 6.2 install. If you know how to accomplish the same goal with just OpenBSD base system, please let me know.

Installation

I assume you are already familiar with simple install, so I won't write here about preparation of install media, choosing boot device, setting serial port's baud rate and com port, system hostname, IP parameters, root password and creation of non-privileged system user - you can see these in video tutorial.

After completion of above install stages, we arrive at the most important part for our read-only setup , which is appropriate manual partitioning. When asked, Allocate (W)hole disk to OpenBSD, choose (C)ustom layout, and partition as follows:

label     mountpoint     size
a         /              512m
b         <none>           1g
d         /usr             1g
e         /usr/local     256m
f         /home            1g
g         /mfs             1g
h         <none>           1g

Continue to the end of the install and reboot when instructed so, but not before removing install media USB flash disk.

First single-user mode boot

At this stage you are going to move some file systems around, which requires booting into single-user mode. After gaining access to local shell, make sure to:

  • check if / partition is clean.
  • mount / partition as writable.
  • mount all the partitions listed in fstab.
  • move complete /var folder into /mfs/ folder, so that our /var memory file system can be populated from its contents on reboot by using -P option in fstab (you can read more about this option in mount_mfs manpage).
  • re-create empty /var folder which will be used as mount point for our /var memory file system.
  • create /mfs/dev folder which will provide device nodes to /dev memory file system, also by means of -P option in fstab.
  • move MAKEDEV script from /dev into /mfs/dev/.
  • change directory to /mfs/dev/ and execute MAKEDEV all so that all device nodes are created there.
  • create file system on /dev/rsd0h, as this partition wasn't formatted during installation because we did not assign mountpoint to it. We will use this partition to store syspatch binary patches in /var/syspatch on disk, and not in /var memory file system, as these can come to consume quite a bit of space over lifetime of a release.

Here's how above looks in a shell:

fsck -p /
mount -uw /
mount -A
mv /var /mfs/
mkdir /var
mkdir /mfs/dev
cp /dev/MAKEDEV /mfs/dev/
cd /mfs/dev/ && ./MAKEDEV all
newfs /dev/rsd0h

Now that we have prepared partitions and mountpoints, we need to edit our fstab so that memory file systems are used for /var, /dev and /tmp.

Below fstab is not copy-pastable as I'm using unique disk identifiers - DUUIDs!

Before invoking vi in single-user mode you have to set TERM environment variable, eg export TERM=vt220

4684a15caa1363f1.a / ffs rw 1 1
4684a15caa1363f1.b none swap sw
4684a15caa1363f1.d /usr ffs rw,nodev 1 2
4684a15caa1363f1.e /usr/local ffs rw,wxallowed,nodev 1 2
4684a15caa1363f1.f /home ffs rw,nodev,nosuid 1 2
4684a15caa1363f1.g /mfs ffs rw,nodev,nosuid 1 2

swap /var mfs rw,nodev,nosuid,-s256m,-P=/mfs/var 0 0
swap /dev mfs rw,nosuid,noexec,-s4m,-P=/mfs/dev,-i128 0 0
swap /tmp mfs rw,nodev,nosuid,-s256m 0 0

4684a15caa1363f1.h /var/syspatch ffs rw,nodev,nosuid 1 2

You can now reboot.

First standard boot

When you get the prompt, our router should already have /var, /dev and /tmp partitions mounted as memory file systems. However, it is still as vulnerable to unclean shutdowns as standard installation, as its disk file systems are still mounted in writeable mode. Before we can make them read-only we need to install rsync so that contents of /var memory partition can be periodically backed up to /mfs/var directory on disk. Using this method makes sure that our /var memory file system gets populated with relatively fresh content, such as logs, package database, configuration files of daemons which reside in /var (such as unbound) etc., on reboots. If we didn't sync, our /var memory file system would get populated with contents created during intial install, and all the changes would be deleted on next reboot.

Setting installurl and adding rsync package is too trivial to describe here, but you can see how I do it in video tutorial. After installing rsync, a script named syncvar.sh, which syncs /var memory file system to /mfs/var folder on disk needs to be created in /root/ folder. Besides mounting all disk file systems in writeable mode, and remounting them back in read-only mode after syncing, I have also added random_seed() function from rc, after nice people of OpenBSD misc@ mailing list explained me why it is so important.

The script looks as follows:

#!/bin/sh

/sbin/mount -uwA -t nomfs
dd if=/var/db/host.random of=/dev/random bs=65536 count=1 status=none
chmod 600 /var/db/host.random
dd if=/dev/random of=/var/db/host.random bs=65536 count=1 status=none
dd if=/dev/random of=/etc/random.seed bs=512 count=1 status=none
chmod 600 /etc/random.seed
/usr/local/bin/rsync -avx --delete --no-specials --no-devices /var /mfs/
/sbin/mount -urA -t nomfs

Don't forget to chmod the script to 755 so that you can invoke it from command line

Above script should be invoked periodically - not too often because of SD card wear and vulnerability of the system to unclean shutdowns during the time of its running, but also not too rarely as we want to retain as fresh information from logs as possible. Running it daily, at midnight, from cron, and also on intended shutdowns and reboots seems OK, so we append the following to root's crontab:

@daily /root/syncvar.sh >/dev/null 2>&1

And create rc.shutdown file with the following contents:

#!/bin/sh
/root/syncvar.sh

Last, but not the least, we will add custom function to rc.local which will mount disk file systems in read-only mode only after initial kernel relinking is done - of course we want to make use of the latest OpenBSD security feature!

mount_ro() {
  while sleep 5; ([ -n "$(pgrep -f 'reorder_kernel')" ]); do done
  /usr/local/bin/rsync -avx --delete --no-specials --no-devices /var /mfs/
  /sbin/mount -urA -t nomfs
}

mount_ro &

I was initially modifying rc directly in order to achieve late mount of disk file systems in read-only mode, which is a terrible idea. I asked around on @misc mailing list and got some answers publicly, however the above nice solution was forwarded to me off-list, by another list member who also obtained it off-list. I will be more than happy to give credit here, just drop me a line if you are an author of above solution.

You can now reboot into OpenBSD system which is hopefully much more resilient to unexpected power outages without sacrificing any of its security or functionality.

Pay attention to the fact that read-only setups aren't officially supported by OpenBSD developers!

In order to make changes on disk partitions you need to mount them in writeable mode with mount -uwA -t nomfs

Mount disk partitions back to read-only mode with mount -urA -t nomfs afterwards!

Video Tutorial

I hate those, but some people asked me if I could record my session.

Got any feedback? Drop me a line, please.

Next Post Previous Post