FreeBSD Advanced Installation: UEFI And Swap on Gmirror + Root on ZFS Mirror

Hosting services is much more fun when we can be sure that malfunction of a single disk won't shut down services and cause data loss. FreeBSD Handbook has a chapter about Creating a Mirror with Two New Disks, but it suggests some outdated practices such as legacy BIOS boot and MBR partitioning scheme as opposed to UEFI boot and GPT partitoning scheme described in this article. Furthermore, contrary to examples in FreeBSD Handbook which use device nodes such as ada directly, this article describes using gpt labels for gmirror creation. We use gmirror for efi and swap partitions. We use mirror zpool for OS.

Assuming we have two disks, da0 (FreeBSD USB install media), and nda0 and ada1 (40Gb NVMes):

sysctl kern.disks
kern.disks: da0 nda1 nda0

We create gpt scheme on both disks:

gpart create -s gpt nda0
gpart create -s gpt nda1

Create gpt partitions on both disks:

gpart add -t efi -a1m -s200m -l OS00-EFI nda0
gpart add -t efi -a1m -s200m -l OS01-EFI nda1
gpart add -t freebsd-zfs -a1m -s32g -l OS00-OS nda0
gpart add -t freebsd-zfs -a1m -s32g -l OS01-OS nda1
gpart add -t freebsd-swap -a1m -l OS00-SWAP nda0
gpart add -t freebsd-swap -a1m -l OS01-SWAP nda1

Load gmirror kernel module and create two separate mirrors, one for efi, the other one for swap:

gmirror load
gmirror label EFI /dev/gpt/OS00-EFI /dev/gpt/OS01-EFI
gmirror label SWAP /dev/gpt/OS00-SWAP /dev/gpt/OS01-SWAP

Format mirrored efi partition as FAT32, mount it under /mnt, create folder structure, copy loader files to appropriate folders and unmount /mnt:

newfs_msdos -F32 -c1 /dev/mirror/EFI
mount_msdosfs /dev/mirror/EFI /mnt
mkdir -p /mnt/efi/boot
mkdir -p /mnt/efi/freebsd
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt

Mount tmpfs to /mnt, create mirror zpool and set its top-level mountpoint to none:

mount -t tmpfs tmpfs /mnt
zpool create -o altroot=/mnt OS mirror /dev/gpt/OS00-OS /dev/gpt/OS01-OS
zfs set mountpoint=none OS

Create additional zfs datasets as expected by boot environments:

zfs create -o mountpoint=none OS/ROOT
zfs create -o mountpoint=none OS/ROOT/15.1-RELEASE

Mount zfs dataset onto which FreeBSD will be installed on /mnt, configure boot environment, and exit back to installer:

mount -t zfs OS/ROOT/15.1-RELEASE /mnt
zpool set bootfs=OS/ROOT/15.1-RELEASE OS
exit

We continue as in standard install. When we get to "Final Configuration" menu and confirm "Exit", choose "Yes" when asked to "... make any final manual modifications".

Use sysrc to ensure /boot/loader.conf contains entries for loading geom_mirror and zfs kernel modules:

sysrc -f /boot/loader.conf geom_mirror_load="YES"
sysrc -f /boot/loader.conf zfs_load="YES"
sysrc -f /etc/rc.conf.d/zfs zfs_enable="YES"

Use echo to ensure /etc/fstab contains entry for swap:

echo "/dev/mirror/SWAP none swap sw 0 0" > /etc/fstab

Finalize installation by typing "Exit" and choosing "Reboot" or "Shutdown", and make sure to configure system to boot from either one of the two nvme disks.