My main rig is running Ubuntu 18.04. To be more precise I opted out at install time to use Ubuntu Mate 18.04 but later on installed AwesomeWM and use that instead now. But without digressing much, I decided it was time to move my root (/) to ZFS. Why? - Because it is awesome!

ZFS is my favorite FS of choice for some time now. I don’t use it everywhere (am trying to be smart about it) but I prefer to do whenever I have the chance.

This main PC has 2 NVMe drives and root was installed on only one of those, second one was basically unused for the whole time (experimentation and stuff) so I had the wiggle room to do the migration.

Plan:

  1. Create ZPOOL on that spare NVMe
  2. Copy system files to that new ZPOOL
  3. Chroot and make required changes
  4. Install bootloader
  5. Reboot

Note that my system is using UEFI so steps regarding bootloader preparation differ from MBR boot type. Just follow the official Wiki and you’ll be good.

Create ZPOOL

Article from ZFSonLinux wiki page on Github was the main guideline (basically I C/P everything from there).

Destroy everything on the drive and create 2 partitions (EFI and partition which pool will be created

sgdisk --zap-all /dev/disk/by-id/nvme-Force_MP500_17047932000122530589
# Create EFI Partition
sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/nvme-Force_MP500_17047932000122530589
# Create zpool partition
sgdisk -n1:0:0 -t1:BF01 /dev/disk/by-id/nvme-Force_MP500_17047932000122530589

Now that the disk is prepared it is time to create ZPOOL and required filesystems on it

# Create pool
zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD -O xattr=sa -O mountpoint=/ -R /mnt rpool nvme-Force_MP500_17047932000122530589-part1
#  Create filesystem dataset to act as a container (like on FreeBSD)
zfs create -o canmount=off -o mountpoint=none rpool/ROOT
# Root filesystem
zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu
# Mount filesystem (to /mnt ^^)
zfs mount rpool/ROOT/ubuntu

Once the main part is done you can create datasets. I’ve omitted /home as I already use other ZFS pool as my /home

# Create root
zfs create -o mountpoint=/root rpool/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/var
zfs create -o com.sun:auto-snapshot=false rpool/var/cache
zfs create -o acltype=posixacl -o xattr=sa rpool/var/log
zfs create rpool/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/var/tmp
zfs create rpool/srv
zfs create rpool/var/games
zfs create rpool/var/mail
zfs create -o com.sun:auto-snapshot=false -o mountpoint=/var/lib/docker rpool/var/docker
zfs inherit exec rpool/var
zfs create -o com.sun:auto-snapshot=false -o setuid=off rpool/tmp
chmod 1777 /mnt/tmp

Copy system files

In this step I basically just used rsync, but I first had to mount my old systems to some new path, so I wouldn’t copy over /dev, /proc and other files as that would cause issues

# Create temp folder
mkdir /oldroot
# Mount current root to it (You CAN do this while system is live)
mount /dev/nvme1n1p3 /oldroot
# Mount current boot to it
mount /dev/nvme1n1p2 /oldroot/boot

With that out of the way we can start copying the files:

rsync -arAXHvW /oldroot/ /mnt/

Chroot and make required changes

First we need to mount standard directories

mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /bin/bash --login

Once I was there I’ve found that system wouldn’t resolve URLs (no systemd-resolved running) so I’ve just added:

nameserver 8.8.8.8

To the /etc/resolv.conf file

From there we had to install required packages:

apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs

Without proper initramfs system would refuse to boot.

Now I had to create FS for EFI partition

apt install dosfstools
mkdosfs -F 32 -n EFI /dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3
echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3) /boot/efi vfat noatime,nofail,x-systemd.device-timeout=1 0 1 > /etc/fstab
mount /boot/efi
apt install --yes grub-efi-amd64

Note the > /etc/fstab part. I’ve nuked it intentionally, you may not want to do that, so use brain and review what you need and what can/needs to be removed.

Datasets /var/log, /var/tmp and /tmp, if you leave them to be mounted by zfs import cache, will cause issues with race conditions between order of filesystem mounting (done by zfs import service) and daemons which expect some filesystems to already be available, so the solution is to switch them to legacy mountpoint and handle them in fstab

zfs set mountpoint=legacy rpool/var/log
zfs set mountpoint=legacy rpool/var/tmp
zfs set mountpoint=legacy rpool/tmp
cat >> /etc/fstab << EOF
rpool/var/log /var/log zfs noatime,nodev,noexec,nosuid 0 0
rpool/var/tmp /var/tmp zfs noatime,nodev,nosuid 0 0
rpool/tmp /tmp zfs noatime,nodev,nosuid 0 0
EOF

Ensure we mount tmpfs to /tmp

cp /usr/share/systemd/tmp.mount /etc/systemd/system/
systemctl enable tmp.mount

Install bootloader

Before proceeding you can verify from chroot that root filesystem is recognized by the grub

grub-probe /
# should output "zfs"

Update initramfs files, grub and install grub

update-initramfs -u -k all
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck --no-floppy

Verify that module for ZFS is installed

ls /boot/grub/*/zfs.mod

Reboot

At this point I’ve found that unmounting things and rebooting gracefully didn’t work, you may try to exit chroot, umount all filesystems from /mnt and reboot, good luck! When my reboot stopped I just rebooted the system by pressing the reset button.

Add second drive to the rpool

Once the system booted up I’ve nuked the old drive (holding old root) and attached it to the rpool so I have redundancy (mirror).

To achieve that I’ve used:

sgdisk --zap-all /dev/disk/by-id/nvme-Force_MP500_17047932000122530587
# Create EFI Partition
sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587
# Create zpool partition
sgdisk -n1:0:0 -t1:BF01 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587
# Attach drive to the existing rpool
zpool attach rpool /dev/disk/by-id/nvme-Force_MP500_17047932000122530589 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587

Note that you have to specify current zpool drive first so it will attach it to it as a mirror, and not do the striping (raid 0) except that is what you’re aiming for.

And to ensure we can still boot if one drive fails we need to have efi on this drive as well so just dd partition from the first drive to the second one:

umount /boot/efi
dd if=/dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3 of=/dev/disk/by-id/nvme-Force_MP500_17047932000122530587-part3
efibootmgr -c -g -d /dev/disk/by-id/nvme-Force_MP500_17047932000122530587 -p 3 -L "ubuntu-2" -l '\EFI\Ubuntu\grubx64.efi'
mount /boot/efi