NixOS on Btrfs+tmpfs

In 2018, dad bought me a new laptop to replace the good ole Compaq nx7010 whose screen unfortunately got infected by some sort of microbe and dieded shortly afterwards. The new one, whilst having a considerably worse build quality (like all other late-2010s ones when compared to mid-2000s models), had a dozen times as much storage: a 250 GB M.2 SSD and a 500 GB SATA HDD.

My data hoarding habit has grown exponentially ever since. Initially, I used to back up the data from the SSD to the HDD but after a few years, I ran out of space and decided to get some more storage. Instead of buying a portable hard disk like a normal person would, I went for an SATA SSD, as it was rather difficult to find a 7200 rpm 2.5-inch^[1] HDD in the market at the time.

I then asked my father for a spare SATA-to-USB case (he switched to using a dock a while ago, and like other dads, nothing is ever thrown away) and prepared to swap the drives. As cloning the data would have been too easy, I decided to spice things up by reinstalling the OS. Back then I was dual-booting Debian and NixOS, but the former had hardly been ever booted for months so it was time to let it go:

Elsa rolling on the floor crying

In addition, I wanted to hop on the new and shinny^[2] train of Btrfs. It has compression, snapshots and subvolumes, what's not to love? Let's replace something I'd been using for nearly a decade with a file system I had absolutely zero experience with, what could possibly go wrong, right?

Reinstallation
Backup
1. Initialization
2. Repetition

Reinstallation

I was going to reinstall NixOS with an ephemeral root, which had been covered to death in the following brilliant resources:

The only twist here is that I was using Btrfs instead of ZFS or ext4 like in other guides. This choice would influence how to back up in the later section.

Preparation

First of all, I temporarily copied data to the SATA SSD from the M.2, including my Nix configurations. Using either cp or rsync didn't seem to have any effect on the performance, and in the mean time I also went ahead and grabbed a NixOS unstable live image and dd'ed it to a flash drive. As I'm tracking unstable, installing from the same version would allowed me to skip switching the channel and a lot of downloading.

Partitioning

After booting up the live image, I opened up a root shell with sudo -i. As expected, fdisk reports the M.2 SSD as /dev/nvme0n1. Paranoid as always, I decided to give the EFI system partition a whole gibibyte, swap eight to match memory^[3] and the rest as a single chonky Btrfs partition:

parted /dev/nvme0n1 -- mklabel gpt
parted /dev/nvme0n1 -- mkpart ESP fat32 1MiB 1GiB
parted /dev/nvme0n1 -- set 1 boot on
mkfs.vfat /dev/nvme0n1p1

parted /dev/nvme0n1 -- mkpart Swap linux-swap 1GiB 9GiB
mkswap -L Swap /dev/nvme0n1p2
swapon /dev/nvme0n1p2

parted /dev/nvme0n1 -- mkpart primary 9GiB 100%
mkfs.btrfs -L Butter /dev/nvme0n1p3

As I typed this, I realized that I should have set up encryption for the last partition so I would probably need to reinstall in the near future to fix this mistake. Anyway, with the target system's root mounted as tmpfs, I would need to persist /nix (obviously), /etc (mostly for authentication and other secret stuff not included in configuration.nix that I was too lazy to opt in individually), /var/log, /root and /home:

mount /dev/nvme0n1p3 /mnt
btrfs subvolume create /mnt/nix
btrfs subvolume create /mnt/etc
btrfs subvolume create /mnt/log
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
umount /mnt

Most subvolumes can be mounted with noatime, except for /home where I frequently need to sort files by modification time. All of them should have forced compression though:

mount -t tmpfs -o mode=755 none /mnt
mkdir -p /mnt/{boot,nix,etc,var/log,root,home}
mount /dev/nvme0n1p1 /mnt/boot
mount -o subvol=nix,compress-force=zstd,noatime /dev/nvme0n1p3 /mnt/nix
mount -o subvol=etc,compress-force=zstd,noatime /dev/nvme0n1p3 /mnt/etc
mount -o subvol=log,compress-force=zstd,noatime /dev/nvme0n1p3 /mnt/var/log
mount -o subvol=root,compress-force=zstd,noatime /dev/nvme0n1p3 /mnt/root
mount -o subvol=home,compress-force=zstd /dev/nvme0n1p3 /mnt/home

Configuration

With everything mounted, nixos-generate-config --root /mnt could be run to generate a basic configuration. But wait, didn't I say something about my dot files? That's correct, but it's not easy to handcraft the hardware-configuration.nix. After making sure all are mounted with the right options and services.fstrim.enable is true, I copied other configuration files to /etc/nixos and finished this step.

Installation

NixOS installation is as simple as running nixos-install. But my job was not done after setting the root password and rebooting into the new system. It was working, but not functional. There was nothing meaningful for me to do on it, so I had to log in (as root), passwd'ed the user and copied the home folder back from the temporary drive.

After freeing the new SATA SSD, I also filled it with butter. Yes, all the way, no GPT, no MBR, just Btrfs, whose subvolumes were used in place of partitions:

mkfs.btrfs -f -L Fly /dev/sdb
mkdir -p /mnt
mount /dev/sdb /mnt
btrfs subvolume create /mnt/movies

At that time the only disposable data I had were my movies collection. The HDD also contained other data but they were rebalanced at /home (on the M.2). After swapping the SATA SSD inside the laptop, I logged in as the normal user and get the exact same environment before the reinstallation.

Profits

Thanks to subvolumes and compression, the free spaces were no longer fragmented and I think I gained like 100 GB (not counting the old Debian's root). Backup would also be less painful with Btrfs snapshots (instead of plain rsync like I used to) as shown as follows.

Backup

With all data migrated, the HDD could be used for backing up. First, some legacy data I no longer access were moved there, then I started to back up my /home partition:

Initialization

Having learned my lesson, I did not forget to set up LUKS this time:

cryptsetup luksFormat /dev/sdb
cryptsetup luksOpen /dev/sdb backup

To make use of snapshots, the backup drive gotta be Btrfs as well. The compression level was turned up to 14 this time (default was 3):

mkfs.btrfs -L Backup /dev/mapper/backup
mkdir /backup
mount -o noatime,compress-force=zstd:14 /dev/mapper/backup /backup

Following Btrfs Wiki, I made the first /home snapshot and sent it to the backup drive:

btrfs subvolume create /backup/home
today=$(date --iso-8601)
btrfs subvolume snapshot -r /home /home/$today
sync
btrfs send /home/$today | btrfs receive /backup/home
sync

Repetition

For next backups, I also mounted the drive and created a snapshot:

cryptsetup luksOpen /dev/sdb backup
mkdir -p /backup
mount -o noatime,compress-force=zstd:14 /dev/mapper/backup /backup
today=$(date --iso-8601)
btrfs subvolume snapshot -r /home /home/$today
sync

Say the latest snapshot was on the $previous day, I only needed to send the difference between the old and new backup. Afterwards, it is safe to delete the local $previous snapshot to save some space.

btrfs send -p /home/$previous /home/$today | btrfs receive /backup/home
btrfs subvolume delete /home/$previous
sync

Finally, unmount the drive and close the LUKS volume:

umount /backup
cryptsetup luksClose backup

Is this more complicated than good ole rsync? Yes. Is it safer? Also yes, thanks to copy-on-write. Would I bother using one of the tools suggested in the wiki? Probably not, I've already documented everything in this article in case I forget anything.

[1]	63.5 mm for those outside of the land of guns and burgers

[2]	OK, maybe not new, but certainly shinny

[3]	Slightly larger since some of the memory is dedicated to graphics

Tags: fun recipe nix —Nguyễn Gia Phong, 2021-11-14

Comments

Follow the anchor in an author's name to reply. Please read the rules before commenting.