Grub, BtrFS + Snapshots, and possibly UKI?

So, I’m trying to determine best approaches here, and open a discussion on the subject of bootable snapshots and possibly UKI.

Firstly, I have a pretty extensive setup of BTRFS with subvolumes that are similar to the way openSUSE kinda does it, but also at the same time different. I’ll cover more on the details later. My main goal is to determine what I can do, best practices, and viable options, regarding what I have, and what I can do with what I have.

I used to use kernel-modules-hook and a hook I myself wrote, kernel-rotate-hook, that would basically try to retain X versions older of kernels and their representing modules to be available, to prevent a “Emergency Mode” boot, and have, basically, fallbacks. But this proved, over time, to actually not be very effective, ultimately and even booting snapshots still just outright failed.

So, I’m looking at new options here. I’m using grub. /boot is ext4, /boot/efi is the usual fat, and / is btrfs with subvolumes as shown in this quick and simple mount table:

/dev/nvme0n1p3 on / type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@)
/dev/nvme0n1p3 on /.snapshots type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=272,subvol=/.snapshots)
/dev/nvme0n1p3 on /home type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/@home)
/dev/nvme0n1p3 on /opt type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=261,subvol=/@opt)
/dev/nvme0n1p3 on /root type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=271,subvol=/@root)
/dev/nvme0n1p3 on /srv type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@srv)
/dev/nvme0n1p3 on /usr/local type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=262,subvol=/@usrlocal)
/dev/nvme0n1p3 on /var/lib/containers type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=264,subvol=/@containers)
/dev/nvme0n1p3 on /var/cache type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=258,subvol=/@cache)
/dev/nvme0n1p3 on /var/lib/crash type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=263,subvol=/@crash)
/dev/nvme0n1p3 on /var/lib/libvirt/images type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=265,subvol=/@libvirt)
/dev/nvme0n1p3 on /var/lib/machines type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=266,subvol=/@machines)
/dev/nvme0n1p3 on /var/lib/portables type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=267,subvol=/@portables)
/dev/nvme0n1p3 on /var/spool type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=268,subvol=/@spool)
/dev/nvme0n1p3 on /var/log type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@log)
/dev/nvme0n1p3 on /var/tmp type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=269,subvol=/@tmp)
/dev/nvme0n1p3 on /var/www type btrfs (rw,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=270,subvol=/@www)

As-is, what happens when I try to boot a kernel that is current, into a snapshot that doesn’t have that kernel,it usually just fails into Emergency Recovery mode where you enter root’s password or Ctrl+D, which fails again. Root login works at that point, though.

I’ve considered splitting /usr/lib/modules/ into it’s own subvolume as well, so no matter what scenario is, /boot and /usr/lib/modules will match up.

I’m considering UKI as an option, but eos-dracut makes explicit /boot/initramfs-* images accordingly.

I’ve heard, may not be accurate but heard, dracut’s UKI options can “retain” X number of kernels on it’s own. Is this any different than the traditional kernel+initramfs and worth exploring at all?

I’d like thoughts, suggestions, and ideas on these, so I can hopefully bridge this up in a way that is a little more stable when need-be than the current, “oh crap, that thing’s broken, boot snapshot, oh crap, snapshot also failed.” scenarios..

That setup making this more complicated than it could be.

If you stopped having so many separate subvolumes it would “just work” and you wouldn’t need to do anything special.

If you use grub and /boot, / and any other relevant things for boot are all part of the same btrfs subvolume, it makes things vastly simpler.

Is there a reason you need them so separated?

Well, /home I always want separate. /opt is used for some packages, but also some manually installed things as well. Technically it could be part of / not as a subvolume. /root, is similar to /home, it’s a home directory that shouldn’t be snapshot along with /. /usr/local is very similar to /opt, but definitely not package installed. Which is funny in itself, but, hey. It could still be part of / though that said.
The rest, though, /var/lib/containers, /var/cache, /var/lib/libvirt/images, /var/lib/machines, /var/lib/portables, /var/spool, /var/log, /var/tmp, /var/www, and /srv, I believe should never be snapshot with / – Just doesn’t make sense.

None of those things you listed is a problem. That begin said, I would advocate for /opt to be part of / since packages do write data there.

But you also said that /boot is separate from /. That is an issue.

Yeah, I have heard /boot not being part of / being an issue more or less lately, but it’s also got a double-edged sword. Making /boot be part of / which is BtrFS, the option: GRUB_DEFAULT='saved' no longer works. While it’s not a HUGE problem, I do switch between kernels on need, usually just between linux-zen and linux-lts. I’ve tested others in certain scenarios just in case, but usually found no major differences between those two and similar.

Moving /boot into / isn’t a huge issue. It’s only 2GB itself.

But, is that truly the only way to safely utilize bootable btrfs snapshots?

And then, is UKI worth anything or not?

It is definitely the easiest and most reliable. Otherwise you need to use scripts to try to save the state of /boot and then have some other script to restore it along with your / restore.

UKIs have the exact same problem as a separate /boot. Basically you are removing the kernel/initrds from the btrfs snapshot and so they don’t get restored and you can end up with a mismatch.

Yeah, that’s basically in essence, what my hook, https://github.com/erenfro/kernel-rotate-hook was doing. Only with selected kernel versions though, according to settings, and I would have to select the right kernel version to what was available in the modules. But with kernel-modules-hook, that was mostly there too anyway, and yet… Still…

1 Like

Well, I’ve changed my /boot to being /@/boot on my BtrFS, did the grub-install to update that, and now… My EOS won’t boot. Weirdness out of that though is the boot log message from it.

Here’s my QR Coded output of the boot failure, which I notice shows nothing from NVMe, just sdX:

```
Panic Report
Arch: x86_64
Version: 6.17.4-zen2-1-zen
[ 0.668318] check access for rdinit=/init failed: -2, ignoring
[ 0.812580] usb 5-1: new high-speed USB device number 2 using xhci_hcd
[ 0.826453] ata6: SATA link down (SStatus 0 SControl 330)
[ 0.826527] ata10: SATA link down (SStatus 0 SControl 300)
[ 0.874582] usb 1-1: new full-speed USB device number 2 using xhci_hcd
[ 0.940733] usb 5-1: New USB device found, idVendor=05e3, idProduct=0610, bcdDevice=68.75
[ 0.940735] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 0.940736] usb 5-1: Product: USB2.1 Hub
[ 0.940737] usb 5-1: Manufacturer: GenesysLogic
[ 0.981327] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 0.981519] ata2.00: Model ‘Samsung SSD 860 EVO 500GB’, rev ‘RVT01B6Q’, applying quirks: noncqtrim zeroaftertrim noncqonati nolpmonati
[ 0.981634] ata2.00: supports DRM functions and may not be fully accessible
[ 0.981635] ata2.00: ATA-11: Samsung SSD 860 EVO 500GB, RVT01B6Q, max UDMA/133
[ 0.982333] ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[ 0.985658] ata2.00: Features: Trust Dev-Sleep DIPM NCQ-sndrcv
[ 0.986076] ata2.00: supports DRM functions and may not be fully accessible
[ 0.989200] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 0.989335] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 0.990120] hub 5-1:1.0: USB hub found
[ 0.990299] ata2.00: configured for UDMA/133
[ 0.990731] hub 5-1:1.0: 2 ports detected
[ 1.000268] ata1.00: ATA-8: OCZ-AGILITY3, 2.25, max UDMA/133
[ 1.000816] ata1.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 32), AA
[ 1.001320] ahci 0000:02:00.1: port does not support device sleep
[ 1.010530] ata1.00: configured for UDMA/133
[ 1.011462] scsi 0:0:0:0: Direct-Access ATA OCZ-AGILITY3 2.25 PQ: 0 ANSI: 5
[ 1.011583] sd 0:0:0:0: [sda] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[ 1.011589] sd 0:0:0:0: [sda] Write Protect is off
[ 1.011590] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 1.011596] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[ 1.011605] scsi 1:0:0:0: Direct-Access ATA Samsung SSD 860 1B6Q PQ: 0 ANSI: 5
[ 1.011606] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[ 1.011706] ata2.00: Enabling discard_zeroes_data
[ 1.011712] sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[ 1.011717] sd 1:0:0:0: [sdb] Write Protect is off
[ 1.011718] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 1.011733] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[ 1.011748] sd 1:0:0:0: [sdb] Preferred minimum I/O size 512 bytes
[ 1.011868] ata2.00: Enabling discard_zeroes_data
[ 1.017678] ata9.00: ATA-10: ST4000DM004-2CV104, 0001, max UDMA/133
[ 1.020479] sd 0:0:0:0: [sda] Attached SCSI disk
[ 1.021469] sdb: sdb1 sdb2
[ 1.022493] sd 1:0:0:0: [sdb] supports TCG Opal
[ 1.022494] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 1.026774] ata9.00: 7814037168 sectors, multi 16: LBA48 NCQ (depth 32), AA
[ 1.053947] usb 6-1: new SuperSpeed USB device number 2 using xhci_hcd
[ 1.067898] usb 6-1: New USB device found, idVendor=05e3, idProduct=0620, bcdDevice=68.75
[ 1.067900] usb 6-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1.067901] usb 6-1: Product: USB3.1 Hub
[ 1.067902] usb 6-1: Manufacturer: GenesysLogic
[ 1.076531] ata9.00: configured for UDMA/133
[ 1.085972] hub 6-1:1.0: USB hub found
[ 1.086270] hub 6-1:1.0: 2 ports detected
[ 1.178599] usb 5-2: new high-speed USB device number 3 using xhci_hcd
[ 1.238789] usb 1-1: New USB device found, idVendor=046d, idProduct=c547, bcdDevice= 4.02
[ 1.238791] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1.238792] usb 1-1: Product: USB Receiver
[ 1.238792] usb 1-1: Manufacturer: Logitech
[ 1.258644] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1:1.0/0003:046D:C547.0001/input/input2
[ 1.258692] hid-generic 0003:046D:C547.0001: input,hidraw0: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:02:00.0-1/input0
[ 1.265703] input: Logitech USB Receiver Keyboard as /devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1:1.1/0003:046D:C547.0002/input/input3
[ 1.309869] usb 5-2: New USB device found, idVendor=05e3, idProduct=0610, bcdDevice=68.75
[ 1.309874] usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1.309875] usb 5-2: Product: USB2.1 Hub
[ 1.309876] usb 5-2: Manufacturer: GenesysLogic
[ 1.316510] hid-generic 0003:046D:C547.0002: input,hidraw1: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-0000:02:00.0-1/input1
[ 1.320702] hid-generic 0003:046D:C547.0003: hiddev96,hidraw2: USB HID v1.11 Device [Logitech USB Receiver] on usb-0000:02:00.0-1/input2
[ 1.374405] hub 5-2:1.0: USB hub found
[ 1.374882] hub 5-2:1.0: 2 ports detected
[ 1.438174] usb 6-2: new SuperSpeed USB device number 3 using xhci_hcd
[ 1.452389] usb 6-2: New USB device found, idVendor=05e3, idProduct=0620, bcdDevice=68.75
[ 1.452394] usb 6-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1.452395] usb 6-2: Product: USB3.1 Hub
[ 1.452396] usb 6-2: Manufacturer: GenesysLogic
[ 1.453270] tsc: Refined TSC clocksource calibration: 3393.619 MHz
[ 1.453277] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x30eac2c7e4f, max_idle_ns: 440795371804 ns
[ 1.453490] clocksource: Switched to clocksource tsc
[ 1.469974] hub 6-2:1.0: USB hub found
[ 1.470245] hub 6-2:1.0: 2 ports detected
[ 1.493582] usb 1-5: new full-speed USB device number 3 using xhci_hcd
[ 1.548366] usb 5-1.1: new high-speed USB device number 4 using xhci_hcd
[ 1.557382] ata5: failed to resume link (SControl 0)
[ 1.557398] ata5: SATA link down (SStatus 0 SControl 0)
[ 1.557572] scsi 8:0:0:0: Direct-Access ATA ST4000DM004-2CV1 0001 PQ: 0 ANSI: 5
[ 1.557740] sd 8:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[ 1.557742] sd 8:0:0:0: [sdc] 4096-byte physical blocks
[ 1.557746] sd 8:0:0:0: [sdc] Write Protect is off
[ 1.557748] sd 8:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 1.557753] sd 8:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[ 1.557761] sd 8:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[ 1.574089] sd 8:0:0:0: [sdc] Attached SCSI disk
[ 1.574215] /dev/root: Can’t open blockdev
[ 1.574225] VFS: Cannot open root device “UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f” or unknown-block(0,0): error -6
[ 1.574226] Please append a correct “root=” boot option; here are the available partitions:
[ 1.574228] 0800 234431064 sda
[ 1.574230] driver: sd
[ 1.574231] 0810 488386584 sdb
[ 1.574232] driver: sd
[ 1.574233] 0811 307200 sdb1 f7175546-1a45-4da1-b418-41449ada4639
[ 1.574234]
[ 1.574235] 0812 488077303 sdb2 45b04130-c043-4fcc-ad3e-bc25da5b6571
[ 1.574235]
[ 1.574237] 0820 3907018584 sdc
[ 1.574237] driver: sd
[ 1.574238] List of all bdev filesystems:
[ 1.574238] ext3
[ 1.574239] ext2
[ 1.574239] ext4
[ 1.574239] fuseblk
[ 1.574240] btrfs
[ 1.574240]
[ 1.574241] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 1.574247] CPU: 14 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.4-zen2-1-zen #1 PREEMPT(full) 5cee63b14794fdbaa08757cd329f0e5276ec5728
[ 1.574251] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P5.60 10/20/2022
[ 1.574253] Call Trace:
[ 1.574255]
[ 1.574256] dump_stack_lvl+0x5d/0x80
[ 1.574262] vpanic+0xc4/0x2a0
[ 1.574266] panic+0x6b/0x6b
[ 1.574268] mount_root_generic+0x1cf/0x270
[ 1.574273] prepare_namespace+0x1dc/0x230
[ 1.574275] kernel_init_freeable+0x27f/0x2b0
[ 1.574277] ? __pfx_kernel_init+0x10/0x10
[ 1.574281] kernel_init+0x1a/0x140
[ 1.574283] ret_from_fork+0x1c1/0x1f0
[ 1.574286] ? __pfx_kernel_init+0x10/0x10
[ 1.574288] ret_from_fork_asm+0x1a/0x30
[ 1.574292]
[ 1.574421] Kernel Offset: 0xea00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
```

Did you update /etc/fstab, /etc/default/grub, run sudo dracut-rebuild and sudo grub-mkconfig -o /boot/grub/grub.cfg?

I did update /etc/fstab mostly to disable the ext4 /boot.

/et/default/grub to disabe the saved option since it won’t work on BtrFS. Otherwise it has no actual needed changes.

I did not do the dracut-rebuild, but, I think I might have the issue. I previously added uefi=yes to my dracut.d and hadn’t rebooted since. I’m adding to my drivers.conf nvme ahci since maybe for some reason that didn’t get included. And about to try again.

The one major issue I’m seeing too, is since moving /boot from ext4 to btrfs, grub-mkconfig running grub-btrfs is showing additionally this issue:

Detecting snapshots ...
No snapshots found.

Okay. Booting is back in business. But grub is still failing to add snapshots.

Is it only existing snapshots or if you take a new snapshots is it also not seeing it?

Well, I did a simple “re-install” of grub-btrfs, and accordingly, it seems to be using a different setup for snapshots, because now I have snapshots, but, they’re not matching up with snapper.

Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2025-10-24 14:12:36 | .snapshots/1115/snapshot           | post | grub-btrfs                                                               |
Found snapshot: 2025-10-24 14:12:35 | .snapshots/1114/snapshot           | pre  | pacman -S grub-btrfs                                                     |
Found 2 snapshot(s)

So, that’s incredibly weird..

The full snapper ls shows it all, though:

   # │ Type   │ Pre # │ Date                        │ User │ Cleanup │ Description                                                              │ Userdata
─────┼────────┼───────┼─────────────────────────────┼──────┼─────────┼──────────────────────────────────────────────────────────────────────────┼─────────
   0 │ single │       │                             │ root │         │ current                                                                  │
1061 │ post   │  1060 │ Mon 06 Oct 2025 11:08:01 PM │ root │         │ plasma5-integration                                                      │
1100 │ pre    │       │ Tue 21 Oct 2025 02:38:46 PM │ root │ number  │ pacman --sync -y -u --                                                   │
1101 │ post   │  1100 │ Tue 21 Oct 2025 02:42:07 PM │ root │ number  │ atuin azahar-git bat blender bluetui boost-libs chezmoi citron-git clama │
1102 │ pre    │       │ Tue 21 Oct 2025 02:59:34 PM │ root │ number  │ pacman -U /var/lib/repo/aur/linux-zen-6.17.2.zen1-1-x86_64.pkg.tar.zst   │
1103 │ post   │  1102 │ Tue 21 Oct 2025 03:00:22 PM │ root │ number  │ linux-zen                                                                │
1104 │ pre    │       │ Tue 21 Oct 2025 04:33:16 PM │ root │ number  │ pacman --remove -s -- kernel-modules-hook kernel-rotate-hook             │
1105 │ post   │  1104 │ Tue 21 Oct 2025 04:33:17 PM │ root │ number  │ kernel-modules-hook kernel-rotate-hook                                   │
1106 │ pre    │       │ Tue 21 Oct 2025 04:44:34 PM │ root │ number  │ pacman --sync -y -u --                                                   │
1107 │ post   │  1106 │ Tue 21 Oct 2025 04:45:16 PM │ root │ number  │ linux-zen qownnotes                                                      │
1108 │ pre    │       │ Tue 21 Oct 2025 10:37:34 PM │ root │ number  │ pacman --sync -- linux-zen-headers                                       │
1109 │ post   │  1108 │ Tue 21 Oct 2025 10:37:58 PM │ root │ number  │ linux-zen-headers                                                        │
1110 │ pre    │       │ Wed 22 Oct 2025 11:08:08 AM │ root │ number  │ pacman --sync -- zoom                                                    │
1111 │ post   │  1110 │ Wed 22 Oct 2025 11:08:09 AM │ root │ number  │ qt5-location qt5-remoteobjects qt5-webchannel qt5-webengine zoom         │
1112 │ pre    │       │ Thu 23 Oct 2025 10:14:13 PM │ root │ number  │ pacman --sync -- kcharselect                                             │
1113 │ post   │  1112 │ Thu 23 Oct 2025 10:14:13 PM │ root │ number  │ kcharselect                                                              │
1114 │ pre    │       │ Fri 24 Oct 2025 02:12:35 PM │ root │ number  │ pacman -S grub-btrfs                                                     │
1115 │ post   │  1114 │ Fri 24 Oct 2025 02:12:36 PM │ root │ number  │ grub-btrfs                                                               │

Is it perhaps because your older snapshots aren’t bootable since they are missing the kernels/initrds?

I suppose that very much could be what it is. And.. If so, that’ll be fine, I suppose. LOL

I’ll make a new manual snapshot point from here, and see how this goes. I think the missing part was mostly dracut, and the uefi=yes change that was never tested.

The last issue I can see is this weirdness.. I have to press a key every time I boot now.

1 Like

And.. Fixed.. It was the GRUB_SAVEDEFAULT=true still being set to true.

1 Like

Of course, the fun part of all this now, the desktop was easy….

The tabtop (Lenovo Yoga laptop).. Had a similar setup, except uses LUKSv2 outside of /boot and /boot/efi….

1 Like

Just to add my two ¢:

I had a setup very, very similar to yours, modeled according to OpenSuse layout, with snapper running and making the snapshots.

I went back to a simplified version some time ago because the complexity of this setup was too much to deal with over time.

It would have made sense with maybe 20+ machines and working with them daily.

But having this on only one computer, sometimes forgetting what I did for setup when I didn’t look into it for some time, was too much. Maintenance proved to be too complex and time-consuming.

Now, I keep separate partitions for / and /home, maybe separate /boot and /efi on some systems, and that’s about it. Regular partition backups have replaced snapper.

It’s simpler, and fits my needs. I’m curious to hear about your experience one or two years from now. Too bad that the forum doesn’t have a RemindMe!-Bot…