EndOS encrypted BTRFS, swap partition with snapper, hibernation and multiboot

droid · April 4, 2024, 7:59pm

Hi there,
since i got a bigger SSD for my notebook i want to do a clean fresh install of endeavourOS with encrypted BTRFS, swap partition, snapper and hibernation in multiboot with windows 11.

For now i use encrypted ext4, hibernate, timeshift and multiboot.

Before i really install on real system i tested ob VM (without multiboot). I installed endeavourOS with BTRFS and encryption using calamares.
After booting into the system i installed snapper snap-pac grub-btrfs. Enabled grub-btrfs, snapper-timeline and snapper-cleanup.

After setting btrfs-default volume to my 256, taking a snapshot, update grub using grub-mkconfig i tried to boot up into a snapshot and got

Failed to start Remount Root and Kernel File Systems.

System won’t start.

After a little research i tested out a couple of things:

outcomment the / mount line on /etc/fstab → The Failed to start Error disapears, but systemsboot stucks
creating 95-bootbackup_pre.hook and 95-bootbackup_post.hook like mentioned at https://wiki.archlinux.org/title/System_backup#Snapshots_and_/boot_partition → nothing changed
removed the “resume=…” argument in grub-menu from the snapshot → nothing changed

any idea where the problem is?

dalto · April 4, 2024, 8:15pm

There is no need to do this. It won’t help anything.

You don’t need to do this if you are using grub.

Using snapper or manually?

Do you have the eos-dracut package installed?

droid · April 4, 2024, 8:30pm

Ah okay. Watched a youtube-video for handling btrfs subvolumes, they did it. Thats why i did it to.

Okay.

I tried both.

No, endeavouros/eos-dracut is not installed, but extra/dracut is. I ran dracut-rebuild.

dalto · April 4, 2024, 8:42pm

That seems pretty strange. Can you share the output of pacman -Q | grep -E "dracut|grub|mkinitcpio"

droid · April 4, 2024, 9:46pm

Through testing i had to delete the VM and create it again. This time i did not change btrfs default subvolume and i did not create the 95-bootbackup-hooks.

It seems that i did have eos-dracut installed:

dracut 059-6
eos-dracut 1.3-1
grub 2:2.12-2
grub-btrfs 4.13-1

Maybe it is important:
I also changed the /etc/calamares/modules/mount.conf before install. in section btrfsSubvolumes i changed

/var/cache to /.snapshots; 
/@cache to /@snapshots
/var/log to /var
/@log to /@var

After install i did

umount /.snapshots
rm -r /.snapshots
snapper -c root create-config /
btrfs subvolume delete /.snapshots
mkdir /.snapshots
mount -a

Now after creating snapshots. One via pre and post pacman

pacman -S neofetch

And one with

snapper -c root create -d “BASE”
grub-mkconfig -o /boot/grub/grub.cfg

and try to boot from them i get the Failed to start Remount Root and Kernel File Systems Message and boot stuck at Reached target Graphical Interface.

dalto · April 4, 2024, 9:59pm

You shouldn’t do this on an Arch-based system. /var should be part of the root since pacman stores files there that need to be in-sync with the root.

Can I make a suggestion? Don’t make any changes. Just install EndeavourOS using btrfs, encryption and the default subvolumes it creates. Then install snapper, snap-pac, grub-btrfs and inotify-tools. Lastly systemctl enable --now grub-btrfsd

Then just configure snapper as normal without that other nonsense.

snapper -c root create-config /

droid · April 4, 2024, 10:21pm

dalto:

droid:
/var/log to /var
/@log to /@var
You shouldn’t do this on an Arch-based system. /var should be part of the root since pacman stores files there that need to be in-sync with the root.

Can I make a suggestion? Don’t make any changes. Just install EndeavourOS using btrfs, encryption and the default subvolumes it creates. Then install snapper, snap-pac, grub-btrfs and inotify-tools. Lastly systemctl enable --now grub-btrfsd

Then just configure snapper as normal without that other nonsense.
snapper -c root create-config /

Thanks for your help so far and thanks for your suggestion which i followed.

I installed endeavouros without any changed an just activated encryption, btrfs and swap with hibernation. After install OS i did

yay -S snapper snap-pac grub-btrfs inotify-tools
sudo systemctl enable --now grub-btrfsd
sudo snapper -c root create-config /
yay -S neofetch
sudo snapper -c root create -d “BASE”
sudo grub-mkconfig -o /boot/grub/grub.cfg

I am still not able to boot any snapshot (pre and post neofetch, BASE).

Still

Failed to start Remount Root and Kernel File Systems Message

and boot stuck at Reached target Graphical Interface.

dalto · April 4, 2024, 11:45pm

Can you share the output of findmnt --real

droid · April 4, 2024, 11:53pm

findmnt --real
TARGET SOURCE                   FSTYPE   OPTIONS
/      /dev/mapper/luks-626004dd-6815-464c-8b9b-2ece97f04b62[/@]
│                               btrfs    rw,noatime,compress=zstd:3,space_cache=v2,subvolid=256,subvol=/@
├─/home
│      /dev/mapper/luks-626004dd-6815-464c-8b9b-2ece97f04b62[/@home]
│                               btrfs    rw,noatime,compress=zstd:3,space_cache=v2,subvolid=257,subvol=/@home
├─/run/user/1000/doc
│      portal                   fuse.por rw,nosuid,nodev,relatime,user_id=1000,group_id=1001
├─/var/log
│      /dev/mapper/luks-626004dd-6815-464c-8b9b-2ece97f04b62[/@log]
│                               btrfs    rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/@log
└─/var/cache
       /dev/mapper/luks-626004dd-6815-464c-8b9b-2ece97f04b62[/@cache]
                                btrfs    rw,noatime,compress=zstd:3,space_cache=v2,subvolid=258,subvol=/@cache

btrfs subvolume list /
ID 256 gen 65 top level 5 path @
ID 257 gen 65 top level 5 path @home
ID 258 gen 40 top level 5 path @cache
ID 259 gen 64 top level 5 path @log
ID 260 gen 19 top level 256 path var/lib/portables
ID 261 gen 19 top level 256 path var/lib/machines
ID 262 gen 45 top level 256 path .snapshots
ID 263 gen 34 top level 262 path .snapshots/1/snapshot
ID 264 gen 35 top level 262 path .snapshots/2/snapshot
ID 265 gen 44 top level 262 path .snapshots/3/snapshot

droid · April 6, 2024, 9:04am

//Edit:
Okay i don’t really know what to do next. I tested out a little bit:

created a new EndveavourOS VM without swap-partition, just btrfs and encryption → same error
created a new EndveavourOS VM with swap-partition but without encryption → same error
I installed on a real notebook (swap-partition, btrfs, encryption) → same error

//Edit2:
4. created a new EndveavourOS VM without swap-partition, without encryption just btrfs → same error

droid · April 6, 2024, 11:54am

Looking for a solution i found https://bbs.archlinux.org/viewtopic.php?id=178329

Tested to comment out the / mount of /etc/fstab. The error while booting a snapshot disapperas. But boot still stuck at

[OK] Startet Simple Desktop Display Manager.
[OK] Reached target Graphical Interface.

droid · April 6, 2024, 7:39pm

Okay, after testing out 1000 things i guess now the main problem ist display manager.
Neither sddm nor light-dm are able to start a snapshot.

Failed to read display number from pipe
Could not start Display server on vt 1

Found no solution for this right now.

dalto · April 6, 2024, 7:41pm

Just to be sure, you didn’t remove eos-dracut as part of your many troubleshooting steps, correct?

It is needed to properly boot a read-only snapshot.

droid · April 6, 2024, 8:09pm

No i didn’t removed it.
For almost every test i create a new VM with a new install. I also test on a real notebook to make sure its not a VM related problem.

But even on real notebook i get the same results as on the VMs.

So right now i don’t really have a clue what to do next. I also tested a installation with timeshift. With timeshift created snapshots i am able to boot into them - but timeshift-autosnap won’t work. And because snapper is the recommended i really like to get this to work.

Running dracut-rebuild gets me booting right into a grub-shell.

BluishHumility · April 6, 2024, 11:30pm

Can you try switching to the LTS kernel and see if you get a different result? I haven’t looked into it too much yet but I do see this recent issue here:

github.com/Antynea/grub-btrfs

Unable to boot into a snapshot since kernel 6.8 (read-only), but i can with 6.6

opened 03:35AM - 04 Apr 24 UTC

HanM23

Hi, I cannot boot into a snapshot since kernel 6.8. I have two kernels instal…led in my system 6.6 lts and 6.8. Presets for both are exactly the same, and i can boot into any snapshot if i choose kernel 6.6 in the grub menu after i have chosen the snapshot itself. If i boot on a snapshot with kernel 6.8, it does not want to start KDE, i can switch with ctrl+alt F<number> to tty, i can then login in, and when i want to start KDE with startx i have the following message : `Can't move /var/log/Xorg.0.log to /var/log/Xorg.0.log.old`. This is surprising because` /var/log` is a subvolume, and i do not make any snapshot of it, only the subvolume /. By checking the logs, yes, my snapshot's filesystem is read-only, therefore i cannot boot to kde. But i really do not understand why, Once again i do not have this issue at all with 6.6, i can boot my snapshots normally with it. I do not know really how to investigate more. Do you have any nice idea ? I have Manjaro/KDE with Snapper and Btrfs-Assistant. Thank you *Here is the hook line from mkinitcpio.conf : `HOOKS=(base udev autodetect microcode kms modconf block keyboard keymap consolefont plymouth encrypt filesystems grub-btrfs-overlayfs)` *Here is my config ``` #!/usr/bin/env bash GRUB_BTRFS_VERSION=4.12-master-2023-04-28T16:26:00+00:00 # Disable grub-btrfs. # Default: "false" #GRUB_BTRFS_DISABLE="true" # Name appearing in the Grub menu. # Default: "Use distribution information from /etc/os-release." #GRUB_BTRFS_SUBMENUNAME="Arch Linux snapshots" # Custom title. # Shows/Hides "date" "snapshot" "type" "description" in the Grub menu, custom order available. # Default: ("date" "snapshot" "type" "description") #GRUB_BTRFS_TITLE_FORMAT=("date" "snapshot" "type" "description") # Limit the number of snapshots populated in the GRUB menu. # Default: "50" #GRUB_BTRFS_LIMIT="50" # Sort the found subvolumes by "ogeneration" or "generation" or "path" or "rootid". # # See Sorting section to https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-subvolume#SUBCOMMAND # "-rootid" means list snapshot by new ones first. # Default: "-rootid" #GRUB_BTRFS_SUBVOLUME_SORT="+ogen,-gen,path,rootid" # Show snapshots found during run "grub-mkconfig" # Default: "true" #GRUB_BTRFS_SHOW_SNAPSHOTS_FOUND="false" # Show Total of snapshots found during run "grub-mkconfig" # Default: "true" #GRUB_BTRFS_SHOW_TOTAL_SNAPSHOTS_FOUND="true" # By default, "grub-btrfs" automatically detects most existing kernels. # If you have one or more custom kernels, you can add them here. # Default: ("") #GRUB_BTRFS_NKERNEL=("kernel-custom" "vmlinux-custom") # By default, "grub-btrfs" automatically detects most existing initramfs. # If you have one or more custom initramfs, you can add them here. # Default: ("") #GRUB_BTRFS_NINIT=("initramfs-custom.img" "initrd-custom.img" "otherinit-custom.gz") # By default, "grub-btrfs" automatically detects most existing microcodes. # If you have one or more custom microcodes, you can add them here. # Default: ("") #GRUB_BTRFS_CUSTOM_MICROCODE=("custom-ucode.img" "custom-uc.img "custom_ucode.cpio") # Additonal kernel command line parameters that should be passed to the kernel # when booting a snapshot. # For dracut based distros this could be useful to pass "rd.live.overlay.overlayfs=1" # or "rd.live.overlay.readonly=1" to the Kernel for booting snapshots read only. # Default: "" #GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.overlayfs=1" # Comma seperated mount options to be used when booting a snapshot. # They can be defined here as well as in the "/" line inside the respective snapshots' # "/etc/fstab" files. Mount options found in both places are combined, and this variable # takes priority over `fstab` entries. # NB: Do NOT include "subvol=..." or "subvolid=..." here. # Default: "" #GRUB_BTRFS_ROOTFLAGS="space_cache,commit=10,norecovery" # Ignore specific path during run "grub-mkconfig". # Only exact paths are ignored. # e.g : if `specific path` = @, only `@` snapshot will be ignored. # Default: ("@") GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@") # Ignore prefix path during run "grub-mkconfig". # Any path starting with the specified string will be ignored. # e.g : if `prefix path` = @, all snapshots beginning with "@/..." will be ignored. # Default: ("var/lib/docker" "@var/lib/docker" "@/var/lib/docker") GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker") # Ignore specific type/tag of snapshot during run "grub-mkconfig". # For snapper: # Type = single, pre, post. # For Timeshift: # Tag = boot, ondemand, hourly, daily, weekly, monthly. # Default: ("") #GRUB_BTRFS_IGNORE_SNAPSHOT_TYPE=("") # Ignore specific description of snapshot during run "grub-mkconfig". # e.g: timeline # Default: ("") #GRUB_BTRFS_IGNORE_SNAPSHOT_DESCRIPTION=("") # By default "grub-btrfs" automatically detects your boot partition, # either located at the system root or on a separate partition or in a subvolume, # Change to "true" if your boot partition isn't detected as separate. # Default: "false" #GRUB_BTRFS_OVERRIDE_BOOT_PARTITION_DETECTION="true" # Location of the folder containing the "grub.cfg" file. # Might be grub2 on some systems. # Default: "/boot/grub" #GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2" # Location of kernels/initramfs/microcode. # Use by "grub-btrfs" to detect the boot partition and the location of kernels/initrafms/microcodes. # Default: "/boot" #GRUB_BTRFS_BOOT_DIRNAME="/boot" # Location where grub-btrfs.cfg should be saved. # Some distributions (like OpenSuSE) store those files at the snapshot directory # instead of boot. Be aware that this direcory must be available for grub during # startup of the system. # Default: $GRUB_BTRFS_GRUB_DIRNAME #GRUB_BTRFS_GBTRFS_DIRNAME="/boot/grub" # Location of the directory where Grub searches for the grub-btrfs.cfg file. # Some distributions (like OpenSuSE) store those file at the snapshot directory # instead of boot. Be aware that this direcory must be available for grub during # startup of the system. # Default: "\${prefix}" # This is a grub variable that resolves to where grub is # installed. (like /boot/grub, /boot/efi/grub) # NOTE: If variables of grub are used here (like ${prefix}) they need to be escaped # with `\` before the `$` #GRUB_BTRFS_GBTRFS_SEARCH_DIRNAME="\${prefix}" # Name/path of grub-mkconfig command, use by "grub-btrfs.service" # Might be 'grub2-mkconfig' on some systems (Fedora ...) # Default paths are /sbin:/bin:/usr/sbin:/usr/bin, # if your path is missing, report it on the upstream project. # For example, on Fedora : "/sbin/grub2-mkconfig" # You can use only name or full path. # Default: grub-mkconfig #GRUB_BTRFS_MKCONFIG=/usr/bin/grub2-mkconfig # Name of grub-script-check command, use by "grub-btrfs" # Might be 'grub2-script-check' on some systems (Fedora ...) # For example, on Fedora : "grub2-script-check" # Default: grub-script-check #GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check # Path of grub-mkconfig_lib file, use by "grub-btrfs" # Might be '/usr/share/grub2/grub-mkconfig_lib' on some systems (Opensuse ...) # Default: /usr/share/grub/grub-mkconfig_lib #GRUB_BTRFS_MKCONFIG_LIB=/usr/share/grub2/grub-mkconfig_lib # Password protection management for submenu,snapshots # Refer to the Grub documentation https://www.gnu.org/software/grub/manual/grub/grub.html#Authentication-and-authorisation # and this comment https://github.com/Antynea/grub-btrfs/issues/95#issuecomment-682295660 # # Add authorized usernames separate by comma (foo,bar) # When Grub's password protection is enabled, the superuser is authorized by default, it isn't necessary to add it # Default: "" #GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS="foo,bar" # # Disable authentication support for submenu of Grub-btrfs only (--unrestricted) # doesn't work if GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS isn't empty # Default: "false" #GRUB_BTRFS_DISABLE_PROTECTION_SUBMENU="true" ```

droid · April 7, 2024, 7:22am

Oh wow, first sign of success Thanks so far BluishHumility!

I installed 6.6.25-1-lts and removed 6.8.2, booted into LTS, created snapshots, booted into a snaphot and it works! So i guess it is a kernel-related issue (at least for a couple of people).

I wanted to test it with another kernel, 6.7 or something, but repositories only shows hardened an zen-kernels.

Apperently i have to use LTS-Kernel then. I will test it on the real notebook also later on.

So regarding to my first post in this topic the problem, not be able to boot a snapshot, has nothing to do with the original error message

Failed to start Remount Root and Kernel File Systems.

Solution for this seems to comment out the “/” mount in /etc/fstab.

journalctl -b -p err

endos-pc systemd-remount-fs[552]: /usr/bin/mount for / exited with exit status 32.
endos-pc systemd[1]: Failed to start Remount Root and Kernel File Systems.

Does is have any negative effect to comment out the line in /etc/fstab?:

#/dev/mapper… / btrfs subvol=/@,noatime,compress=zstd 0 0

Solution to boot into a snapshot as from BluishHumility referenced is to use LTS-Kernel. At least it seems to be, i will test it out today a little bit more.

BluishHumility · April 7, 2024, 2:50pm

You should not leave it like that. It sounds like there is an issue with your fstab. Have you modified it for any purpose?

This looks fine. Does the /dev/mapper part match the output from findmnt above?

droid · April 9, 2024, 5:53pm

I did not modified the /etc/fstab primary. I modified only to test if the Remount fs error still appears with the / commented out (i found this solution somewhere in arch-forum)

So i tested out a little bit:

running the original LTS kernel, no snapshot and / is not outcommented:

findmnt --real

/ /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=256,subvol=/@
├─/home /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@home] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=257,subvol=/@home
├─/run/user/1000/doc
│ portal fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=1001
├─/var/cache /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@cache] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=258,subvol=/@cache
├─/var/log /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@log] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=259,subvol=/@log

cat /etc/fstab
UUID=7ECC-2F22 /boot/efi vfat fmask=0137,dmask=0027 0 2
/dev/mapper/luks-a6b66b0d-5fb4-4a5d-a18b-f4859b35cea0 swap swap defaults 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f / btrfs subvol=/@,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /home btrfs subvol=/@home,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/cache btrfs subvol=/@cache,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/log btrfs subvol=/@log,noatime,compress=zstd 0 0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
vfat rw,relatime,fmask=0137,dmask=0027,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount

running a snapshot and / is not outcommented:

findmnt --real
/home /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@home] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=257,subvol=/@home
/run/user/1000/doc
portal fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=1001
/var/cache /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@cache] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=258,subvol=/@cache
/var/log /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@log] btrfs rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=259,subvol=/@log
/boot/efi /dev/sda1

cat /etc/fstab
UUID=7ECC-2F22 /boot/efi vfat fmask=0137,dmask=0027 0 2
/dev/mapper/luks-a6b66b0d-5fb4-4a5d-a18b-f4859b35cea0 swap swap defaults 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f / btrfs subvol=/@,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /home btrfs subvol=/@home,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/cache btrfs subvol=/@cache,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/log btrfs subvol=/@log,noatime,compress=zstd 0 0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0

running a snapshot and / is outcommented:

findmnt --real
/home /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@home] btrfs rw,noatime,ssd,space_cache=v2,subvolid=257,subvol=/@home
/run/user/1000/doc
portal fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=1001
/var/cache /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@cache] btrfs rw,noatime,ssd,space_cache=v2,subvolid=258,subvol=/@cache
/var/log /dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f[/@log] btrfs rw,noatime,ssd,space_cache=v2,subvolid=259,subvol=/@log
/boot/efi /dev/sda1

cat /etc/fstab
UUID=7ECC-2F22 /boot/efi vfat fmask=0137,dmask=0027 0 2
/dev/mapper/luks-a6b66b0d-5fb4-4a5d-a18b-f4859b35cea0 swap swap defaults 0 0
#/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f / btrfs subvol=/@,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /home btrfs subvol=/@home,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/cache btrfs subvol=/@cache,noatime,compress=zstd 0 0
/dev/mapper/luks-63e0389e-3fe6-483f-9ac2-63aee37e5e7f /var/log btrfs subvol=/@log,noatime,compress=zstd 0 0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0