Dracut overlayfs missing?

after you upgrade the /etc/default/grub-btrfs/conf you have to run sudo /etc/grub.d/41_snapshot-btrfs to get the kernel parameter added to all the snapshots. That does work for me

As @dalto pointed out you need to use grub-btrfs that you get from cloning the github repository. The version in the AUR is not up to date with the latest changes to support the kernel parameter

I’m not sure, the question leads me to believe I do not.

The documentation explicitly says to set this option (or the RO version of it), but does not mention anything about a minimum version. I just installed it normally through pacman.

Gotcha, I’ll give that a shot.

I think grub-btrfs has not officially released the version with the kernel parameter stuff yet, but what you get cloning the repo has it.

Just did a quick switch of the Display Manager on an XFCE system to SDDM and now it boots and I can log in. Just like KDE. However, I did some failures on Bluetooth as xfce started up.

Now I’m going to try the grub-btrfs kernel parameter stuff

So switching xfce to sddm solves a lot of my problem. I still get the Bluetooth failure but I can run btrfs-assistant and restore to a snapshot and then reboot and the default boot is that snapshot. That’s what I’d need to recover in a non booting system.

However, having the kernel cmd line parameter for the overlay does nothing for me. I still thing something needs to be added to dracut and then the initramfs needs to be rebuilt.

I cloned the grub-btrfs repo and installed it that way, which enabled booting with the kernel parameter. However:

This was my experience as well. The environment wasn’t the actual overlay environment, it was just a read-only filesystem.

Also, LightDM still wouldn’t load but I discovered if you hit Enter when it hangs on the black screen with the sporadically blinking cursor, then you can switch to a TTY to login and run startxfce4. Weird, but okay.

I started wondering why this setup works fine on Garuda Linux, where I have an XFCE install that uses dracut/ grub-btrfs/ snapper/ snap-pac. After some poking around on the Garuda install, I found a 91btrfs-snapshot-overlay directory in /usr/lib/dracut/modules.d/, with these two scripts inside:

exa -1 /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay/
module-setup.sh
snapshot-overlay.sh

I took a peek and discovered they are just a couple small bash scripts.

cat /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay/module-setup.sh
File: /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay/module-setup.sh
#!/bin/bash

# called by dracut
check() {
    dracut_module_included btrfs || return 1
    return 0
}

# called by dracut
depends() {
    return 0
}

# called by dracut
install() {
    inst mktemp
    hostonly='' instmods overlay
    inst_hook pre-pivot 000 "$moddir/snapshot-overlay.sh"
}
cat /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay/snapshot-overlay.sh
File: /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay/snapshot-overlay.sh
#!/bin/bash
function mount_snapshot_overlay() {
    local root_mnt="$NEWROOT"
    if [[ "$(findmnt --mountpoint "$root_mnt" -o FSTYPE -n)" = "btrfs" ]] && [[ "$(btrfs property get ${root_mnt} ro)" != "ro=false" ]]; then
        local ram_dir=$(mktemp -d -p /)
        mount -t tmpfs cowspace ${ram_dir}
        mkdir -p ${ram_dir}/{upper,work}
        mount -t overlay -o lowerdir=${root_mnt},upperdir=${ram_dir}/upper,workdir=${ram_dir}/work rootfs ${root_mnt}
    fi
}

mount_snapshot_overlay

I added this directory and the two scripts to usr/lib/dracut/modules.d/ on my Endeavour XFCE and ran sudo dracut-rebuild, then installed a package to generate a snapshot for testing.

And it works! I am able to boot the snapshot, LightDM loads normally, and the environment is the proper overlay (not the read-only filesystem).

I’m not sure who made these scripts and I don’t fully understand the significance of what they do, but they do seem to resolve this particular issue.@jfabernathy I guess you were right, it seems like it’s a dracut thing after all.

@BluishHumility Thanks you for finding this out. I’ve confirmed it works now. To clear up what was done:

  1. Create fresh VM with EOS XFCE
  2. Create /usr/lib/dracut/modules.d/91btrfs-snapshot-overlay directory
  3. Create within that directory the 2 scripts you listed above: module-setup.sh and snapshot-overlay.ch
  4. Run sudo dracut-rebuild
  5. Install snapper and AUR package btrfs-assistant.
  6. Use btrfs-assistant to create snapper setup for configs root and home
  7. Clone grub-btrfs github repo and install using manual instructions. (maybe version using pacman? not tested yet)
  8. Install inotify-tools package
  9. sudo grub-mkconfig -o /boot/grub/grub.cfg
  10. sudo systemctl enable --now grub-btrfsd
  11. Install snap-pac
  12. Edit /etc/snap-pac.ini to include snapper configs other than root which is default. For me that means ‘home’.
  13. Install a package to generate pre and post snapshots. I used ‘aisleriot’

I did not do any of the grub-btrfs dracut workarounds. No variables or kernel cmd line additions.

I could boot into a pre snapshot and everything was as expected. I could even run btrfs-assistant and restore root and home snapshots for the pre snapshot I was booted into at the time. On reboot to the default image, it was missing the app associated with the pre snapshot as expected.

Note: Don’t forget to sudo rm /var/lib/pacman/db.lck before installing packages after restoring a snapshot.

I can confirm that you can use the pacman version of grub-btrfs instead of cloning the github repo version.

I also tested this solution with EOS Cinnamon.

So all that is really needed is for EOS to include the dracut modules in 91btrfs-snapshot-overlay i.e. steal them from Garuda :grinning:

Nice, I’m glad I was able to help crack the case!

Garuda has used grub-btrfs much longer than they have used dracut; maybe those scripts were written when they switched over, to preserve the familiar behavior when snapshot booting.

Perhaps a future version of dracut will address using the overlay in a way that will make using the extra scripts unneeded.

The current version of dracut has an overlayfs module.

The version in the Arch repos is out of date due to packaging dispute between dracut and the Arch maintainers about signing.

The result of that the repo version is 3 major versions out-of-date. When we last it looked into it it seemed like that was on-track to getting resolved. It seems that still hasn’t happened.

We may need to look into packaging dracut ourselves.

Out of curiosity, what is the dispute about?

If Endeavour packaged dracut themselves, how would the package signing dispute be avoided?

It has been a while since I looked at it, if I remember correctly, the issue was that at one point, the source code archives for releases was being signed. When dracut switched hosting, they stopped signing the source releases. Arch maintainers were waiting on signed source to proceed.

It isn’t package signing, it is the source. I would have two points.

  • Lots of the source for packages in the repos isn’t signed
  • The risk of running outdated code is probably higher than the using the risk of unsigned source.

You don’t need it. :stuck_out_tongue_winking_eye:

yay -Si sddm|grep Depends 
Depends On      : qt5-declarative  xorg-xauth  xorg-server  ttf-font

Looks like it was solved, but it’s not :

I see, they are waiting on a cross-signing key.

I do need it! I can’t survive without my KDE. :wink:

Funny story about how good this fix is. I tried installing EOS-Cinnamon on real hardware and got all the overlay stuff working and tested by booting into a snapshot.

And I immediately forgot I was in a snapshot and went about setting up my network bridge with nmcli. On reboot to test that I found I have no network bridge.

Then I remembered I was on a snapshot when I setup it up. Duh. I needed more coffee this morning and less whiskey.
:sweat_smile:

I don’t like this idea at all. All of the vital components of EndeavourOS come from the Arch repos. One thing is providing custom config, scripts, and hooks, but repackaging a vital OS component into the endeavouros repository means that EndeavourOS drifts away from the Arch base.

Also, where does it end? What is going to be packaged next? Systemd? Kernel? Glibc?

It isn’t something we want to do but the version in the Arch repos isn’t being updated. The version in the Arch repos is 3 versions behind and over a year old.

On the other hand, we prefer not to maintain it so if the version in the Arch repos starts getting updated we could drop the version in our repos.

Our goal here isn’t to customize the package, just to keep it maintained.

As a general rule, we try to package as little as possible. Our goal is not to repackage something unless there is a compelling reason to do so.

As an example, we sometimes package software from the AUR because it is needed to install one of our community editions.

Another package we are looking at is lsb-release. This is little more than a script and some text files that provides information about the distro. Currently, we have hooks that run that modify files after each update of the package which objectively makes little sense. We should probably just be installing our own versions of those files in the first place.