Linux-lts fails to mount with normal initramfs, but works with failsafe initramfs, while linux and linux-zen work without an issue

[psi-jack]

So this is absolutely wild and weird. I can’t boot linux-lts or linux-lqx for pretty much the exact same reasons, slightly differently reported. In their normal initramfs.
linux and linux-zen work fine, and linux-lts with initrd fallback works too, but not the normal initrd. (Note, I’m using dracut, not mkinitcpio).
The problem is, it’s stalling on mounting the / filesystem from nvme0n1p3, which is BtrFS, and on NVMe.
On linux-lts, it reports this as if /dev/dev/disab3e-5de13dd60e2f - on linux-lqx it’s correctly reported as /dev/disk/by-uuid/d6972f3e-0865-4440-ab3e-5de13dd60e2f, and in systemd.log_level in debug mode, on linux-lts, it even mentions this, as shown here: https://i.imgur.com/V7e9Jyj.jpeg (systemd debug_log)
https://i.imgur.com/1tb5vzD.jpeg (normal boot)

What gets me is linux, and linux-zen work fine, but a 3rd party kernel, linux-lqx has the exact same issue, however instea of the /dev/dis*, it’s actually showing the /dev/disk/by-uuid/UUID path.

For technical information, here’s fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>                           <mount point>             <type>  <options>  <dump>  <pass>
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /                         btrfs   subvol=/@,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=9316a7ca-f21a-466c-bd89-b9ef4199df77 /boot                     ext4    noatime    0 2
UUID=4A5B-032B                            /boot/efi                 vfat    fmask=0137,dmask=0027 0 2
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /home                     btrfs   subvol=/@home,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/cache                btrfs   subvol=/@cache,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/log                  btrfs   subvol=/@log,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /srv                      btrfs   subvol=/@srv,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /opt                      btrfs   subvol=/@opt,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /usr/local                btrfs   subvol=/@usrlocal,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/lib/crash            btrfs   subvol=/@crash,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/lib/containers       btrfs   subvol=/@containers,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/lib/libvirt/images   btrfs   subvol=/@libvirt,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/lib/machines         btrfs   subvol=/@machines,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/lib/portables        btrfs   subvol=/@portables,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/spool                btrfs   subvol=/@spool,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/tmp                  btrfs   subvol=/@tmp,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /var/www                  btrfs   subvol=/@www,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /root                     btrfs   subvol=/@root,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f /.snapshots               btrfs   subvol=/.snapshots,noatime,space_cache=v2,compress=zstd:1,ssd,discard=async 0 0
tmpfs                                     /tmp                      tmpfs   defaults,noatime,mode=1777 0 0

# Mounts
UUID=2114b1b9-74af-4025-878c-7cd91515faf6 /mnt/ssd                  xfs     defaults 0 0
UUID=74111e09-5c37-4244-b140-2f588f4ad070 /mnt/storage              xfs     defaults 0 0
UUID=acf970c6-711f-47ef-b24a-0fb6271956d9 /mnt/home                 ext4    defaults 0 0

# Spare OS: Fedora
UUID=462b254d-e309-47a3-960e-34e10fb63f52 /mnt/fedora-root          btrfs   subvolid=5,compress=zstd:1,noatime,space_cache=v2,ssd,discard=async 0 0
UUID=946c05ab-a723-4ee2-b40b-3346007ee6c2 /mnt/fedora-boot          ext4    defaults 0 0
UUID=2993-BBD3                            /mnt/fedora-efi           vfat    fmask=0137,dmask=0027 0 0

and grub.cfg:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod btrfs
search --no-floppy --fs-uuid --set=root d6972f3e-0865-4440-ab3e-5de13dd60e2f
    font="/@/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=1920x1080x32,auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_CA
  insmod gettext
fi
terminal_output gfxterm
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
insmod gfxmenu
loadfont ($root)/grub/themes/fallout-grub-theme/fixedsys-regular-32.pf2
insmod png
set theme=($root)/grub/themes/fallout-grub-theme/theme.txt
export theme
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
export tuned_params
set tuned_initrd=""
export tuned_initrd
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'EndeavourOS Linux' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
	echo	'Loading Linux linux-zen ...'
	linux	/vmlinuz-linux-zen root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
	echo	'Loading initial ramdisk ...'
	initrd	/initramfs-linux-zen.img
}
submenu 'Advanced options for EndeavourOS Linux' $menuentry_id_option 'gnulinux-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	menuentry 'EndeavourOS Linux, with Linux linux-zen' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-zen-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-zen ...'
		linux	/vmlinuz-linux-zen root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-zen.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux-zen (fallback initramfs)' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-zen-fallback-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-zen ...'
		linux	/vmlinuz-linux-zen root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-zen-fallback.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux-lts' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-lts ...'
		linux	/vmlinuz-linux-lts root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-lts.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux-lts (fallback initramfs)' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-fallback-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-lts ...'
		linux	/vmlinuz-linux-lts root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-lts-fallback.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux-lts.old' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts.old-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-lts.old ...'
		linux	/vmlinuz-linux-lts.old root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-lts.img.old
	}
	menuentry 'EndeavourOS Linux, with Linux linux-lqx' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lqx-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-lqx ...'
		linux	/vmlinuz-linux-lqx root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-lqx.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux-lqx (fallback initramfs)' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lqx-fallback-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux-lqx ...'
		linux	/vmlinuz-linux-lqx root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-lqx-fallback.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux ...'
		linux	/vmlinuz-linux root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux.img
	}
	menuentry 'EndeavourOS Linux, with Linux linux (fallback initramfs)' --class endeavouros --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-d6972f3e-0865-4440-ab3e-5de13dd60e2f' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 9316a7ca-f21a-466c-bd89-b9ef4199df77
		echo	'Loading Linux linux ...'
		linux	/vmlinuz-linux root=UUID=d6972f3e-0865-4440-ab3e-5de13dd60e2f rw rootflags=subvol=@  quiet splash video=1920x1080@165 bgrt_disable nowatchdog nvme_load=YES splash loglevel=3 amd_pstate=active amdgpu.ppfeaturemask=0xffffffff mitigations=off
		echo	'Loading initial ramdisk ...'
		initrd	/initramfs-linux-fallback.img
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/15_ostree ###
### END /etc/grub.d/15_ostree ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/25_bli ###
if [ "$grub_platform" = "efi" ]; then
  insmod bli
fi
### END /etc/grub.d/25_bli ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
	fwsetup --is-supported
	if [ "$?" = 0 ]; then
		menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
			fwsetup
		}
	fi
fi
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/41_snapshots-btrfs ###
if [ ! -e "${prefix}/grub-btrfs.cfg" ]; then
echo ""
else
submenu 'EndeavourOS snapshots' {
    configfile "${prefix}/grub-btrfs.cfg"
}
fi
### END /etc/grub.d/41_snapshots-btrfs ###

[dalto]

You can fix this with sudo pacman -Syuu

This will downgrade dracut to 106.

[psi-jack]

Wow. I didn’t even guess dracut would be the issue, but sure enough, rolling back to 106 does solve the issue here. for both linux-lts and linux-lqx both. Never had the issue with linux or linux-zen.

[dalto]

The issue is described in more detail here:

Issue with dracut 107 and the LTS kernel causing "failed to mount /sysroot"

There is currently a known issue with dracut 107 and the LTS kernel. When booting the system and selecting the LTS kernel, you will recieve the error “failed to mount /sysroot”. You can avoid this issue by booting the fallback initrd or a different kernel. It only manifests in the following conditions: You have both the LTS kernel and one of the other kernels installed You use both kernels You use a filesystem that is modular in linux-lts but not modular in linux such as ext4 It is worth n…

[psi-jack]

Interesting. Yeah, sadly I saw the post. But, since I didn’t match the exact scenario of the original, I didn’t read into it much.
I never got an actual error, it just simply stalled trying to mount, and I waited the usual 5 minutes you should wait. It simply just never mounted / which is btrfs, non-encrypted.
I noticed in the forum posting about adding ext4 kernel modules to the dracut modules? Is this a workaround, or is this just nothing I should be focused on?

[dalto]

@psi-jack

I noticed in the forum posting about adding ext4 kernel modules to the dracut modules? Is this a workaround, or is this just nothing I should be focused on?

I wouldn’t do that. It was a workaround but not needed if you have downgraded dracut.

[psi-jack]

Okay. Good to know.

One more thing of note, it seems based on the fact I’d tried to use my snapshots to recover and got a recovery prompt instead of a proper boot, this also effected snapshots as well. This rollback has made it possible to boot snapshots as well.