So, I have seem to brick my system due to an update somehow. For exactly this I run BTRFS with Snapshots in Grub, but Grub seems to be dead? I can only boot into my UEFI, which is… unfortunate. What are my next steps from here on?
It’s hard to say without more information, but reinstalling Grub from a chroot is a good place to start.
https://discovery.endeavouros.com/system-rescue/repair-a-non-booting-grub/2021/03/
1 Like
So, after some research I asume my Kernel is somehow missing during Boot. I try to follow now Discover regarding Arch-Chroot: https://discovery.endeavouros.com/system-rescue/arch-chroot/2022/12/
First the mounting stuff, this is my fstab, I cut off the lower part (only NFS mounts):
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=BDEB-C54C /boot/efi vfat defaults,noatime 0 2
UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c / btrfs subvol=/@,defaults,noatime,compress=zstd 0 0
UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /home btrfs subvol=/@home,defaults,noatime,compress=zstd 0 0
UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /var/cache btrfs subvol=/@cache,defaults,noatime,compress=zstd 0 0
UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /var/log btrfs subvol=/@log,defaults,noatime,compress=zstd 0 0
UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /Virtual btrfs subvol=/@virtual,defaults,noatime,compress=zstd 0 0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
#local disks
UUID=fc6a43bb-ec9a-4f23-94e9-eff7cf5f250c /Data ext4 defaults,noatime 0 2
UUID=e0e1bff4-1418-4574-b3da-823cfe4fc668 /Misc ext4 defaults,noatime 0 2
UUID=545f70e0-5226-4d38-862e-cf159fbf6e7f /Stuff ext4 defaults,noatime 0 2
UUID=8c7a3875-418a-49de-8a6e-47a480315c22 /backup ext4 defaults,noatime 0 2
# NFS shares
....
As one can see I have a BTRFS mirror running. To mount it correctly I will need some help here. I asume it is this way:
sudo mount UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /mnt -o subvol=@
sudo mount UUID=BDEB-C54C /boot/efi
sudo arch-chroot /mnt
My questions: Do I need to mount the mirrored disk as well? Do I have to care for subvolumes, and if yes is this the correct approach?
Ty, was writing my other post and did not read Discover carefully, because I am a little bit in ppanikmode right now - sorry.
Will re-read your post and Discover and edit this post with what I actually think is right to perform the chroot…
So the mounting will be like:
sudo mount -o subvol=@ UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /mnt
sudo mount -o subvol=@log UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /mnt/var/log
sudo mount -o subvol=@cache UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /mnt/var/cache
sudo mount -o subvol=@home UUID=178ba18b-3faa-4b38-b7e5-5b0f5d2f285c /mnt/home
sudo mount UUID=BDEB-C54C /mnt/efi
Two questions are still in my mind:
Carrng for the mirror?
My efi is on /boot/efi, not on efi, should I mount it to /mnt/boot/efi or /mnt/efi like above and what Discover says?
Mounting the root subvolume and the EFI partition should be enough for reinstalling Grub.
You should adjust the guidance to use whatever mount point you are using, i.e. /mnt/boot/efi.
1 Like
Understood, will do so. When I am in my archroot, how do I analyze what the problem is? More specificly asked: How do I know that I have to reinstall grub?
Dont reinstall grub.
Use pacman to reinstall your kernel packages.
1 Like
So just do (in archroot) yay -S linux?
Edit: Just gave it a try, with lots of errors:
yay -S linux
-> Avoid running yay as root/sudo.
Sync Explicit (1): linux-6.6.4.arch1-1
warning: linux-6.6.4.arch1-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Package (1) Old Version New Version Net Change
core/linux 6.6.4.arch1-1 6.6.4.arch1-1 0.00 MiB
Total Installed Size: 128.77 MiB
Net Upgrade Size: 0.00 MiB
:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring [------------------------------------------------------] 100%
(1/1) checking package integrity [------------------------------------------------------] 100%
(1/1) loading package files [------------------------------------------------------] 100%
(1/1) checking for file conflicts [------------------------------------------------------] 100%
:: Running pre-transaction hooks...
(1/1) Performing snapper pre snapshots for the following configurations...
==> root: 1491
:: Processing package changes...
(1/1) reinstalling linux [------------------------------------------------------] 100%
:: Running post-transaction hooks...
(1/7) Arming ConditionNeedsUpdate...
(2/7) Updating module dependencies...
(3/7) Updating initramfs...
:: Building initramfs for linux (6.6.4-arch1-1)
dracut: Executing: /usr/bin/dracut --force --hostonly --no-hostonly-cmdline /usr/lib/modules/6.6.4-arch1-1/initrd 6.6.4-arch1-1
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: btrfs-snapshot-overlay ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: Skipping udev rule: 70-persistent-net.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode: real
dracut: Method: sha256
dracut: Files: 817
dracut: Linked: 2 files
dracut: Compared: 0 xattrs
dracut: Compared: 42 files
dracut: Saved: 822 B
dracut: Duration: 0.020011 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/usr/lib/modules/6.6.4-arch1-1/initrd' ***
dracut: *** Creating initramfs image file '/usr/lib/modules/6.6.4-arch1-1/initrd' done ***
:: Building fallback initramfs for linux (6.6.4-arch1-1)
dracut: Executing: /usr/bin/dracut --force --no-hostonly /boot/initramfs-linux-fallback.img 6.6.4-arch1-1
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoemon' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoeadm' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nbd' will not be installed, because command 'nbd-client' could not be found!
dracut: dracut module 'nvmf' depends on 'network', which can't be installed
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: crypt ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: nvdimm ***
dracut: *** Including module: qemu ***
dracut: *** Including module: qemu-net ***
dracut: *** Including module: btrfs-snapshot-overlay ***
dracut: *** Including module: lunmask ***
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: virtiofs ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode: real
dracut: Method: sha256
dracut: Files: 2027
dracut: Linked: 9 files
dracut: Compared: 0 xattrs
dracut: Compared: 455 files
dracut: Saved: 1.42 MiB
dracut: Duration: 0.017000 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/boot/initramfs-linux-fallback.img' ***
dracut: *** Creating initramfs image file '/boot/initramfs-linux-fallback.img' done ***
(4/7) Refreshing PackageKit...
Error connecting: Could not connect: No such file or directory
error: command failed to execute correctly
(5/7) Check if user should be informed about rebooting after certain system package upgrades.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
(6/7) Checking which packages need to be rebuilt
(7/7) Performing snapper post snapshots for the following configurations...
fatal library error, lookup self
==> root: 1492
I’d asume I have a dracut issue?
Looks OK to me at a glance. That fatal error you are receiving at the end is probably your snapshots failing. That shouldn’t be a problem.
What does ls -l /boot return?
As a side issue, you should not use yay in a chroot, just use pacman. That being said, what you did is fine.
1 Like
Will make n knot regarding “use pacman instead of yay in archroot”, ty!
My boot entry looks good:
[root@EndeavourOS /]# ls -l /boot
total 76348
-rw-r--r-- 1 root root 81920 Nov 22 17:01 amd-ucode.img
drwxr-xr-x 3 root root 4096 Jan 1 1970 efi
drwxr-xr-x 1 root root 196 Dec 6 14:07 grub
-rw------- 1 root root 46201799 Dec 6 14:21 initramfs-linux-fallback.img
-rw-r--r-- 1 root root 19003637 Dec 6 14:21 initramfs-linux.img
-rw-r--r-- 1 root root 12884800 Dec 6 14:21 vmlinuz-linux
Shall I reboot and check if my issue is solved? If I do, do I have to manually unmount the mounts that I did?
You don’t have to manually unmount them.
You might want to run grub-mkconfig -o /boot/grub/grub.cfg before exiting the chroot though.
1 Like
That did the trick, I am back on track, thank you both of you!
2 Likes
While I was documenting this to have a fast available reference for possible future occurrences I was wondering the following: Why did you know it is the missng / wrong installed kernel? How could I check this to know it is exactly this issue which is my root cause? And could I have done something bad to provoke this? So do I have to adapt?
You stated “my Kernel is somehow missing during Boot”. The actual kernel you boot from is actually copied to that location by a post-transaction pacman hook.
The are a few common causes of a missing kernel during boot:
- Uninstalling the kernel package
- Not having enough disk space in the partition where the kernel is copied to
- Running an update and not allowing to finish all the way (Rebooting too soon, battery dies, system fails during update, etc, etc)
The last one is the most common cause by far.
Regardless of the cause, reinstalling the kernel package should always fix that issue.
It is difficult after the fact. You could go back through the pacman log and see if you had any incomplete updates or any errors while running post transaction hooks.
It is possible you did but it may be hard to find the issue after the fact. The pacman log is probably your best bet.
1 Like
I did my yay as always, and I always reboot if I am asked to do it. I will check my pacmanlog after I have dealt with what is now not working, will kake ANOTHER topic… ty, sir!
1 Like
This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.