Trouble booting after non-booting GRUB repair

I’m trying to repair grub and get my system booting after booting into Windows and a Windows update seemed to modify my boot partition.

I have used Arch-chroot and I think I have mounted items properly. Here is the output of lsblk -f

[liveuser@eos-2024.04.20 ~]$ lsblk -f
NAME        FSTYPE   FSVER          LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0       squashfs 4.0                                                                  0   100% /run/archiso/airootfs
sda                                                                                                
├─sda1      exfat    1.0            Ventoy     958E-562A                                           
│ └─ventoy  iso9660  Joliet Extensi EOS_202404 2024-04-20-15-57-10-00                     0   100% /run/archiso/bootmnt
└─sda2      vfat     FAT16          VTOYEFI    223C-F3F8                                           
sdb                                                                                                
nvme0n1                                                                                            
├─nvme0n1p1 vfat     FAT32                     76C9-FD48                                           
├─nvme0n1p2                                                                                        
├─nvme0n1p3 ntfs                               D882CECC82CEAE76                                    
├─nvme0n1p4 ntfs                               7E109294109252CF                                    
├─nvme0n1p5 swap     1                         19346681-a6a2-4e2f-a055-8960d6d9694c                
├─nvme0n1p6 vfat     FAT32                     DA76-2352                                           
├─nvme0n1p7 ext4     1.0                       06b152cd-5746-4829-8a87-efe539552b1f                
└─nvme0n1p8 btrfs                              2f473d01-c1ea-4602-91bf-dda83777f813  171.1G    53% /mnt

I have run the following mount commands as nvme0n1p6 should be /boot/efi and nvme0np7 should be /boot

sudo mount -o subvol=@ /dev/nvme0n1p8 /mnt
sudo mount /dev/nvme0n1p7 /mnt/boot
sudo mount /dev/nvme0n1p6 /mnt/boot/efi

Then I sudo arch-chroot /mnt
When in in chroot, I have run the following from Repair a non-booting GRUB.

[root@EndeavourOS /]# grub-install
Installing for x86_64-efi platform.
Installation finished. No error reported.
[root@EndeavourOS /]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found background: /usr/share/endeavouros/splash.png
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/intel-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot:  intel-ucode.img initramfs-linux-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
ERROR: mkdir /var/lock/dmraid
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2024-06-06 18:00:00 | timeshift-btrfs/snapshots/2024-06-06_18-00-00/@ | daily            | N/A                                           |
Found snapshot: 2024-06-05 18:00:00 | timeshift-btrfs/snapshots/2024-06-05_18-00-00/@ | daily            | N/A                                           |
Found snapshot: 2024-06-04 22:00:00 | timeshift-btrfs/snapshots/2024-06-04_22-00-00/@ | weekly           | N/A                                           |
Found snapshot: 2024-06-04 18:00:01 | timeshift-btrfs/snapshots/2024-06-04_18-00-01/@ | daily            | N/A                                           |
Found 4 snapshot(s)
Unmount /tmp/grub-btrfs.7Ong3Kyd4c .. Success
done

However when booting, I get an error from my boot partition and I boot into emergency mode. Within emergency mode, I get the following output:

× boot.mount - /boot
     Loaded: loaded (/etc/fstab; generated)
     Active: failed (Result: exit-code) since Fri 2024-06-14 19:05:18 EDT; 1min 15s ago
      Where: /boot
       What: /dev/disk/by-uuid/06b152cd-5746-4829-8a87-efe539552b1f
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
        CPU: 3ms

Jun 14 19:05:19 arch-t470s mount[536]: mount: /boot: unknown filesystem type 'ext4'.
Jun 14 19:05:19 arch-t470s mount[536]:        dmesg(1) may have more information after failed mount system cal
l.
Jun 14 19:05:18 arch-t470s systemd[1]: boot.mount: Directory /boot to mount over is not empty, mounting anyway
.
Jun 14 19:05:18 arch-t470s systemd[1]: Mounting /boot...
Jun 14 19:05:18 arch-t470s systemd[1]: boot.mount: Mount process exited, code=exited, status=32/n/a
Jun 14 19:05:18 arch-t470s systemd[1]: boot.mount: Failed with result 'exit-code'.
Jun 14 19:05:18 arch-t470s systemd[1]: Failed to mount /boot.

Did I potentially follow a step incorrectly?

When in chroot, I’m not able to see ext4 support.

[root@EndeavourOS /]# modprobe -v ext4
modprobe: FATAL: Module ext4 not found in directory /lib/modules/6.8.7-arch1-1

How would ext4 support drop from the kernel?

Potentially.
You skipped checking fstab, after mounting main partition (/)

cat /mnt/etc/fstab (to check the mount point of your ESP)

It seems /boot was not (correctly?) mounted, so grub installation must have gone wrong (?).

Directory /boot to mount over is not empty, 

Also check boot contents before mounting the boot partition.

ls /mnt/boot

Are you sure you use grub for bootloader, and not systemd-boot ?

I’m positive I use grub as I setup btrfs snapshots with grub. You are right that I didn’t post my output from /etc/fstab

[liveuser@eos-2024.04.20 /]$ cat /mnt/etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=DA76-2352                            /boot/efi      vfat    fmask=0137,dmask=0027 0 2
UUID=06b152cd-5746-4829-8a87-efe539552b1f /boot          ext4    noatime    0 2
UUID=2f473d01-c1ea-4602-91bf-dda83777f813 /              btrfs   subvol=/@,noatime,compress=zstd 0 0
UUID=2f473d01-c1ea-4602-91bf-dda83777f813 /home          btrfs   subvol=/@home,noatime,compress=zstd 0 0
UUID=2f473d01-c1ea-4602-91bf-dda83777f813 /var/cache     btrfs   subvol=/@cache,noatime,compress=zstd 0 0
UUID=2f473d01-c1ea-4602-91bf-dda83777f813 /var/log       btrfs   subvol=/@log,noatime,compress=zstd 0 0
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0

The UUIDs seem to match to how I mounted.

Try again to chroot (exit chroot, if you are in) and this time

  1. Add -v to mount command to see more details
  2. After mounting root (/), check /boot/ for contents
  3. If /boot is empty, continue mounting (with -v), unless errors.

Is it possible that BIOS was reset, and you don’t have disks to AHCI?

I think you can also mount only root, chroot, and then run mount -v -a to mount all disks inside fstab.
It would give useful output.

This may be needed, in case you have to update your system, for good measures.

pacman -Syyu

So I think that may be it. When just mounting /, boot is NOT empty. Should I empty it and then mount the other partitions?

[liveuser@eos-2024.04.20 /]$ sudo mount -v -o subvol=@ /dev/nvme0n1p8 /mnt
mount: /dev/nvme0n1p8 mounted on /mnt.
[liveuser@eos-2024.04.20 /]$ ls /mnt/boot
grub

I don’t think so. This BIOS on the thinkpad doesn’t support changing disks to AHCI.

This folder must be from a failed/wrong action. You should delete it. Just have a look at the time stamps of the contents for better understanding.

If there is an integrated raid card/chip, there must be an option to disable it. Look for it .
TBO, I’ve never seen this error during grub operations. Why and who would want to create a lock file for dmraid?

Edit: In chroot, check for dmraid in grub files

grep dmraid /etc/grub.d/*

I cleared out /mnt/boot without the boot partition mounted then mounted /mnt/boot and /mnt/boot/efi again and then reinstalled grub. I got the same error on dmraid

ERROR: mkdir /var/lock/dmraid

When running grep dmraid /etc/grub.d/* I do not get any output.

[root@EndeavourOS grub.d]# grep dmraid /etc/grub.d/*
[root@EndeavourOS grub.d]#

I went over every option in the BIOS and I don’t see anything for defining the HDD mode or anything about a raid. It may be helpful but I had this nvme drive in a ThinkPad t470s and tried to make a windows bootable USB for my daughter as I was migrating to a T480s. The boot partition got messed up on the T470s prior to my repair and I moved this nvme drive over to the T480s. I also checked the BIOS on the T470s and I don’t see anything related to HDD mode or raid.

Should I maybe just run pacman -S linux linuxheaders and then run grub-mkconfig -o /boot/grub/grub.cfg again?

No, it does not seem a problem right now, unless we see later that it is.
Were there any errors in the rest of the commands?

Only error I can see is the dmraid error.

[liveuser@eos-2024.04.20 ~]$ sudo mount -v -o subvol=@ /dev/nvme0n1p8 /mnt
mount: /dev/nvme0n1p8 mounted on /mnt.
[liveuser@eos-2024.04.20 ~]$ sudo mount -v /dev/nvme0n1p7 /mnt/boot
mount: /dev/nvme0n1p7 mounted on /mnt/boot.
[liveuser@eos-2024.04.20 ~]$ sudo mount -v /dev/nvme0n1p6 /mnt/boot/efi
mount: /dev/nvme0n1p6 mounted on /mnt/boot/efi.
[liveuser@eos-2024.04.20 ~]$ sudo arch-chroot /mnt
[root@EndeavourOS /]# grub-install
Installing for x86_64-efi platform.
Installation finished. No error reported.
[root@EndeavourOS /]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found background: /usr/share/endeavouros/splash.png
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/intel-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot:  intel-ucode.img initramfs-linux-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
ERROR: mkdir /var/lock/dmraid
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Found snapshot: 2024-06-06 18:00:00 | timeshift-btrfs/snapshots/2024-06-06_18-00-00/@ | daily            | N/A                                           |
Found snapshot: 2024-06-05 18:00:00 | timeshift-btrfs/snapshots/2024-06-05_18-00-00/@ | daily            | N/A                                           |
Found snapshot: 2024-06-04 22:00:00 | timeshift-btrfs/snapshots/2024-06-04_22-00-00/@ | weekly           | N/A                                           |
Found snapshot: 2024-06-04 18:00:01 | timeshift-btrfs/snapshots/2024-06-04_18-00-01/@ | daily            | N/A                                           |
Found 4 snapshot(s)
Unmount /tmp/grub-btrfs.d4VXSCbEKG .. Success
done

Test with reboot :wink:
Exit from chroot first!!!

I’m sad to report that the problem still exists and I boot into emergency mode. The same error exists for systemctl status boot.mount

× boot.mount - /boot
     Loaded: loaded (/etc/fstab; generated)
     Active: failed (Result: exit-code) since Fri 2024-06-14 22:19:46 EDT; 1min 10s ago
      Where: /boot
       What: /dev/disk/by-uuid/06b152cd-5746-4829-8a87-efe539552b1f
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
        CPU: 4ms

Jun 14 22:19:46 arch-t470s mount[531]: mount: /boot: unknown filesystem type 'ext4'.
Jun 14 22:19:46 arch-t470s mount[531]:        dmesg(1) may have more information after failed mount system call.
Jun 14 22:19:46 arch-t470s systemd[1]: Mounting /boot...
Jun 14 22:19:46 arch-t470s systemd[1]: boot.mount: Mount process exited, code=exited, status=32/n/a
Jun 14 22:19:46 arch-t470s systemd[1]: boot.mount: Failed with result 'exit-code'.
Jun 14 22:19:46 arch-t470s systemd[1]: Failed to mount /boot.

I’m wondering if I’m really missing ext4 support

[root@EndeavourOS /]# modprobe -v ext4
modprobe: FATAL: Module ext4 not found in directory /lib/modules/6.8.7-arch1-1

Actually after posting this, the directory /lib/modules/6.8.7-arch1-1 doesn’t exist. Only /lib/modules/6.6.2-arch1-1

Chroot again, mount all system partitions and properly update the system.

You haven’t mentioned any problem with pacman update process or anything similar. Age problems? :older_adult: :rofl: It happens to me all the time! :cry:

1 Like

Thanks so I tried what you just sent which is to mount only / (root) chroot and then mount -v -a

Output is:

[root@EndeavourOS /]# mount -v -a
mount: /boot/efi: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
/boot                    : successfully mounted
/                        : ignored
/home                    : successfully mounted
/var/cache               : successfully mounted
/var/log                 : successfully mounted
/tmp                     : successfully mounted

It looks like /boot/efi won’t mount this way and I see no errors in the dmesg output. I checked the UUID label and it does match to the output of lsblk -f in the live cd console.

Funny enough if I run it again, it mounts successfully:

[root@EndeavourOS /]# mount -v -a
/boot/efi                : successfully mounted
/boot                    : already mounted
/                        : ignored
/home                    : already mounted
/var/cache               : already mounted
/var/log                 : already mounted
/tmp                     : already mounted

:rofl:
It’s just because ESP (/boot/efi) was first in (fstab) order, when /boot was not already mounted, and so the mount point (folder) was missing . After /boot was mounted (which has an efi subfolder), the mount point was present.

Confirmed. I changed the order in /etc/fstab unmounted and remounted everything and it mounted fine. I had a small hope that this would fix the booting issue and it did not. Same error as before after reboot.

I also attempted to run pacman -Syuu, but many packages are failing the integrity check with pgp failures. I updated archlinux-keyring and removed timeshift-autosnap (as btrfs snapshots were failing) and ran pacman -Syuu successfully. Going to reboot now with fingers crossed.

1 Like

Success! After a successful run of pacman -Syuu, the laptop now boots successfully.

Thank you @petsam I owe you a beer and/or coffee!

1 Like

Coffee is fine, although tsipouro at a nice Greek tavern would be awesome! :beach_umbrella:

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.