Systemd-boot with new ESP but existing / - /efi fails to mount

General system Kernel, boot, graphics & hardware
1

Hi, after I did a BIOS update and a Windows Update, I could no longer boot from my Linux ESP.
However, main problem was that Windows took ESP for itself, so I moved it (it being windows) to a different drive.

Now, trying to repair my ESP and getting everything back to normal; My boots are always failing with /efi can’t be mounted during bootup.

I chrooted like 5 times now but I don’t see what’s wrong.

What I did was: (leaving out everything unrelated, because I deleted the ESP and created a new one instead)

  1. delete the ESP volume on that drive; created a new vfat volume w/ esp and boot flags set, 500MB
  2. fired up rescue system
  3. mounted / and /efi → arch-chrooted
    4 # bootctl install and install kernels
  4. checked my UUIDs were correct
  5. rebooted.

And I don’t see what I was missing. Currently in my rescuesys, now I try and dump everything I’ve got

My symptom is:
I get a boot menu with all (4) Kernels as I was used to; Regular + Zen + fallbacks for each.
So no space issue; then when loading an entry, /efi fails to mount and I get stuck on boot when it says something to the effect of:
“root password or Ctrl + D to continue to boot”

I do NOT use secure boot or fast boot.

What bootctl says about that entry:

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/982c4491-xxxx-xxxx-xxxx-a77a55882877
         File: └─/EFI/systemd/systemd-bootx64.efi

I see my fstab has the /efi listed as UUID not PARTUUID though. I never changed that:

# <file system>                          <mount point>  <type>  <options>  <dump>  <pass>
UUID=7XXX-XXX7                            /efi           vfat    defaults,noatime 0 2

I have no clue if they are supposed to match, but I’ll do that now, until I reboot back in my live usb.

that’s this one:

/dev/nvme0n1p1: UUID="7xxx-xxx7" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="XXEFI" PARTUUID="982c4491-xxxx-xxxx-xxxx-a77a55882877"

So, my Windows ESP is now on a different disk, so I don’t expect there to be any more interventions from Windows. /root and /home get mounted before it quits no problem (it seemed) - as I did not get any error or warning.

(maybe the fstab entry solves it; but I need to reboot first)

Any ideas?

Edit:
[root@EndeavourOS EFI]

~# cat /efi/loader/entries/bc33e34161c741e9b5ff224d2170edb0-6.5.9-arch2-1.conf

# Boot Loader Specification type#1 entry
# File created by /etc/kernel/install.d/90-loaderentry.install (systemd 254.1-1-arch)
title      EndeavourOS
version    6.5.9-arch2-1
machine-id bc33e34161c741e9b5ff224d2170edb0
sort-key   endeavouros-6.5.9-arch2-1
options    nvme_load=YES nowatchdog rw root=UUID=4fa508e4-xxxx-xxxx-xxxx-f3e68d59786d systemd.machine_id=bc33e34161c741e9b5ff224d2170edb0
linux      /bc33e34161c741e9b5ff224d2170edb0/6.5.9-arch2-1/linux
initrd     /bc33e34161c741e9b5ff224d2170edb0/6.5.9-arch2-1/initrd

the xxxx786d is also my root in fstab and blkid (UUID, not PARTUUID.)

2

Does the UUID for the new ESP match the one in your /etc/fstab:

UUID=7XXX-XXX7                            /efi           vfat    defaults,noatime 0 2

?

If so, arch-chroot again in your system and run

bootctl install

reinstall-kernels

Also EnOS uses 1000MB for ESP to have enough space for the kernels.

You have four of them. Perhaps you would need to make it bigger than what it is now.

3

so here’s the kicker; the UUID did NOT match. that’s what I was referring to earlier when I said fstab shows the UUID.
HOWEVER, the UUID matched the same Volume. The UUID in fstab and the PARTUUID in ESP → both the same volume, but just different descriptors.

1: fstab

# <file system>                          <mount point>  <type>  <options>  <dump>  <pass>
UUID=7XXX-XXX7                            /efi           vfat    defaults,noatime 0 2

Bootctl:

Partition: /dev/disk/by-partuuid/982c4491-xxxx-xxxx-xxxx-a77a55882877

and blkid summarizes: they point to the same volume:

/dev/nvme0n1p1: UUID="7xxx-xxx7" <..cut..> PARTUUID="982c4491-xxxx-xxxx-xxxx-a77a55882877"

Solution steps?

However, here’s what I did differently this time:

I copied my fstab entry like so:
the meta here is; same entry, switched UUID for PARTUUID (as given in blkid)

# UUID=7XXX-XXX7    /efi       vfat    defaults,noatime 0 2
UUID=982c4491-xxxx-xxxx-xxxx-a77a55882877       /efi     vfat    defaults,noatime 0 2

and did a reboot.
This time my system did not decide to hang itself off a cliff before handing me a shell. However I got similar errors, a FS dependency for /efi was not fulfilled.

Step 2

Filled was my hope though, since I did nothing but switched the fstab entry again back to the old one like so:

UUID=7XXX-XXX7    /efi       vfat    defaults,noatime 0 2
# UUID=982c4491-xxxx-xxxx-xxxx-a77a55882877       /efi     vfat    defaults,noatime 0 2

and suddenly it just booted my system just fine.
I have no clue where the error was or how it got messed up in the first place.

–
Maybe something to share though: when I ran
~# kernel-install
for the first time, I did a
~# kernel-install --help
and my face froze as I realized, there was no help, it just jumped along and did the thing. I don’t believe there was any relationship to the issue, but I had a hard time to believe that was the expected behavior there.

All seems fine, except I don’t have a timeout or kernel-selector on boot, but I am no longer in live-rescue mode, so Ill take it from here,

thanks anyway - to anyone who already read this.
If I have a subsequent error I’ll add it here, but in a day or two this can be marked as solved.

Also EnOS uses 1000MB for ESP to have enough space for the kernels.

Thanks, I didn’t know - back when I installed my systems, EOS said to use 500 IIRC; I have never had my ESP use more than 60% space, but I’ll consider resizing. Since the Win volume is no longer present, I have more room for ESP sitting right next to it.