Latest grub bricked my system (grub_is_shim_lock_enabled not found)

Wait, I have to check…

I updated system including the grub package yesterday, grub from r566 to grub r591. After that system didn’t boot.
I fixed this with option --bootloader-id.

Previous system update (grub r566) was 1st of July, and the system worked with this version.

I tried with the following commands:

sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=endeavouros
sudo grub-mkconfig -o /boot/grub/grub.cfg

But no luck, had the same shim_lock error and had to resort to chrooting and restoring the previous version of GRUB via downgrade.
I noticed that i have the following situation in my /boot/efi/EFI directory:

ls -al /boot/efi/EFI               
total 16
drwxr-xr-x 4 root root 4096  5 dic  2022 .
drwxr-xr-x 3 root root 4096  1 gen  1970 ..
drwxr-xr-x 2 root root 4096  5 dic  2022 boot
drwxr-xr-x 2 root root 4096  5 dic  2022 endeavouros

Idk if having 2 folders in there is good or not tbh.

Some points to check:

  • What’s the version of the grub package installed in the system? You need to arch-chroot into it if unknown.
  • Can you show the output of ls -la /boot/efi/EFI/endeavouros?
  • What’s the output of command efibootmgr?

It shouldn’t matter if you have some more folders in /boot/efi/EFI.

  • What’s the version of the grub package installed in the system? You need to arch-chroot into it if unknown.
pacman -Qi grub | grep -i version
Version         : 2:2.06.r566.g857af0e17-1

the newer version I’m trying to install is this one:

pacman -Ss grub                  
core/grub 2:2.06.r591.g6425c12cd-1 [installed: 2:2.06.r566.g857af0e17-1]
    GNU GRand Unified Bootloader (2)
  • Can you show the output of ls -la /boot/efi/EFI/endeavouros?

Sure, here’s both endeavouros and boot:

┌──(chromiell🔒endeavour-laptop)-[~]
└─$ ls -la /boot/efi/EFI/endeavouros                             
total 360
drwxr-xr-x 2 root root   4096  5 dic  2022 .
drwxr-xr-x 4 root root   4096  5 dic  2022 ..
-rwxr-xr-x 1 root root 360448  7 lug 14.57 grubx64.efi
                                                                                                       
┌──(chromiell🔒endeavour-laptop)-[~]
└─$ ls -la /boot/efi/EFI/boot            
total 376
drwxr-xr-x 2 root root   4096  5 dic  2022 .
drwxr-xr-x 4 root root   4096  5 dic  2022 ..
-rwxr-xr-x 1 root root 376832  5 dic  2022 bootx64.efi
  • What’s the output of command efibootmgr?
┌──(chromiell🔒endeavour-laptop)-[~]
└─$ efibootmgr
BootCurrent: 0000
Timeout: 0 seconds
BootOrder: 0000,0001,2001,2002,2003
Boot0000* HDD0: HFM512GD3JX016N	PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/NVMe(0x1,AC-E4-2E-00-1A-31-11-D4)/HD(1,GPT,3fe1f943-c9cf-a244-83df-d58dfb1c5ff8,0x1000,0x96000)RC
Boot0001* endeavouros	HD(1,GPT,3fe1f943-c9cf-a244-83df-d58dfb1c5ff8,0x1000,0x96000)/File(\EFI\endeavouros\grubx64.efi)
Boot2001* EFI USB Device	RC
Boot2002* EFI DVD/CDROM	RC
Boot2003* EFI Network	RC

I might also add that I’m using BTRFS, LUKS encryption for root and swap and i currently have grub-btrfs package to boot into a snapshot.

Here’s my partition setup, but I don’t think it should make any difference :frowning:

┌──(chromiell🔒endeavour-laptop)-[~]
└─$ lsblk -f                 
NAME                 FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1                                                                                     
├─nvme0n1p1          vfat   FAT32       7A37-5497                             298,7M     0% /boot/efi
├─nvme0n1p2          crypto 1           2f9f5668-7b60-4b4e-9e93-3df9e311e38b                
│ └─luks-2f9f5668-7b60-4b4e-9e93-3df9e311e38b
│                    btrfs              edffef8f-c285-454d-9889-cb03930f2102  143,2G    69% /var/log
│                                                                                           /var/cache
│                                                                                           /home
│                                                                                           /
└─nvme0n1p3          crypto 1           8c9f6a9d-6937-46c2-99d7-704ca94368f5                
  └─luks-8c9f6a9d-6937-46c2-99d7-704ca94368f5
                     swap   1     swap  9135661b-2817-48f3-97a7-ed1e7ebbefb7                [SWAP]

EDIT: Forgot to mention, this is the situation after I already downgraded to the previous version, did you mean to run these commands after I upgraded to the newer version of GRUB?

/boot/efi/Efi/boot is the fallback bootpath. If I’ve got it it right , that is where the UEFI firmware will be looking if there is no other boot EFI entry in NVRAM or if the “main” entry fails to boot.

15 posts were split to a new topic: Remove unknown boot entries found after updating grub

What are you trying to accomplish with that?

Like others, I could not get r591 to work at all on my laptop, even after chroot from live and sudo grub-install --efi-directory=/boot/efi.

In the end I downgraded to r566 from chroot.

I’m thankful that, having seen grub in the upgrade list, I had the presence of mind to test on my (expendable) laptop rather than my main desktop system.

Grub r566 is going to the IgnorePkg corner indefinitely on all systems. [edit2: not a good idea long term, see long discussion below]

Next install will stick with systemd-boot. Some moderate amount of breakage is expected and tolerable when upgrading, yes. Casually bricking the system (unless you have a liveUSB on hand, and until you have the time to perform a chroot or several) does not qualify as moderate in my book. Twice in 6 months is way too much for something that critical.

I assume he, like me, is trying to get on with his life and not to have to worry about grub randomly breaking on update anymore.

Not really, unless you need something specific in the new versions. Nothing depends on the grub package. [edit: not a good idea long term, see below]

The update does not touch your /boot/efi directory, so GRUB cannot randomly break on update. It can break if you regenerate the config file with the wrong version of GRUB.

Consider, what happens when you update, for example, glibc, but you ignore an update to grub package? What would happen if you tried to install a new kernel?

You’re only making it more difficult for yourself down the line.

It did last time. That it can break after an update is the entire point of the previous, long discussion: Full transparency on the GRUB issue - Updated 2022-08-29

Personally, I upgraded as usual, did not run grub-install or -mkconfig, (I never did) and… brick. Fortunately on my disposable laptop, again. And another time in a VM.

Right. If glibc updates I expect it’ll break then. Fantastic.

Alternative to IgnorePkg:

Do not run grub-install or -mkconfig. Worked on both my laptop and my main machine.

Will hold until the next time they do something like r322, where you had to, which is why I had a hook set up to do so after every grub update.

Hopefully it will take longer for grub to break again in that special way than there is time between glibc upgrades.

I think you did, you just don’t know it. You probably have an automated hook set up to run grub-mkconfig as default on some earlier ISOs of EndeavourOS. That was a mistake on the part of EndeavourOS, not GRUB’s fault. This lesson was learnt last August.

No, it won’t break, it will be much worse than that. It will work fine. Then, one day, you will run grub-mkconfig and it may break, but if you run grub-install to fix it, that won’t work, because you’ll have a wrong version of glibc. That’s why partial updates are not supported on Arch.

2 Likes

I have now, (or had until 10 minutes ago :wink: ) because I set it up after r322. And I have the git logs to prove it.

And since then I read the output of the hooks carefully on each update; there was only one instance on that output, each time. If there had been a preexisting hook I’d have seen it run twice, each time.
And the systems I updated without running them were bricked until I did run the commands explicitly in chroot. If the problem was that they were run automatically, then running them again would not have fixed the bricked systems…

If you read the previous thread, you will see adding a hook mentioned as solution for future instances of the problem; there were other threads and posts to that effect on the boards at that time.

I would really like for you to be right, for grub updates to be safe by default. But I don’t see any way that can be the case here.

Again, you have three valid options with GRUB:

  1. Not update grub package. A bad idea in the long run.
  2. Don’t run grub-mkconfig and grub-install when you don’t need to.
  3. Run grub-mkconfig and grub-install every time the grub package updates, but be careful to do it correctly and manually (do not try to use hooks to automate stuff that shouldn’t be automated). If you mess up, you’ll have to chroot, so be ready for it, keep an USB on hand and do it only when you have 5 or so minutes to spare.

Or use systemd-boot. Of course, there is no guarantee that that won’t ever break.

Personally, I find GRUB to be very reliable.

I only got screwed once, and that was not GRUB’s fault, but because of the EndeavourOS hook. It took only 5 minutes to fix it.

Unfortunately, this was on my mum’s computer. I updated it when I visited her earlier that day, shut it down, and went home, and then later that night realised that her GRUB is borked. I was travelling next day to Norway, and I would be gone for two weeks, my mum unable to use her computer. So, I had to go back to her flat to fix it, and it was already 11 in the evening, and I had to catch the plane at 7 o’clock next morning. When I got back home, it was already very late, close to 2 I think, and I didn’t even begin to pack. But that’s just bad luck. :rofl:

On my vanilla Arch machine, I’ve never had any issues with GRUB.

1 Like

Which hook is that ? I don’t see anything with any screwing potential in the relevant package:

yay -Ql eos-hooks                                                                                                                                                                                      /etc/pacman.d/Σ 9:46 vhugot@Sekhmet
eos-hooks /usr/
eos-hooks /usr/bin/
eos-hooks /usr/bin/eos-hooks-runner
eos-hooks /usr/bin/eos-repo-before-arch-repos
eos-hooks /usr/share/
eos-hooks /usr/share/libalpm/
eos-hooks /usr/share/libalpm/hooks/
eos-hooks /usr/share/libalpm/hooks/eos-hooks.hook
eos-hooks /usr/share/libalpm/hooks/eos-lsb-release.hook
eos-hooks /usr/share/libalpm/hooks/eos-os-release.hook
eos-hooks /usr/share/libalpm/hooks/eos-repo-before-arch-repos.hook

I always reboot immediately after updating, no matter how trivial the update, precisely to avoid such nightmare scenarios :sweat_smile:

… not counting the hours it took you to go back to and from location, obviously. :frog:

Not update grub package. A bad idea in the long run.

The problem is that I’d love to upgrade to r591, but everything I’ve tried so far hasn’t worked, while r566 works flawlessly. I probably can refrain from running grub-install and end up with a working system for the time being, but the same logic applies, the first time I need to run it to fix some issue with grub-mkconfig will brick the system.

It seems like a bug to me on GRUB side, there’s too many reports from people to be a config problem on our end, even this thread on Arch Forum. I personally think that I’ll skip a couple revisions and will try to update GRUB in a couple of months to see if anything changed, but it looks more and more like a bug in r591. I’ll keep looking into it, but GRUB error messages are about as useful as an ashtray on a motorbike.

Look inside /etc/pacman.d/hooks and see if you got something in there.

It was a hook that ran grub-mkconfig every time a kernel was updated. I’ve deleted it since from all the computers I look after. It’s no longer part of the EndeavourOS install, but if you’re on an older install you might still have it. This was the hook that caused the legendary Grubbening of August 2022. :rofl:


EDIT: Found it!

That was the one I was referring to; I linked the wrong stuff previously, I wanted the original breakage thread for r322 in august 2022:

I think I finally understood something:

1° you’re talking about a hook that runs only mkconfig, while my own ran both grub-install and mkconfig.

2° The existence and role of that hook in the Great Grubbening of 22 was not mentioned as a cause of the problem in the thread, and so I developed a warped understanding of how grub works: I thought part of the grub package files passively played a role in the later phase of the boot, somehow:

So what was implied was “When you call grub-mkconfig, whether manually or because the hook we set up on kernel update does that for you, that updates only the userspace components.”
Pinging @dalto for sanity check.


One last thing I want to check: are we agreed that there is no reason to run grub-mkconfig on kernel updates, only on install/removal of entirely new kernels (eg zen or something like that)? (Or of course upon modification of /etc/default/grub, or after a grub-install)

If that is the case, since I’ve all the kernels I need (vanilla, lts, zen) already, and I’m happy with my grub config and unlikely to tinker with it anymore, now that I’ve removed my -install + -mkconfig hook, I can basically ignore updates to the grub package entirely (until I want to tinker again, of course).

Yes.

Well, you shouldn’t ignore updating the grub package, as in keeping it on your IgnorePkg list. Just update it normally.

That’s pretty much how I do it, I almost always forget to run grub-mkconfig and grub-install when the grub package is updated. I mostly worry about that when I change my config or install or remove a kernel.

The only downside is that you’re not getting any security updates or bug fixes to GRUB.