Full transparency on the GRUB issue - Updated 2022-08-29

[dalto]

Since the recent grub issue has impacted a lot of people, we wanted to provide full transparency based on the information we have so far. The situation with this package is still evolving and we will update this post with more information as it becomes available.

The issue

After updating to grub 2.06.r322 many users reported that their machines could fail to boot or booted directly into the BIOS or another OS.

What caused the issue

Starting with this commit, grub introduced a call to fwsetup --is-supported in /etc/grub.d/30_uefi-firmware. If the version of grub you have installed via the grub-install command didn’t support that command, it caused grub to fail.

How come not everyone was impacted

Prior to the most recent version, grub only registered the fwsetup if detected support. If your machine detected support, you would have had the fwsetup command registered and the failure wouldn’t occur.

I have already updated and my machine is broken, what should I do?

Follow the instructions in this post to chroot into your system and run grub-install to install the latest version.

I haven’t updated yet, is there anything I should do

Follow the instructions in this post that relate to that scenario. Basically, run grub-install after upgrading but before rebooting.

What happens next with the grub package

According to the bug report, Arch will produce a version of the package without that commit while working with grub upstream to determine next steps

Arch decided to release a new version which was functionally the same but which includes a post install message to run grub-install and grub-mkconfig

What about new installs?

For online installs, this issue won’t impact new online installs since grub-install gets run as part of the bootloader getting installed and online installs always have the newest available packages.

For offline installs, a rebuilt Neo ISO is being released which will have the updated grub package.

Why wasn’t this caught in testing

We can’t answer this question absolutely but there are at least two factors to consider:

• Not all grub users were impacted by this issue
• Many Arch users don’t run grub
• Only UEFI installs were impacted

What should we do differently in the future to avoid this type of problem with grub

We are exploring all options here but the reality is that this has never happened before. Blindly running grub-install every time would be knee-jerk reaction and probably create more problems than it would solve.

We were already considering moving away from grub by default and that may happen at some point in the future.

We are also investigating the possibility of an overlay repo which would give us more control over packages coming from Arch. Currently, we have no ability to influence packages which come from Arch.

First we will wait to see what Arch decides to do moving forward and then we will make a long-term decision.

[keybreak]

@dalto

We were already considering moving away from grub by default and that may happen at some point in the future.

rEFInd?

[yourearandom]

Many Arch users don’t run grub

I’ve mentioned this on another thread, regardless of the default, why not let users select a bootloader during the Endeavour install?

[keybreak]

Because it’s technical nightmare to create iso that way…The only real way it could work is release separate iso for different bootloader, which seems highly unnecessary

[yourearandom]

Well maybe we should have a poll.

I would vote systemd-boot since it is always going to be there.

[dalto]

@keybreak

rEFInd?

It is certainly one of the things we have discussed.

@yourearandom

I’ve mentioned this on another thread, regardless of the default, why not let users select a bootloader during the Endeavour install?

This has also been discussed. However, it is a lot more work both from an implementation and a support perspective.

@yourearandom

I would vote systemd-boot since it is always going to be there.

There are some challenges with systemd-boot if used as the default.

• It doesn’t support an encrypted initramfs
• It breaks support for booting from a snapshot.

That being said, the discussions we have had about switching are only discussions at this point. Nothing has been decided. We may stay with grub or we may switch.

[dalto]

Please don’t use this topic as a place to discuss the various merits of different bootloaders. Let’s try to stay focused on the issue at hand.

[ringo]

if its goes very blend on grub version…eos-grub could be an uption to freeze it a bit by repackaging it…until some move, but at the end ,upstream also other distro’s goes to sysd-boot… it pretty hard time for each unique situations…

[dalto]

Even if we repackaged grub, we would need an overlay repo to make a difference on existing installs. This is also something that is being discussed.

[ringo]

on long end is no option if nothing changes in grub offcourse …

[Gamall]

I had this happen on my Latitude 9520. Thankfully I wanted to reinstall anyway.

Why is this not in the https://archlinux.org/ news page ? This is a serious breakage, and not at all specific to EOS.

[dalto]

@Gamall

Why is this not in the https://archlinux.org/ news page ? This is a serious breakage, and not at all specific to EOS.

Sadly, Arch was only notified about the issue earlier today. It is possible there will be an announcement at some point.

I am also not sure how many Arch users are actually using grub.

[petsam]

Really strange issue, so have a couple more things for discussion.

• IMHO grub should remain the main bootloader, unless it may become possible to choose alternative at initial installation, like grub/syslinux for BIOS setup and refind for UEFI, or even manually by the user.
• Grub installation installs a small grub image on EFI. This remains unupdated with new grub/osprober updates, which is no problem, until it is (something critical is changed). Something similar to kernel/systemd/gcc etc. We can never know when it needs reinstallation, unless someone points out, either an upstream dev, or a discovered bug report. I should add a possibly unrelated notice. The latest grub commits modified some pgp keys. I am not sure it ha nothing to do with this issue, as firmware/software use secure communication when talking. Shouldn’t those pgp keys be identical between latest installed grub image at $ESP and system grub package, or compatible with system’s/kernel’s? I have no idea…
• Mounting $ESP at /boot/efi requires frequent grub-update. Using /boot, or /efi, eliminates this need. Yes, Calamares… If it is universal tool, they should provide such option for such an important system configuration, especially when Arch default is as I just described.
• A disaster is never 100% preventable. We have to be prepared and have a plan.
• A grub entry that would boot to bash, or to timeshift restore command, where it applies.
• A grub addon, or custom menu like supergrub2, which I had started working on, some time ago, but left it because of other priorities.

[fbodymechanic]

Grub has been pretty rock solid over the years, I’m not sure it’s necessary to abandon ship over one thing that really just required a reinstall.

We’re a rolling release with a target audience of folks who should be vaguely familiar with the process, if not at the very least everyone should know how to arch-chroot into their system- It’s a good learning opportunity for folks as well.

I think striving for a perfectly stable experience is great, but to have an expectation nothing will break ever, while also being brand new all the time. . . It’s just not realistic for users or developers.

There was a major problem, and in extremely short order we not only figured out a work around, made it well known in the forum and groups, and got people up and going. Well done.

“It does not matter how many times you get knocked down - only how many times you get up.”
-Vince Lombardi

[ringo]

seems somemove ?https://github.com/archlinux/svntogit-packages/commit/1cd6ce212cb456340beba6093b89e2bd196f908b

[dalto]

@ringo

seems somemove ?

They initially made that change and pushed it to testing but it was quickly pulled back.

@fbodymechanic

Grub has been pretty rock solid over the years, I’m not sure it’s necessary to abandon ship over one thing that really just required a reinstall.

To be clear, the discussions of replacing grub started long before this issue came about.

Also, grub has never been even vaguely “rock solid”. Especially when combined with os-prober which it almost always is.

[fbodymechanic]

@dalto

Also, grub has never been even vaguely “rock solid”. Especially when combined with os-prober which it almost always is.

Interesting. It’s pretty much all I’ve used for the last decade and I don’t really recall having more than a couple issues, but anyways. It’s been pretty rock solid “for me”.

I can’t comment much on os-prober as I’ve almost never used that and even my dual boots don’t use it.

[joekamprad]

not to put fire in that but i think if you are using Linux for a long time over the past decades like me too… we are simply used to using grub, so we know how to handle it tweak it and we know what to do if it does not work … and indeed we know about arch-chroot and how it will help us to fix such issues…

Grub’s killing feature is that it can do them all Bios/legacy Uefi any filesystem encryption…

And this is the main reason why it is easy to implement into installer frameworks…
But today there are optionals to grub that can handle the bootup more easily, more robust, and in a more minimal way.

But the current issue with Grub shows that it is complex to troubleshoot and the bootloader is a vital part of a running Operating System for what we have to make a wise decision.