Full transparency on the GRUB issue - Updated 2022-08-29

@jonathon, I agree that blaming the Arch team for the issue is completely unfair. It took a lot of things to happen for this issue to present itself. However, claiming the Arch team did everything right and has no culpability at all seems equally unreasonably.

The hook was a contributing factor but the issue wasn’t caused solely by those hooks. If those hooks weren’t the issue would still occur whenever you ran grub-mkconfig in the future. Also, there was no documentation that states that grub-install needs to be run when grub is updated.

As stated above it took a lot of things to happen for this to occur, it wasn’t only, “Because of the hook”

Three things here:

  1. Having to create an ISO, boot off said ISO, build a chroot, run grub-install and then deal with the fallout of that, isn’t what I would call “trivial”.
  2. Certainly anyone using Arch should expect that sometimes applications won’t work exactly as intended. However, expecting that your system will be bootable after a normal update does seem like a reasonable expectation. Blaming the end user because they expect their system to boot seems over the top to me. To be 100% clear, I am also not blaming the Arch devs here.
  3. That grub release had issues with beyond this one which were reported. It seems like rolling back that package temporarily until those issues were better understood, may have been prudent in this case.

This one is just not the case. There are lots of grey areas here which is why generally I think trying to “find someone to blame” is both hard to do and a useless thing to do in the first place. The one area where I feel the Arch team clearly could have responded differently is in the announcement. They acknowledged and an announcement was needed days before making the announcement.

5 Likes

would not call installing kernel or adding boot parameters is totally an edge-case but i understand what you want to say indeed :dizzy:

1 Like

Wait, wouldn’t a lot of edge cases add up to not being an edge case anymore? I mean at some point the “edge” becomes a mile wide doesn’t it? Or a kilometer anyway :grinning:

They did what was needed for Arch. Anyone running Arch is more than capable of using a chroot - because they have to do that to install.

Users of downstream derivatives blaming the Arch devs for a failure in their derivative is unfair.

I didn’t say that.

But the fix was already known - run grub-install.

This I very much agree with, and was the crux of my posts.

1 Like

I agree with @dalto
If there is fire, you put it off first.
Second, be sure it will not happen again or if it happens it would be limited not catastrophic.
Third, if home and furniture gone due to fire, find yourself another place to sleep (not necessarily changing distro), maybe rebuilding same home same design and watching out (grub install), or change where the kitchen that caused the fire, make it electric not gas! (other boot loader?!)
Fourth, if you found the one “to blame”, better be sure he will not burn your new house.
Fifth, I do not know of a way to let him pay if you could ever get him to pay! As this is open source I can say he was trying to help by lighting fire in your back yard so you can grill something delicious, but sh?t happened unintentionally.

A few things:

  • This is objectively untrue. Arch ships with an installer so you no longer need to use a manual chroot to install Arch.
  • Even if it didn’t include an installer, following the Arch install instructions at some time in the past doesn’t imply you know how to build a rescue chroot.
  • It is irrelevant. Knowing how to chroot has nothing to do with anything. It doesn’t mean you have what you need on hand and it doesn’t stop your system from breaking in the first place.

We need to stop finger pointing and creating conflict here where there doesn’t need to be any conflict in the first place.

Blaming Arch is unfair whether they are downstream users or not. I stated that clearly in my post. You seem to have excluded that part in your quote.

We clearly agree that nobody should be blaming Arch here.

However, that doesn’t change the fact that their response to the situation could have been better. Hopefully all parties involved are considering ways that things could be improved in the future in the spirit of continuous improvement. That goes for the Grub team, the Arch team and all the teams of any impacted downstream distros.

That only fixed one of the problems with that version of the package. There were others, and they were known at the time.

Also, it was also unclear at the time if this was a temporary problem that grub was planning on reversing or if this was a permanent change.

Again, I am not trying to place any blame, just trying to be realistic about this situation.

6 Likes

No one should blame anyone in such a case anyway.
We all should stop doing this.

It is not fair in all cases because it was not done with the intent to produce an issue, it happen by accident.

What I really don’t like is that there is still no friendly cohesion between the Arch-based distributions and Archlinux itself.
In the last few years, a lot has developed positively and this one problem shouldn’t lead to a deterioration in the relationship.

We should find ways to work better together.
With respect and fun.

:two_hearts: :peace_symbol: :partying_face:

9 Likes

This is the spirit and power of open source as I knew it. It is a win win relation, not a zero sum game! (Unless I come from another world or I am still living in the past)

1 Like

I agree it’s better to work cohesively. It’s all an Endeavour! :rofl:

3 Likes

Yes, we’re all astroauts. On some kind of star trek. :rofl:

The summary of all these thoughts & reasonings for me boils down to this:

  1. Every Arch-Distro user should really try to be more self sufficient. This can arise by installing Arch Linux in a VM quite easily. The gain of knowledge this way is just tremendous. It cannot be gained by learning to point-and-click the right way in calamres or other gui-installers, or install-scripts. So just try doing it the Arch-Wiki way should be considered a must, when wanting to learn all this. (I know, I repeat myself here.)

  2. Blame is bloat! Always when pointing fingers in one direction, three fingers are pointing back at the one who poses blame away from himself. It is another thing to learn.

  3. Without blame, there can be true cooperation and friendship across borders visible, and unvisible.

  4. Should I go on? - Spare me…

:see_no_evil: :hear_no_evil: :speak_no_evil:

2 Likes

I can’t easily quote across several messages.

The hook in EOS and other distros: The need to call grub-mkconfig after kernel updates is called for in the Arch Grub wiki. Ref in green Tip box.

Just above this section is the warning about updating both sides of Grub if the syntax of the config file changes. If we only knew such a change was within build #1.

Arch users posting in Reddit and the Arch bbs forum have been hitting the issue - I’m assuming they are DIY’ers, which may not be valid. This bbs user recently hit it when updating a theme, so his config likely doesn’t match the bootloader.

We’ll see these recent issues until probably the next grub version (and for some time beyond) makes the ISOs and Arch Core (and people successfully upgrade each side). Within the last day I was helping a user who had done the chroot procedure two weeks ago, did another -Syu and likely got build 4 and was stuck with the 452 pointer error message - he missed the terminal messages and did not grub-install / grub-mkconfig, but might have an implicit grub-mkconfig if he got a new kernel and the famous hook.

As Joe said, more cohesion, work together. +100000000000 Arch! And think of all the users.

6 Likes

Not to say this isn’t a good practice, Arch didn’t say anything about the issue until like a week or more later, I’m not sure how this would have helped anyone.

I hope this is true. The devs clearly use grub and my guess is they will be exhausted installing grub on every update they will get rid of this quickly hahahaha

2 Likes

Maybe they don’t run Arch, but run something static like Debian. One never knows, the whole world (strangely) isn’t a rolling release.
Also it was stated earlier in the thread (which is humongous now), that releases FROM grub are quite rare. In fact it has been more than a year since the last release (vs pulling from git) I believe it was said.

Arch Grub wiki calls for both sides of grub to be updated when the syntax of the config file changes. When the fwsetup change was being done, that should have been considered and ideally a notification in advance made. Maybe ASCII art in the -Syu output too. That’d be cool.

Something akin to the second pipewire/wireplumber attempt with it’s advanced notification, although that team felt pain months before.

1 Like

The grub manual by the grub team doesn’t even mention updates and updating grub. I perceive they think of grub as, installed once at system install time, and never touched again. It seems that other distros use that model per my other post about using distrowatch (haha) data and some anecdotal evidence.

Grub releases can be seen on their site: https://ftp.gnu.org/gnu/grub/

5 releases since May 2011. Estimating the next group release will become between April and July 2023 assuming they are still reliable.

1 Like

Where is the grub wiki?

1 Like

Good question!

Several sites are available. The first go-to for EnOS-users should, as always be the arch Wiki:
https://wiki.archlinux.org/title/GRUB#UEFI

Then, there is OSDev:
https://wiki.osdev.org/GRUB

And the GNU Grub Manual:

For future issues, one might bookmark these above links.

:wink:

1 Like

I am familiar with all of those but I don’t see where the osdev wiki or the grub documentation mention what @andrewb was describing:

That is why I asked for the location of the wiki he was referencing.

1 Like

During an update that installs a new kernel, grub-mkconfig is executed also on pure Arch, isn’t it? That will put into troubles Arch users as well, if I understand correctly.

Running grub-install has a few drawbacks, at least in my workflow (where I don’t want the grub from Arch to be the main one on my computer).

I prefer to simply get rid of the culprit grub entry (since I never used the UEFI entry from grub: if I want to enter UEFI I’m using a key combination at boot):

sudo rm /etc/grub.d/30_uefi-firmware
sudo grub-mkconfig -o /boot/grub/grub.cfg

Of course, I have to do that if they keep on releasing upgrades of that version of grub.

Just in case, here are my other notes on the subject: https://www.lorenzobettini.it/2022/09/my-notes-on-the-grub-incident-in-linux-arch/