My Grub breaks after the last update of my system

[ricklinux]

@pebcak

A bit frustrating since the reason(s) why this happened never came into light

Frustrating at the time yes … but I’m over it now.

[anonymix]

Sure, will try next time.

[petsam]

@anonymix

I’m already running grub-install in arch-chroot and it appears to fix grub until the next update. I had to fix grub boot loader about 7 times already this month.

A system update, especially without a grub upgrade, would not change EFI vars and the default boot entry. Check your hooks for a grub-install command.

grep grub /usr/share/libalpm/*/*
grep grub /etc/pacman.d/hooks/*

@anonymix

I’ve tried running it immediately after pacman -Suy and just before the reboot and it did nothing. Worked in arch-chroot in a live USB though.

This should not normally happen, so this is the real problem, after having more info about your system.

@anonymix

I had to install custom kernel recently. Basically 6.9 with a few ath12k patches.

I have to assume that this is not an archlinux official kernel. When you succesfully fix the problem, you are running an official archlinux kernel (from the ISO). This is a potential reason for the problem, becuase of the grub limitations, as discussed. I also have to assume that you are booting and using your custom kernel, and run grub-install under that.
If you have another kernel (official, preferably not linux-lts, to exclude an older kernel), use that one when installing grub. Next time, or now, if you want to be preempting.
For the same reason (prevent), you should delete all Linux EFI entries and recreate a uniqe one (with grub-install as proposed). The way UEFI firmware selects what to boot and when is not standard, as manufacturers use different algorithms for their own reasons. The issue came up because you think you boot your new UEFI entry, while you are booting an old/different one, which cannot read/discover f2fs partitions.

At least, that’s all my Magic Crystal Ball has told me in confidense .

[ricklinux]

I think @petsam is correct.

https://wiki.archlinux.org/title/GRUB#Unsupported_file_systems

[anonymix]

$ sudo grep grub /usr/share/libalpm/*/*
$ sudo grep grub /etc/pacman.d/hooks/*
grep: /etc/pacman.d/hooks/*: No such file or directory

@petsam

I have to assume that this is not an archlinux official kernel.

I built it with official PKGBUILD, just a few ath12k patches on top (for hibernation to work). These patches should be in 6.10, but before that I have no choice other than to build the kernel. Originally I was on 6.8 where my wifi card simply didn’t work at all.

@petsam

For the same reason (prevent), you should delete all Linux EFI entries and recreate a uniqe one (with grub-install as proposed).

Sure, I removed all of them and reinstalled with grub-install. Hopefully I’ll never see this issue again, but who knows.

[petsam]

@anonymix

I removed all of them and reinstalled with grub-install. Hopefully I’ll never see this issue again,

If you did as suggested , it should not happen again, or the issue should be different.

@petsam

If you have another kernel (official, preferably not linux-lts, to exclude an older kernel), use that one when installing grub.

[anonymix]

Well, it happened again.

[anonymix]

I can’t do anything in grub rescue, because it says unknown filesystem for everything except (hd0,gpt3) which is my home partition.

[pebcak]

Maybe you could create a “Grub compatible” boot partition with a supported filesystem as implied by ArchWiki: https://wiki.archlinux.org/title/GRUB#Unsupported_file_systems

[anonymix]

Will it be enough to make a separate /boot partition (not just /boot/efi like it currently is) and format is as fat32?

[pebcak]

In the continuation, if you get any error message at any stage, stop and post the whole output here on the forum.

Also, if anything is unclear, stop and ask about it here.

This is how I would do it from a live session.

Note that the following is only an outline of things to do. The command lines need to be adapted to the actual setup, like partitions, UUIDs etc

1. carve out something like 1024 -2048 MB to be used as boot partition. Format it as EXT4.

2. mount the system partition at /mnt.
   Then rename the current boot directory (sudo mv /mnt/boot /mnt/boot.old)

3. create a new boot and an efi directory inside it: sudo mkdir -p /mnt/boot/efi

4. mount the newly created boot partition to the newly created boot directory: /mnt/boot

5. mount your existing ESP to /mnt/boot/efi

6. chroot into your system: arch-chroot /mnt

In chroot:

1. reinstall your kernels plus the headers, for example:

   pacman -S linux linux-headers

This should trigger automatically the generation of your initramfs. Pay attention to the output in the terminal and make sure that this is the case.

2. backup your/etc/default/grub

   cp /etc/default/grub /etc/default/grub.bckp

3. reinstall the package Grub:

   pacman -S grub

4. reinstall the grub’s bootloader

   grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=EnOS

5. check if you need to add some boot option to your /etc/default/grub. You can have a look at the backup file you made before. Regenerate grub.cfg

   grub-mkconfig -o /boot/grub/grub.cfg

6. find out the UUID of your new /boot partition. Edit your /etc/fstab and add a new entry for it. For example: UUID=xxxx-yyyy....blablabla /boot ext4 defaults 0 1

Also check that the UUID for other partitions are correct.

If everything is fine so far, type exit followed by enter to quit chroot.

Now cross your fingers and reboot

[petsam]

@anonymix

Well, it happened again.

@petsam

If you did as suggested , it should not happen again, or the issue should be different.

If you have another kernel (official, preferably not linux-lts, to exclude an older kernel), use that one when installing grub.

Have you installed a normal kernel? You may only need it for installing the grub bootloader (with grub-install command).

Doing the same thing and expecting a different result is a worldwide known phenomenon.

[anonymix]

“Normal” kernel doesn’t work with my WiFi card, so I can’t use it. In fact, I have to use USB tethering to actually use installer iso.

Obviously, I can’t reboot into a different kernel before updating the grub because it may die after reboot.

I guess I’ll try moving /boot onto a separate partition, the only complicated part is to make enough space for it.

[petsam]

@anonymix

“Normal” kernel doesn’t work with my WiFi card, so I can’t use it.

You may have misunderstood.
Installing the bootloader (whenever this is required) can be done off-line (no internet required).

@petsam

You may only need it for installing the grub bootloader (with grub-install command).

As for

@anonymix

I can’t reboot into a different kernel before updating the grub because it may die after reboot.

the statistics for having a broken system after a reboot, are less if you install an archlinux kernel (the default for Arch).

But, what the… ?
It’s your system

[anonymix]

@petsam

the statistics for having a broken system after a reboot, are less if you install an archlinux kernel

Yes, because one can’t update offline. Jokes aside, it’s a bit better with 6.9 though as now I only need custom firmware to have WiFi, but hibernation only works in 6.10+

@petsam

You may have misunderstood.
Installing the bootloader (whenever this is required) can be done off-line (no internet required).

So what are you suggesting me to do? Upgrading arch is already painful enough because of having to reboot. Making it twice as painful (reboot into arch kernel, update, reboot back) isn’t the solution. And I’m 90% sure it won’t help because why it would? I don’t see any technical reason for that.

What I really need is to run whatever kernel I have to and never deal with this insane grub error.

[anonymix]

@pebcak

• carve out something like 1024 -2048 MB to be used as boot partition. Format it as EXT4.

This is the hardest part, resizing filesystems just isn’t working well. Can I reuse ESP partition for this? It’s 1000 MiB right now and /boot is about 360 MiB currently.

[pebcak]

@anonymix

@pebcak

• carve out something like 1024 -2048 MB to be used as boot partition. Format it as EXT4.

This is the hardest part, resizing filesystems just isn’t working well.

You should be able to use Gparted on the live session to resize the relevant partition. Note that the partition to be resized should be unmounted.

@anonymix

Can I reuse ESP partition for this? It’s 1000 MiB right now and /boot is about 360 MiB currently.

You could. However you would need to skip/modify some of the commands/instructions posted above.

Disclaimer:
Given the discussion above about the custom kernel in use and it being possible culprit in causing the boot issue, I cannot guarantee a successful resolution to this issue.

[joekamprad]

changed the title… we have still a grub breaks trauma
And it was reading like a general grub issue breaking boot.

And welcome back after a year without issues

[anonymix]

@pebcak

Gparted on the live session

If only it was there.

[pebcak]

You might be able to install it?

Also, perhaps KDE partition manager is already on the live iso?