Can't boot after an update

[Budoka]

I have been running Endeavour on my box without issue for a couple of years now but have run into a rather serious issue I hope you can help with.

After running pacman -Syu I am not able to reboot into EOS. I am not sure what came down but it seems unlikely to me that it is a coincidence that it occurred immediately after the update.

This is what happens when I rebooted

[ 0.093628] call_irq_handler: 0.188 No irq handler for vector

:: running early hook [udev]

Starting systemd-udevd version 257.2-2-arch

:: running hook [udev]

:: Triggering uevents…

:: running hook [keymap]

:: Loading keymap…done.

:: performing fsck on ‘UUID=dаба648e-ef32-479a-8991-c9bd01225c1d’ /dev/nvmeln1p7: clean, 337986/1966080 files, 3661046/7864320 blocks

:: mounting "UUID=da6a648e-ef32-479a-8991-c9bd01225c1d’ on real root

mount: /new_root: fsconfig system call failed: No such file or directory. dmesg (1) may have more information after failed mount system call.

ERROR: Failed to mount ‘UUID=dаба648e-ef32-479a-8991-c9bd01225c1d’ on real root

You are now being dropped into an emergency shell.

sh: can’t access tty; job control turned off

[rootfs #

This is a multi-boot laptop. I can boot into the Windows partitions no issue. Also when I boot from the live iso I can see all of my EOS partitions are intact and the data is still there.

There are a couple of quirky things that I noticed that may or may not be related. During the Pacman update I was logged out of XFCE. I logged back in no issue.

Also, I initially launched the live iso from USB and it was an older version, Neo. There was no issue and I was able to use laptop no issue and check the installed partitions. But I thought I should run the newer version so added that iso to the USB and it dumped me into the same error as above. So I put NEO back on and now when I run that it dumps me into the same emergency shell.

I can still access everything with Kali, Gparted, Partition Magic etc.

[Budoka]

Another data point. I checked that the UUID is correct in fstab. Not sure why it would have changed but confirmed anyway.

[ManInTheMaze]

Did you try running mkinitcpio -P after chrooting into your live usb to regenerate initramfs?

[Budoka]

I executed mkinitcpio -P after chrooting and successfully regenerated the initramfs. It didn’t resolve the issue. Fallback initramfs is the same. What else could be going on? It is clear to me it was the update that borked it. Not sure what or how though. I rather not reinstall EOS because although small there is the risk of losing the data on home. Yes it is backed up but still not ideal. Any ideas as far as next steps?

[anon93652015]

Assuming that you are using the current kernel, try installing the LTS kernel.

Assuming you are using GRUB and mkinitcpio, then redo what you did earlier.

If you are using Dracut and Systemd-Boot, then follow the relevant instructions here:
https://discovery.endeavouros.com/?s=boot

[Budoka]

Thanks for the assist. Can you clarify for me though? You are saying install LTS kernel and then rerun mkinitcpio? I am a little confused by what you mean redo what I did earlier.

[anon93652015]

Yes. You interpreted what I said correctly.

[Budoka]

Thanks for the assist. Installing the LTS kernel resolved the issue. I checked the Pacman log and I see that the initramfs was regenerated in the update that borked the system. I am guessing that is what did it.

Pacman Log

I will pay closer attention to what Pacman -Syu from now on but is there a way to restrict the kernel updates/changes to only LTS versions?

[anon93652015]

Once you have both kernels installed and one works, just use the one that works.
You don’t have to worry about the other one updating since you won’t be using it anyway.

And yes, there is a way to restrict any package from updating by adding the package’s name to the ignore section of your pacman.conf.

Like this:

sudo nano /etc/pacman.conf

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg   =  linux linux-headers
IgnoreGroup =

THIS SHOULD ONLY BE TEMPORARY OR YOUR SYSTEM WILL EVENTUALLY BREAK

But again, for your case in particular, no need for that, nor is it recommended. Just use the kernel that works on your system.

That said: Don’t uninstall the other one because having two kernels may use up time (like an extra minute) when updating, but may save time in the event that the one you normally use has issues booting. If this happens, you can just boot into the other.

---

Lastly, until very recently, the LTS kernel was 6.6.xx and it is now 6.12.xx, which is only one “major” version behind the mainline kernel, 6.13.xx. The Arch developers did a massive jump for some reason I haven’t researched as yet. It is highly recommended that you use the LTS to avoid zero-day kernel update issues now.

EDIT: Never mind. I found out why now. https://www.kernel.org/category/releases.html
Still kinda weird that it’s been available since November and just now got adopted by Arch.

[Budoka]

Thanks. Good info and appreciated. The only thing is that I can’t boot into the other kernel which is what kicked off this entire issue. I think what I’ll do is add a previous stable kernel as well so that I have another boot option in the event something happens with the current LTS as you indicated.

[anon93652015]

Not sure I follow. Let’s check what you have. Please show the outputs of the following:

inxi -Fxxc0z

And:

pacman -Q | grep -A 0 -E "linux"

[Budoka]

Sorry for the confusion. What I mean is that I was not able to boot into the kernel that Pacman updated. That was my original post. I was just commenting that your suggestion to have two kernels is a good one but in my case, the second kernel, non-LTS, didn’t work so there wasn’t no point keeping it. I already removed it before seeing your message. LOL

[dbarronoss]

It’s likely that in a release or two, the ‘current’ kernel will be fixed for your case. Don’t write it off yet.

[Budoka]

It fixed it. Whatever was going on.

So now running the most recent kernel and have LTS as backup. Thanks everyone.

[dbarronoss]

Yeah, unfortunately it’d take about a full time job to be totally aware of what all happens in the kernel (hey that’s what we have Linus for?) However, this is very typical (and I’ve been there myself).