Kernel bug? Three times kernel panic during system update + Guide how to recover your system

Hi everyone,

it happened twice to me in the last month, both times breaking the system and requiring me to pacman --overwrite \* [...] from tty or live usb.

I noticed this is happening to many other people as well, like:

So I decided to post here a few thoughts and questions, and a short guide on what I did to fix my system.

When did this happen? Pacman/kernel bug?

I noticed that both of the times for me among the broken packages, there was util-linux-libs, probably because I was updating it while the system crashed. Or the system crashed because I was upgrading it?
For sure breaking it is the cause of not being able to start the graphic system, or not being able to boot entirely. But the system should be able to updates the kernel without committing suicide.

Does anybody know what happened and if this is a known bug in pacman or in the kernel? Should I fear updating the next time an update for some core package comes up?

EDIT: I just tried another update, containing two core packages: libassuan and pcre2. Again kernel panic during the install. Had to go through the same process again.

Am I just very very unlucky and 3 crashes exactly while updating core packages?
[I must say that recently I am also getting some random kernel panics on shutdown, so there might be something wrong with my machine, or hardware combo with latest kernel versions.]

At last, here is all the info I collected and put together from arch wiki, forum and other random posts online, mainly for me to remember, but I suppose for others as well, as this is happening to many users.

[Guide] How to fix your system when broken libraries prevent you from booting

My case was: kernel panic during update. Reboot. Getting kernel panic during boot.

Access your system from live USB

  1. Take your USB with arch/EndeavourOS and boot into the live system
  2. Find your root partition with lsblk. If you have a separate /efi partition, it is important to mount it as well, if you are rebuilding initramfs, which is usually happening after system update.
  3. Mount them (replace nvme0n1pX with your partition names)
> sudo mount /dev/nvme0n1p2 /mnt
> sudo mount /dev/nvme0n1p1 /mnt/boot
  1. Now you can chroot into your system with sudo arch-chroot /mnt. This will also make /proc accessible and is generally better than chrooting by hand.
  2. Now you should be able to use pacman as you were on you system.

NB: You may need to sudo rm /mnt/var/lib/pacman/db.lck if the crash happened during an update.

If pacman is broken

Sometimes you screw so bad that pacman in chroot-ed system is also broken. You may need the to run it directly from the live USB. In this case follow the guide below using pacman directly from the live-usb terminal, like this sudo pacman --root /mnt [...]. This will allow to work on your broken system, while using binaries and libraries from the working live system.
(paccheck also has the same --root flag available).

NB: Remember, after fixing the system in this way, to also run

> sudo arch-chroot /mnt
> sudo reinstall-kernels

as post-install scripts/hooks don’t work without being in arch-chroot.

Fixing the system

  1. First find packages that are broken, by checking the checksums of their files
> sudo paccheck --md5 --list-broken			
eos-plasma-sddm-config			#	This is my output on a working system,	
filesystem						#	these are ok. They're edited 
lsb-release						#	on purpose (by eos scripts or by me, I think)

PS: using the --sha256 sum instead, gives few extra broken packages on my working system (texlive-basic, vlc). In my case fixing the ones with broken md5sum was enough.

  1. Force reinstall broken packages using sudo pacman --overwrite \* -S [PACKAGES].
  2. Reboot into your system.

Are you using nvidia drivers? There is a issue with the latest drivers causing kernel memory corruption.


updating a few core packages led to the same situation for me today. my forum browsing (and entire day basically) were rudely interrupted a kernel panic. if only i had seen this topic earlier…

spent time on the live iso browsing countless forums and threads trying tens of solutions. in the end something similar to what you describe worked except i re-installed all packages

Sounds like a true mayhem :wink:

1 Like

oh it was. its the holy month of Ramadan so i couldn’t even curse while i pulled my hairs trying to get pacman to work :zipper_mouth_face:

1 Like