Hi everyone,
it happened twice to me in the last month, both times breaking the system and requiring me to pacman --overwrite \* [...] from tty or live usb.
I noticed this is happening to many other people as well, like:
- Kernel panic during update broke my system. Thanks God and Linux Community for the arch-chroot
- Kernel panic after broken update, pacman not working inside chroot
- Kernel panic during update / issues with pacman sysroot
- [SOLVED] Kernel panic after update crash
So I decided to post here a few thoughts and questions, and a short guide on what I did to fix my system.
When did this happen? Pacman/kernel bug?
I noticed that both of the times for me among the broken packages, there was util-linux-libs, probably because I was updating it while the system crashed. Or the system crashed because I was upgrading it?
For sure breaking it is the cause of not being able to start the graphic system, or not being able to boot entirely. But the system should be able to updates the kernel without committing suicide.
Does anybody know what happened and if this is a known bug in pacman or in the kernel? Should I fear updating the next time an update for some core package comes up?
EDIT: I just tried another update, containing two core packages: libassuan and pcre2. Again kernel panic during the install. Had to go through the same process again.
Am I just very very unlucky and 3 crashes exactly while updating core packages?
[I must say that recently I am also getting some random kernel panics on shutdown, so there might be something wrong with my machine, or hardware combo with latest kernel versions.]
At last, here is all the info I collected and put together from arch wiki, forum and other random posts online, mainly for me to remember, but I suppose for others as well, as this is happening to many users.
[Guide] How to fix your system when broken libraries prevent you from booting
My case was: kernel panic during update. Reboot. Getting kernel panic during boot.
Access your system from live USB
- Take your USB with arch/EndeavourOS and boot into the live system
- Find your root partition with
lsblk. If you have a separate/efipartition, it is important to mount it as well, if you are rebuilding initramfs, which is usually happening after system update. - Mount them (replace
nvme0n1pXwith your partition names)
> sudo mount /dev/nvme0n1p2 /mnt
> sudo mount /dev/nvme0n1p1 /mnt/boot
- Now you can chroot into your system with
sudo arch-chroot /mnt. This will also make/procaccessible and is generally better than chrooting by hand. - Now you should be able to use
pacmanas you were on you system.
NB: You may need to sudo rm /mnt/var/lib/pacman/db.lck if the crash happened during an update.
If pacman is broken
Sometimes you screw so bad that pacman in chroot-ed system is also broken. You may need the to run it directly from the live USB. In this case follow the guide below using pacman directly from the live-usb terminal, like this sudo pacman --root /mnt [...]. This will allow to work on your broken system, while using binaries and libraries from the working live system.
(paccheck also has the same --root flag available).
NB: Remember, after fixing the system in this way, to also run
> sudo arch-chroot /mnt
> sudo reinstall-kernels
as post-install scripts/hooks don’t work without being in arch-chroot.
Fixing the system
- First find packages that are broken, by checking the checksums of their files
> sudo paccheck --md5 --list-broken
eos-plasma-sddm-config # This is my output on a working system,
filesystem # these are ok. They're edited
lsb-release # on purpose (by eos scripts or by me, I think)
PS: using the --sha256 sum instead, gives few extra broken packages on my working system (texlive-basic, vlc). In my case fixing the ones with broken md5sum was enough.
- Force reinstall broken packages using
sudo pacman --overwrite \* -S [PACKAGES]. - Reboot into your system.
