Another Kernel panic during update / pacman --sysroot'ing this time

General system Newbie
1

Disclaimer: This is mainly for documentation purposes for now, not sure yet I’ll need any help, although, there are a few questions I had along the way. Feel free to comment.

I had another kernel panic (AGAIN) during a system update, the interesting bits are:

core/linux                       6.8.7.arch1-2  6.8.8.arch1-1   -0.26 MiB
core/linux-headers               6.8.7.arch1-2  6.8.8.arch1-1    0.08 MiB
core/openssh                     9.7p1-1        9.7p1-2          0.00 MiB
core/openssl                     3.3.0-1        3.3.0-1         10.92 MiB
core/systemd-resolvconf          255.5-2        255.5-3          0.00 MiB
core/systemd-sysvcompat          255.5-2        255.5-3          0.00 MiB

I did another update just a day or two ago, so not much to update, except, a new Kernel version.
Kernel panic happened after (3/11) upgrading systemd.

Having read just yesterday that there’s a brand new EOS iso, I downloaded it with confidence and some hope that I could just pacman --sysroot this time:

  1. cryptsetup and mounted /, /home and /mnt/efi.
  2. sudo arch-chroot /mnt into my system. Ran yay/pacman but got pacman: error while loading shared libraries: /usr/lib/libcrypto.so.3: file too short.
  3. Exited chroot and went for the sudo pacman -Syu --sysroot /mnt
  4. Got a bunch of errors like the following and confirmed (I assume it’s due to different keyring on the live-usb?)
error: openssl: signature from "Pierre Schmitz <pierre@archlinux.org>" is invalid
:: File /mnt/var/cache/pacman/pkg/openssl-3.3.0-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]
  1. Go an error error: failed to commit transaction (conflicting files) in the file conflict check.
  2. Retry… this time going for the heavy method as last time (but using --sysroot instead of chroot’ing) sudo pacman -Syu --sysroot /mnt --overwrite "*" and pacman was doing its job… Got a few concerning messages like
/usr/bin/dkms: line 2512: /dev/fd/63: No such file or directory
Warning: dkms will not function properly if /proc is not mounted.
grep: /proc/cpuinfo: No such file or directory
Error! Bad return status for module build on kernel: 6.8.8-arch1-1 (x86_64)
Consult /var/lib/dkms/nvidia/550.76/build/make.log for more information.
==> WARNING: `dkms install --no-depmod nvidia/550.76 -k 6.8.8-arch1-1' exited 10

Which doesn’t exactly raise my confidence.
At least find /usr/lib -size 0 returns 12 files less than at the beginning, but 605 files more than on the live-usb (which gives me confidence considering it’s a long running system with lots of packages installed)

Going for the reboot, let’s see how it goes…
Wondering why the --sysroot didn’t work last time - my suspicion as a noob was always because the EOS live-CD at the time was pretty outdated, and I probably should’ve done an update first, but that’s just my speculation. Would appreciate enlightenment on the viability of --sysroot in these cases, when it works, when it won’t work.

2

So, that didn’t work, still getting

failed to execute /sbin/init
failed to execute fallback shell

back in the live-usb :thinking:

Going for arch-chroot and pacman -S $(pacman -Qnq) --overwrite '*' now…
It seems like reinstall was done but along the way I got

ldconfig: File /usr/lib/libnss_resolve.so.2 is empty, not checked.
ldconfig: File /usr/lib/libsystemd.so.0 is empty, not checked.
ldconfig: File /usr/lib/libudev.so is empty, not checked.
ldconfig: File /usr/lib/libnss_mymachines.so.2 is empty, not checked.
ldconfig: File /usr/lib/libudev.so.1.7.8 is empty, not checked.
ldconfig: File /usr/lib/libsystemd.so is empty, not checked.
ldconfig: File /usr/lib/libudev.so.1 is empty, not checked.
ldconfig: File /usr/lib/libnss_myhostname.so.2 is empty, not checked.
ldconfig: File /usr/lib/libsystemd.so.0.38.0 is empty, not checked.
ldconfig: File /usr/lib/libnss_systemd.so.2 is empty, not checked.

and at the end

killall: /proc lacks process entries (not mounted ?)
(32/36) Checking which packages need to be rebuilt
fatal library error, lookup self
(33/36) Updating systemd-boot
Skipping "/efi/EFI/systemd/systemd-bootx64.efi", same boot loader version in place already.
Skipping "/efi/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
error: command failed to execute correctly

which I am not sure about. Let’s see how the reboot goes this time…

3

Back in my system :partying_face:

Only odd thing was that Linux Kernel 6.8.7 is still listed in the systemd boot menu.
Should I just delete the respecitve entry in /efi/:uuid.../ and /efi/loader/entries and run bootctl update --no-variables as in the /usr/share/libalpm/hooks/systemd-boot.hook pacman hook? Why do I have these kernel remnants in the first place?

Also, anything else I should do after being back in my system?

4

I am just here to chime in. Had updates related to systems and openssl, started an update and kernel panicked right after systemd was upgraded (3/9). Had the same thought of chrooting first and then reinstalling everything with overwrite. Weirdly during this I found out that /var/cache/pacman/pkg/openssl-3.3.0-1-x86_64.pkg.tar.zst was not a valid file, it was corrupted (tar -tf /var/cache/pacman/pkg//var/cache/pacman/pkg/openssl-3.3.0-1-x86_64.pkg.tar.zst was telling that it cannot recognize it as a tar file) downloaded this from garuda linux mirror and extracted it and later was facing the same issue with keys and later the fatal library error, lookup self. The reboot was smooth I can’t say the same about the kernel since I use lts and g14 kernel and they weren’t updated today.

Normally when I have boot inconsistent boot entries, I just let the system do the rebuilding for me (I use systemd-boot) and to trigger this I just re-install existing kernel like yay -S linux-lts linux-lts-headers and this triggers dracut to do it’s job as I don’t trust myself with boot entries.

pacman.log

[ALPM] upgraded openssl (3.2.1-1 -> 3.3.0-1)
[ALPM] upgraded systemd-libs (255.5-2 -> 255.5-3)
[ALPM] upgraded systemd (255.5-2 -> 255.5-3)
5

Thanks for chiming in @rancour. I did yay -S linux linux-headers linux-lts linux-lts-headers but that didn’t do the trick. I still see the Kernel 6.8.7 systemd-boot entry.
I also previously tried reinstall-kernels which based on the description I would expect to take care of this matter:

The script reinstall-kernels both regenerate the initrds and create/update the boot entries.

I am also not eager to mess with /efi/... files directly.

6

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.