Kernel panic during update / issues with pacman sysroot

ArchMonkey · March 15, 2024, 11:19am

During my last update via yay (so pacman -Syu) I ran into a kernel panic, the last output was

(56/56) upgrading yay
:: Running post-transaction hooks...
( 1/16) Creating system user accounts...
( 2/16) Reloading system manager configuration...

After a hard reboot I am getting (shortened a little where ...)

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
CPU: 2 PID: 1 Comm: systend Tainted: P
Hardware name: ...........................................
5.7.9 arch1-1 #1 ad54415bbff2f0801422a3b76df850f68e71ecal
Call Trace:
< TASK>
dump_stack_1v1+0x47/0x60
panic+0x334/0x360
do_exit+0x9ce/0xb60
do_group_exit+0x31/0x80
—_x64_sys_exit_group+0x18/0x20
do_syscall_64+0x61/0xeD
? exc_page_fault+0x7f/0x180
entry_SYSCALL_
_54_after_hwframe+bxbe/0x75
RIP: 0033:0x75641daef095
30.5663441
Code: ba 01 00 00 00 68 ca 00 00 00 0f 05 c3 0f 1f 84 00 00 00 00 00 f3 Of 1e fa ba e7 00…
...
</TASK>
Kernel Offset: 0x22a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xfffffffffffffff)
 ---L end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]---

So I have downloaded the Endeavour Live ISO mounted my / /home and /efi.
I can now sudo arch-chroot /mnt into my system.

[root@EndeavourOS /]# pacman -Syu
pacman: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short

No wonder as find /mnt/usr/lib -size 0 gives me lots of corrupt .pak files.

Running sudo pacman -Syu --sysroot /mnt on the live ISO didn’t succeed either, resulting in:

:: Retrieving packages...
 run-parts-5.17-1-x86_64                             29.3 KiB   157 KiB/s 00:00 [---------------------------------------------] 100%
 zsh-5.9-5-x86_64                                     2.2 MiB  3.41 MiB/s 00:01 [---------------------------------------------] 100%
 Total (2/2)                                          2.2 MiB  3.29 MiB/s 00:01 [---------------------------------------------] 100%
(2/2) checking keys in keyring                                                  [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
(2/2) checking package integrity                                                [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
error: run-parts: missing required signature
:: File /var/cache/pacman/pkg/run-parts-5.17-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).

Note that the live ISO seems somewhat behind with Linux kernel 6.7.1.arch1-1 and 384 packages to upgrade. This already makes me wonder, is this a good pre-condition for using pacman --sysroot in the first place?

Thank you for any pointers into the right direction!

dalto · March 15, 2024, 11:25am

That looks like a mess that might need several fixes to get it working again.

First try pacman static. It is statically linked so it won’t have dependency issues.

Go back into the arch-chroot and try using pacman static:

wget https://pkgbuild.com/~morganamilo/pacman-static/x86_64/bin/pacman-static
./pacman-static -Syu

ArchMonkey · March 15, 2024, 11:38am

Thank you dalto, using pacman-static -Syu I am getting the same issue:

...
:: Retrieving packages...
 run-parts-5.17-1-x86_64                             29.3 KiB   195 KiB/s 00:00 [---------------------------------------------] 100%
 zsh-5.9-5-x86_64                                     2.2 MiB  5.28 MiB/s 00:00 [---------------------------------------------] 100%
 Total (2/2)                                          2.2 MiB  4.95 MiB/s 00:00 [---------------------------------------------] 100%
(2/2) checking keys in keyring                                                  [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
(2/2) checking package integrity                                                [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
error: run-parts: missing required signature
:: File /var/cache/pacman/pkg/run-parts-5.17-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).

dalto · March 15, 2024, 11:43am

OK, try this from the chroot:

wget https://arch.phinau.de/core/os/x86_64/gnupg-2.4.5-1-x86_64.pkg.tar.zst
wget https://arch.phinau.de/core/os/x86_64/gpgme-1.23.2-1-x86_64.pkg.tar.zst
./pacman-static -U ./gnupg-2.4.5-1-x86_64.pkg.tar.zst ./gpgme-1.23.2-1-x86_64.pkg.tar.zst
./pacman-static -Syu

ArchMonkey · March 15, 2024, 12:02pm

Trying, but more errors:

[root@EndeavourOS _repair]# pacman-static -U ./gnupg-2.4.5-1-x86_64.pkg.tar.zst ./gpgme-1.23.2-1-x86_64.pkg.tar.zst

loading packages...
warning: gnupg-2.4.5-1 is up to date -- reinstalling
warning: gpgme-1.23.2-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (2)  Old Version  New Version  Net Change

gnupg        2.4.5-1      2.4.5-1        9.62 MiB
gpgme        1.23.2-1     1.23.2-1       0.00 MiB

Total Installed Size:  11.10 MiB
Net Upgrade Size:       9.62 MiB

:: Proceed with installation? [Y/n] 
(2/2) checking keys in keyring                                                  [---------------------------------------------] 100%
(2/2) checking package integrity                                                [---------------------------------------------] 100%
(2/2) loading package files                                                     [---------------------------------------------] 100%
(2/2) checking for file conflicts                                               [---------------------------------------------] 100%
error: failed to commit transaction (conflicting files)
gnupg: /usr/bin/addgnupghome exists in filesystem
gnupg: /usr/bin/applygnupgdefaults exists in filesystem
gnupg: /usr/bin/dirmngr exists in filesystem
...
gnupg: /usr/share/man/man8/addgnupghome.8.gz exists in filesystem
gnupg: /usr/share/man/man8/applygnupgdefaults.8.gz exists in filesystem
gnupg: /usr/share/man/man8/dirmngr.8.gz exists in filesystem
Errors occurred, no packages were upgraded.

wget doesn’t work on chroot by the way, I am getting everything through the live-iso, then copying it into /mnt. sudo doens’t work either.

dalto · March 15, 2024, 12:05pm

Try

pacman-static -U ./gnupg-2.4.5-1-x86_64.pkg.tar.zst ./gpgme-1.23.2-1-x86_64.pkg.tar.zst --overwrite "*"
pacman-static -Syu

If we can get to the point where we can get pacman-static working, we are probably going to have to use it to reinstall all your packages.

ArchMonkey · March 15, 2024, 12:22pm

Ok, with the --overwrite "*" option I am getting

ldconfig: File /usr/lib/libGLX_nvidia.so is empty, not checked.
ldconfig: File /usr/lib/libunwind-setjmp.so.0.0.0 is empty, not checked.
ldconfig: File /usr/lib/libOSMesa.so is empty, not checked.
ldconfig: File /usr/lib/libnvidia-opticalflow.so.550.54.14 is empty, not checked.
...
... lots more of these ...
... and then ...
...
:: Running post-transaction hooks...
(1/5) Reloading system manager configuration...
systemd-detect-virt: error while loading shared libraries: /usr/lib/liblzma.so.5: file too short
  Skipped: Current root is not booted.
(2/5) Reloading user manager configuration...
systemd-detect-virt: error while loading shared libraries: /usr/lib/liblzma.so.5: file too short
  Skipped: Current root is not booted.
(3/5) Arming ConditionNeedsUpdate...
(4/5) Checking which packages need to be rebuilt
join: /dev/fd/63: No such file or directory
grep: error while loading shared libraries: /usr/lib/libpcre2-8.so.0: file too short
pacconf: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
grep: error while loading shared libraries: /usr/lib/libpcre2-8.so.0: file too short
pactree: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
pacconf: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
pactree: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
pacman: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
comm: /dev/fd/62: No such file or directory
pacman: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
ps: error while loading shared libraries: /usr/lib/liblzma.so.5: file too short
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory
grep: error while loading shared libraries: /usr/lib/libpcre2-8.so.0: file too short
tee: /tmp/tmp.zYeROZLDLR/16291: No such file or directory
comm: /dev/fd/62: No such file or directory
pacconf: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
pacman: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
tee: /tmp/tmp.zYeROZLDLR/29006: No such file or directory
pacconf: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
comm: /dev/fd/62: No such file or directory
pacman: error while loading shared libraries: /usr/lib/libexpat.so.1: file too short
(5/5) Updating the info directory file...

dalto · March 15, 2024, 12:26pm

A lot of your shared libraries are corrupt.

There are a couple of paths you can take.

You can manually download the broken library packages from a mirror and install them with pacman-static -U as we did above. This may take some time but will get your system back intact eventually.
You can backup your data and reinstall.

ArchMonkey · March 15, 2024, 12:42pm

So, with the first option that would be libexpat, libpcre, and liblzma? Or will there be more after I have patched these, and there could be other upstream dependencies?

I guess we’re talking about https://wiki.archlinux.org/title/Pacman#By_manually_extracting

dalto · March 15, 2024, 12:44pm

There could be more. Basically, keep trying until you run out of errors.

No. Don’t do it that way.

Use pacman-static -U as above with gpg.

ArchMonkey · March 15, 2024, 1:46pm

pacman-static -U $(find . -type f -name \*.zst | tr '\n' ' ') --overwrite "*"

Eventually ended up re-installing all of

Package (7)   Old Version  New Version  Net Change

audit         4.0.1-2      4.0.1-2        1.02 MiB
expat         2.6.2-1      2.6.2-1        0.00 MiB
gnupg         2.4.5-1      2.4.5-1        0.00 MiB
gpgme         1.23.2-1     1.23.2-1       0.00 MiB
libunistring  1.2-1        1.2-1          2.49 MiB
pcre2         10.43-1      10.43-1        0.00 MiB
xz            5.6.1-1      5.6.1-1        0.00 MiB

First I got:

:: Running post-transaction hooks...
(1/6) Reloading system manager configuration...
  Skipped: Running in chroot.
(2/6) Reloading user manager configuration...
  Skipped: Running in chroot.
(3/6) Creating temporary files...
/proc/ is not mounted, but required for successful operation of systemd-tmpfiles. Please mount /proc/. Alternatively, consider using the --root= or --image= switches.
error: command failed to execute correctly
(4/6) Arming ConditionNeedsUpdate...
(5/6) Checking which packages need to be rebuilt
join: /dev/fd/63: No such file or directory
Error, do this: mount -t proc proc /proc
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory
comm: /dev/fd/62: No such file or directory
tee: /tmp/tmp.AC7WjPyhwq/3131: No such file or directory
comm: /dev/fd/62: No such file or directory
tee: /tmp/tmp.AC7WjPyhwq/9330: No such file or directory
comm: /dev/fd/62: No such file or directory
(6/6) Updating the info directory file...

I have then mounted/bound /proc, /dev and /sys and now it seems successful:

... still lots of these ...
dconfig: File /usr/lib/libGLX_nvidia.so is empty, not checked.
ldconfig: File /usr/lib/libunwind-setjmp.so.0.0.0 is empty, not checked.
ldconfig: File /usr/lib/libOSMesa.so is empty, not checked.
...
:: Running post-transaction hooks...
(1/6) Reloading system manager configuration...
  Skipped: Running in chroot.
(2/6) Reloading user manager configuration...
  Skipped: Running in chroot.
(3/6) Creating temporary files...
(4/6) Arming ConditionNeedsUpdate...
(5/6) Checking which packages need to be rebuilt
fatal library error, lookup self
(6/6) Updating the info directory file...

But a subsequent pacman-static -Syu is still resulting in

:: Retrieving packages...
 run-parts-5.17-1-x86_64                             29.3 KiB   175 KiB/s 00:00 [---------------------------------------------] 100%
 zsh-5.9-5-x86_64                                     2.2 MiB  2.59 MiB/s 00:01 [---------------------------------------------] 100%
 Total (2/2)                                          2.2 MiB  2.53 MiB/s 00:01 [---------------------------------------------] 100%
(2/2) checking keys in keyring                                                  [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
(2/2) checking package integrity                                                [---------------------------------------------] 100%
error: GPGME error: Invalid crypto engine
error: GPGME error: Invalid crypto engine
error: run-parts: missing required signature
:: File /var/cache/pacman/pkg/run-parts-5.17-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] 
error: zsh: missing required signature
:: File /var/cache/pacman/pkg/zsh-5.9-5-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] 
error: failed to commit transaction (invalid or corrupted package (PGP signature))
Errors occurred, no packages were upgraded.

dalto · March 15, 2024, 1:50pm

If you had that error. You must have been in a normal chroot, not an arch-chroot

Make sure you are in an arch-chroot

ArchMonkey · March 15, 2024, 1:57pm

Checking my history, you’re right, I went into normal chroot once or twice

Still, arch-chroot and then pacman-static -Syu is still resulting in error: GPGME error: Invalid crypto engine and

error: failed to commit transaction (invalid or corrupted package (PGP signature))
Errors occurred, no packages were upgraded.

dalto · March 15, 2024, 2:10pm

Try manually installing libgcrypt, gnutls and libgpg-error

ArchMonkey · March 15, 2024, 2:21pm

Installed all of

gnutls        3.8.3-1
libgcrypt     1.10.3-1
libgpg-error  1.48-1

But still getting the same error on pacman-static -Syu.

pebcak · March 15, 2024, 2:25pm

@dalto
Could removing the /etc/pacman.d/gnupg/ and building it up from scratch remedy this situation?

dalto · March 15, 2024, 2:37pm

Worth trying.

It could also be a dependency issue caused by the partial update.

ArchMonkey · March 15, 2024, 3:01pm

@pebcak thank you for chiming in, I assume that could be done with

sudo mv /etc/pacman.d/gnupg /etc/pacman.d/gnupg_old
sudo pacman-key --init
sudo pacman-key --populate archlinux
sudo pacman-key --refresh-keys

?

It could also be a dependency issue caused by the partial update.

@dalto I found this issue and also reinstalled gpgme. There are no more errors hinting at dependencies though, how to narrow down on other potentially corrupted dependencies?

pebcak · March 15, 2024, 3:08pm

Yes. Might be worth re-initializing your keys. Won’t harm anyways.

I would add endeavouros as well to:

sudo pacman-key --populate archlinux

Other things I would personally try would be to update the mirrors.

Also perhaps

rm -r /var/lib/pacman/sync/

and then:

./pacman-static -Syu

ArchMonkey · March 15, 2024, 3:34pm

I just tried to disable signature checking and still got a

(1/2) Arming ConditionNeedsUpdate...
(2/2) Checking which packages need to be rebuilt
fatal library error, lookup self

After rm -r /var/lib/pacman/sync/ I am getting a successful pacman -Syu.
I will try to re-initialize keys though.

Commands like nvim still don’t work, so I guess I’ve still got to reinstall all my package. Wondering what’s the best way to do that, without marking all packages as explicitly installed. pacman -Qnq | pacman -S? (as per Arch docs)