System crashed during update, how f'd am I

General system Kernel, boot, graphics & hardware
1

tl;dr: Hard shutdown during update let to missing /boot/vmlinuz-linux, cannot install anything using pacman from live-USB. Is there any hope?

Earlier today I ran yay -Syyu and my laptop froze while applying the update (unfortunately I did not record when exactly). It completely stopped responding so after about 15 minutes I thought I had no choice except to do a hard shutdown.
Now when I try to boot I am greeted by file: /boot/vmlinuz-linux not found.

A similar issue was solved on the forum by arch-chrooting from a live-usb and reinstalling linux.

So I tried that, I’m dual booting with windows and the EOS partition is encrypted. So from the live-install I ran udiskctl unlock -b /dev/PA (where PA is that partition), and mount /dev/mapper/... /mnt.
Then I used cat /mnt/etc/fstab to find out that my ESP should be mounted under /boot/efi, so I ran mount /dev/PS /mnt/boot/efi (where PS is the SYSTEM/FAT32` partition that I assume was my ESP).

Unfortunately when I arch-chroot into /mnt running pacman or yay does nothing. The exit code is 0 but there is not output.

I figured this means that pacman was also broken during the interrupted update. From this thead on the arch forums I got the idea to use pacman --root, but after looking at the man page I decided to try with --sysroot, to reinstall it.
However, (running from the live system) this fails with

(1/1) checking keys in keyring
error: GPGME error: Invalid crypto engine
(1/1) checking package integrity
error: GPGME error: Invalid crypto engine`
error: pacman: missing required signatures

It then asks me if I want to remove the corrupted cached .pkg.tar.zst file, but regardless of my answer it then fails.

Ducking this I found some reddit threads with similar problems, however I was unable to really try the solution since pacman-key also does nothing on the arch-chrooted system.

Installing anything else with pacman --sysroot fails in the same way, so I decided to turn off signature checking in /etc/pacman.conf in the live system, but this had no effect on the error I get when trying to install something.

2

As I was writing this I figured I might as well give pacman --root /mnt a shot too. This actually fails differently, namely when checking for conflicting files.
I am tempted to use the --overwrite option, but I’m worried that this will mess things up even more. There are a lot of conflicting files though, so moving them to backup seems somewhat impractical.
How stupid would it be to try --overwrite.?

3

I think it would be best to use the live ISO and arch-chroot and then re-run the updates.

Edit: https://discovery.endeavouros.com/?s=arch-chroot

4

That’s what I tried to describe doing in my post.

I can’t arch-chroot and then update because pacman does nothing.
I can’t update from the live USB with pacman --sysroot because of error: GPGME error: Invalid crypto engine
And I’m not sure if it’s smart to update from the live USB with pacman --root --overwrite "*".

5

For a very similar case, please read through the whole of this topic and hopefully with enough patience and perseverance, you will be able to recover your system like the OP of that thread did:

6

Since I didn’t see any other options and pacman on the actual install is broken anyway, I decided to try running pacman --root /mnt -S pacman from the live-USB, which resulted in the following output.

...
(4/4) Checking which packages need to be rebuilt
pacman: error while leading shared libraries: /usr/lib/libbz2.so.1.0: file too short
pacman: error while leading shared libraries: /usr/lib/libbz2.so.1.0: file too short
pacman: error while leading shared libraries: /usr/lib/libbz2.so.1.0: file too short
pacman: error while leading shared libraries: /usr/lib/libbz2.so.1.0: file too short

When I now run pacman while arch-chrooted into the installed system I get that same error

pacman: error while leading shared libraries: /usr/lib/libbz2.so.1.0: file too short
7

Thanks a lot, that looks similar to my problem. I’m pretty sure my initial update also froze at

( 2/16) Reloading system manager configuration...
1 Like
8

My system is back up and running and the issue seems to be resolved.

To summarise what I eventually did:
I booted from a live-USB and mounted my root and efi partitions. Then I used the pacman install from the live USB to reinstall pacman on the laptop with sudo pacman --root /mnt --overwrite "*" -S pacman.
At this point I was able to at least get some output from running pacman in the chroot-ed system, but it was just errors about libraries in /usr/lib being too short.
Lucky after just reinstalling one of them pacman worked again and I was able to get a list of all the packages installed on my system with pacman -Qqn.
I used that list to reinstall every package from the live-USB

cat /mnt/package_list | sudo pacman --root /mnt --overwrite "*" -S -

Thanks pebcak for that very helpful link and thanks ricklinux, your solution in said link also worked for me.

2 Likes
9

Very nicely resolved! Glad that you got your system back up and running!

:enos: :wave:t5:

2 Likes
10

Thanks :slight_smile: It’s really amazing how good the arch toolbox is! I was all but certain that I’d have to reinstall the entire system and restore everything from backup.

1 Like
11

Yes it is really amazing!

The tools and the ArchWiki, an unbeatable combination.

Truly impressive that a system so seemingly utterly broken as to warrant a a reinstallation can be rescued and function normally as if the breakage never had occurred.

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.