Can't arch-chroot in order to recover broken system

pebcak · December 22, 2021, 7:06pm

Hi,

Almost two weeks ago, while performing an update, the whole system froze. I couldn’t get into TTY to relaunch the update from there so I had to force a shutdown in the hope to be able to recover the system from chroot.

Now it seems that I can’t arch-chroot into the broken system. This is what I get when trying:

# mount -o subvol=@,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt
# mount -o subvol=@home,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt/home
# mount -o subvol=@pkg,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt/var/cache/pacman/pkg
# mount /dev/nvme0n1p1 /mnt/efi
# arch-chroot /mnt 
chroot: failed to run command ‘/bin/bash’: No such file or directory

Following the advice here:

# chroot /mnt /bin/bash 
chroot: failed to run command ‘/bin/bash’: No such file or directory

# arch-chroot /mnt /bin/bash 
chroot: failed to run command ‘/bin/bash’: No such file or directory

These are the last lines from the relevant date in pacman.log

[2021-12-09T18:02:25+0100] [PACMAN] Running 'pacman -S -y -u --config /etc/pacman.conf --'
[2021-12-09T18:02:25+0100] [PACMAN] synchronizing package lists
[2021-12-09T18:02:27+0100] [PACMAN] starting full system upgrade
[2021-12-09T18:02:39+0100] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-12-09T18:02:39+0100] [ALPM] running '71-dkms-remove.hook'...
[2021-12-09T18:02:39+0100] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod vboxhost/6.1.30_OSE -k 5.10.83-1-lts
[2021-12-09T18:02:39+0100] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod vboxhost/6.1.30_OSE -k 5.15.6-zen2-1-zen
[2021-12-09T18:02:40+0100] [ALPM-SCRIPTLET] ==> depmod 5.10.83-1-lts
[2021-12-09T18:02:54+0100] [ALPM-SCRIPTLET] ==> depmod 5.15.6-zen2-1-zen
[2021-12-09T18:02:58+0100] [ALPM] transaction started

Here is the relevant part of the /etc/fstab:

# <file system>                           <mount point>                         <type>  <options>                                                                  <dump>  <pass>
UUID=5C1D-26F4                            /efi                                  vfat    umask=0077 0 2
/efi/enos-gnome                           /boot                                 none    bind   0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /                                     btrfs   subvol=@,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /home                                 btrfs   subvol=@home,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /var/cache/pacman/pkg                 btrfs   subvol=@pkg,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
#
#
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1                                       /swap                  btrfs    subvol=@swap
/swap/swapfile                                                                  none                   swap     defaults 0 0

This is (was ) my main EnOS system for quite a while. What is ironic is that I had removed all the system snapshots previously to rebuild it all from scratch again.
I would really appreciate any help to recuperate the system if possible. Thank you in advance!

jonathon · December 22, 2021, 7:26pm

If /bin/bash doesn’t exist then this could be due to $PATH not containing /bin (unlikely) or a missing /bin → /usr/bin symlink (more likely). Or you’re mounting the incorrect partition…

After mounting to /mnt, have a look to make sure /mnt/bin/bash exists:

ls -l /mnt/bin/bash
ls -ld /mnt/bin

pebcak · December 22, 2021, 7:43pm

Thanks @jonathon for the reply!

Here is what I get running the commands in your post:

ls -l /mnt/bin/bash 
-rwxr-xr-x 1 root root 948624 Nov 21 20:31 /mnt/bin/bash

and

ls -ld /mnt/bin 
lrwxrwxrwx 1 root root 7 Nov 11 10:14 /mnt/bin -> usr/bin

This is a picture after attempting to start the system (sorry for the bad quality):

Imagepipe_10

Perhaps

ERROR: Root device mounted successfully, but /sbin/init does not exist.

give some other clue?

Edit: Will these be of any help?

ls -l /mnt/sbin/init  
ls: cannot access '/mnt/sbin/init': No such file or directory
 ls -ld /mnt/sbin 
ls: cannot access '/mnt/sbin': No such file or directory

jonathon · December 22, 2021, 7:47pm

Probably means systemd wasn’t installed, or the initramfs wasn’t created, or something along those lines. TLDR; the system is hosed.

A chroot is the correct approach, then re-run the update and check for missing files. You may also need to reinstall all packages (pacman -S $(pacman -Qq)) to get any missed hooks to fire and also replace any corrupt files.

If this:

doesn’t work then also try

# arch-chroot /mnt /usr/bin/bash

pebcak · December 22, 2021, 7:59pm

I was afraid that this might be the case

# arch-chroot /mnt /usr/bin/bash

gives the same as before:

chroot: failed to run command ‘/usr/bin/bash’: No such file or directory

I was hoping so as well, to be able to recover the system from chroot but it seems impossible now.

ERROR: Root device mounted successfully, but /sbin/init does not exist.

Isn’t sbin also a link to /usr/bin ? I don’t seem to have it in the broken system.
Should I make one again or that won’t make any difference at this stage?

jonathon · December 22, 2021, 8:02pm

Grr. What’s being reported as $PATH ?

No, and recovery from booting the broken system is going to be very difficult - a chroot will get you networking so you can download packages etc.

dalto · December 22, 2021, 8:03pm

Have you tried copying the bash executable from the ISO into the target?

pebcak · December 22, 2021, 8:07pm

$PATH 
bash: /usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

I am actually trying from another Arch system on the same disk.
And yes I actually copied the bash executable from this system into the broken one but to no avail.

jonathon · December 22, 2021, 8:10pm

OK, the next thing to check are the directory permissions for the tree,

ls -ld /mnt
ls -ld /mnt/usr
ls -ld /mnt/usr/bin

Each one should be 755 (u=rwX,g=rX,o=rX).

pebcak · December 22, 2021, 8:14pm

# ls -ld /mnt 
drwxr-xr-x 1 root root 252 Dec 22 21:01 /mnt

# ls -ld /mnt/usr 
drwxr-xr-x 1 root root 76 Dec 22 21:01 /mnt/usr
# ls -ld /mnt/usr/bin 
drwxr-xr-x 1 root root 65726 Dec 22 20:15 /mnt/usr/bin

Looks like they are not 755?

jonathon · December 22, 2021, 8:25pm

Directory permissions are all correct. The only other reason I can think of for seeing a “can’t be found” error for a file that exists is when the architecture is wrong (e.g. running a 64-bit binary on a 32-bit OS). But that can’t be the case here.

Have you run a btrfs check /dev/nvme0n1p2 ? (umount the filesystem first)

pebcak · December 22, 2021, 8:32pm

That sounds good! Thanks for checking up!

I’ll do so. I have been trying the recovery from another Arch system on the same partition (in another btrfs subvolume). I’ll boot up my Live EnOS usb and run the disk check.

BTW, is it possible to check a subvolume for corruption/inconsistency?

otherbarry · December 22, 2021, 8:35pm

Did this breakage come after an update or …

What exactly did you do here?

I’ve seen btrfs itself doing less.

Do you have a system backup image to restore?

jonathon · December 22, 2021, 8:37pm

Also try to chroot from there, just in case it was getting confused by the subvolumes.

pebcak · December 22, 2021, 8:39pm

Sort of. The whole system froze in the middle of the update process. I couldn’t get into TTY or shutdown the system gracefully. So there was a hard shutdown which possibly worsened the whole thing.

I had some snapshots with Timeshift. I had deleted them some time before. Ufortunately I postponed making new ones and now I am without one when I most need it.

pebcak · December 22, 2021, 8:41pm

I had tried that before with the same result but I will try again.
I’ll get back in a bit.

dalto · December 22, 2021, 8:43pm

I believe you need to check the whole volume but I never investigated in detail.

manuel · December 22, 2021, 8:53pm

Are you sure your mounts in the beginning are correct?
They look strange to me, but then again I’m no btrfs expert…

otherbarry · December 22, 2021, 8:58pm

When an update freezes it can be handy to take note of what it is doing when it froze, if you can.

Can you navigate the mounted file system normally? If the file system is not corrupt then the update froze during something important and borked the system.

@jonathon @dalto Is it work trying pacstrap /mnt base as a last resort?

dalto · December 22, 2021, 9:00pm

They look fine to me with the exception that there is no need to mount /home in a chroot. Of course, it shouldn’t hurt anything either.