Can't arch-chroot in order to recover broken system

Hi,

Almost two weeks ago, while performing an update, the whole system froze. I couldn’t get into TTY to relaunch the update from there so I had to force a shutdown in the hope to be able to recover the system from chroot.

Now it seems that I can’t arch-chroot into the broken system. This is what I get when trying:

# mount -o subvol=@,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt
# mount -o subvol=@home,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt/home
# mount -o subvol=@pkg,defaults,noatime,space_cache,autodefrag,compress=lzo /dev/nvme0n1p2 /mnt/var/cache/pacman/pkg
# mount /dev/nvme0n1p1 /mnt/efi
# arch-chroot /mnt 
chroot: failed to run command ‘/bin/bash’: No such file or directory

Following the advice here:

# chroot /mnt /bin/bash 
chroot: failed to run command ‘/bin/bash’: No such file or directory

# arch-chroot /mnt /bin/bash 
chroot: failed to run command ‘/bin/bash’: No such file or directory

These are the last lines from the relevant date in pacman.log

[2021-12-09T18:02:25+0100] [PACMAN] Running 'pacman -S -y -u --config /etc/pacman.conf --'
[2021-12-09T18:02:25+0100] [PACMAN] synchronizing package lists
[2021-12-09T18:02:27+0100] [PACMAN] starting full system upgrade
[2021-12-09T18:02:39+0100] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-12-09T18:02:39+0100] [ALPM] running '71-dkms-remove.hook'...
[2021-12-09T18:02:39+0100] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod vboxhost/6.1.30_OSE -k 5.10.83-1-lts
[2021-12-09T18:02:39+0100] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod vboxhost/6.1.30_OSE -k 5.15.6-zen2-1-zen
[2021-12-09T18:02:40+0100] [ALPM-SCRIPTLET] ==> depmod 5.10.83-1-lts
[2021-12-09T18:02:54+0100] [ALPM-SCRIPTLET] ==> depmod 5.15.6-zen2-1-zen
[2021-12-09T18:02:58+0100] [ALPM] transaction started

Here is the relevant part of the /etc/fstab:

# <file system>                           <mount point>                         <type>  <options>                                                                  <dump>  <pass>
UUID=5C1D-26F4                            /efi                                  vfat    umask=0077 0 2
/efi/enos-gnome                           /boot                                 none    bind   0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /                                     btrfs   subvol=@,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /home                                 btrfs   subvol=@home,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1 /var/cache/pacman/pkg                 btrfs   subvol=@pkg,defaults,noatime,space_cache,autodefrag,compress=lzo  0 0
#
#
UUID=0d19c35e-8b2b-4844-a2e6-359ae17575e1                                       /swap                  btrfs    subvol=@swap
/swap/swapfile                                                                  none                   swap     defaults 0 0

This is (was :cry:) my main EnOS system for quite a while. What is ironic is that I had removed all the system snapshots previously to rebuild it all from scratch again.
I would really appreciate any help to recuperate the system if possible. Thank you in advance!

If /bin/bash doesn’t exist then this could be due to $PATH not containing /bin (unlikely) or a missing /bin/usr/bin symlink (more likely). Or you’re mounting the incorrect partition…

After mounting to /mnt, have a look to make sure /mnt/bin/bash exists:

ls -l /mnt/bin/bash
ls -ld /mnt/bin

Thanks @jonathon for the reply!

Here is what I get running the commands in your post:

ls -l /mnt/bin/bash 
-rwxr-xr-x 1 root root 948624 Nov 21 20:31 /mnt/bin/bash

and

ls -ld /mnt/bin 
lrwxrwxrwx 1 root root 7 Nov 11 10:14 /mnt/bin -> usr/bin

This is a picture after attempting to start the system (sorry for the bad quality):

Perhaps

ERROR: Root device mounted successfully, but /sbin/init does not exist.

give some other clue?

Edit: Will these be of any help?

ls -l /mnt/sbin/init  
ls: cannot access '/mnt/sbin/init': No such file or directory
 ls -ld /mnt/sbin 
ls: cannot access '/mnt/sbin': No such file or directory

Probably means systemd wasn’t installed, or the initramfs wasn’t created, or something along those lines. TLDR; the system is hosed. :grin:

A chroot is the correct approach, then re-run the update and check for missing files. You may also need to reinstall all packages (pacman -S $(pacman -Qq)) to get any missed hooks to fire and also replace any corrupt files.

If this:

doesn’t work then also try

# arch-chroot /mnt /usr/bin/bash 
1 Like

I was afraid that this might be the case :sweat_smile: :man_facepalming:t5:

# arch-chroot /mnt /usr/bin/bash

gives the same as before:

chroot: failed to run command ‘/usr/bin/bash’: No such file or directory

I was hoping so as well, to be able to recover the system from chroot but it seems impossible now.

ERROR: Root device mounted successfully, but /sbin/init does not exist.

Isn’t sbin also a link to /usr/bin ? I don’t seem to have it in the broken system.
Should I make one again or that won’t make any difference at this stage?

Grr. What’s being reported as $PATH ?

No, and recovery from booting the broken system is going to be very difficult - a chroot will get you networking so you can download packages etc.

1 Like

Have you tried copying the bash executable from the ISO into the target?

1 Like
$PATH 
bash: /usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

I am actually trying from another Arch system on the same disk.
And yes I actually copied the bash executable from this system into the broken one but to no avail.

OK, the next thing to check are the directory permissions for the tree,

ls -ld /mnt
ls -ld /mnt/usr
ls -ld /mnt/usr/bin

Each one should be 755 (u=rwX,g=rX,o=rX).

1 Like
# ls -ld /mnt 
drwxr-xr-x 1 root root 252 Dec 22 21:01 /mnt

# ls -ld /mnt/usr 
drwxr-xr-x 1 root root 76 Dec 22 21:01 /mnt/usr
# ls -ld /mnt/usr/bin 
drwxr-xr-x 1 root root 65726 Dec 22 20:15 /mnt/usr/bin

Looks like they are not 755?

Directory permissions are all correct. The only other reason I can think of for seeing a “can’t be found” error for a file that exists is when the architecture is wrong (e.g. running a 64-bit binary on a 32-bit OS). But that can’t be the case here.

Have you run a btrfs check /dev/nvme0n1p2 ? (umount the filesystem first)

1 Like

That sounds good! Thanks for checking up!

I’ll do so. I have been trying the recovery from another Arch system on the same partition (in another btrfs subvolume). I’ll boot up my Live EnOS usb and run the disk check.

BTW, is it possible to check a subvolume for corruption/inconsistency?

Did this breakage come after an update or …

What exactly did you do here?

I’ve seen btrfs :poop: itself doing less.

Do you have a system backup image to restore?

1 Like

Also try to chroot from there, just in case it was getting confused by the subvolumes.

1 Like

Sort of. The whole system froze in the middle of the update process. I couldn’t get into TTY or shutdown the system gracefully. So there was a hard shutdown which possibly worsened the whole thing.

I had some snapshots with Timeshift. I had deleted them some time before. Ufortunately I postponed making new ones and now I am without one when I most need it.

I had tried that before with the same result but I will try again.
I’ll get back in a bit.

I believe you need to check the whole volume but I never investigated in detail.

1 Like

Are you sure your mounts in the beginning are correct?
They look strange to me, but then again I’m no btrfs expert…

1 Like

When an update freezes it can be handy to take note of what it is doing when it froze, if you can.

Can you navigate the mounted file system normally? If the file system is not corrupt then the update froze during something important and borked the system.

@jonathon @dalto Is it work trying pacstrap /mnt base as a last resort?

1 Like

They look fine to me with the exception that there is no need to mount /home in a chroot. Of course, it shouldn’t hurt anything either.

2 Likes