Preventing unrecoverable system in the future

I got a new laptop about 6 months ago and settled on endevouros as I enjoyed its minimalist take. I have no issues to speak of except when the doing updates. For some reason, my laptop has frozen twice during system updates, specifically the part where it is installing the new kernel. Rebooting the system tells me something of the effect of kernel not found. While not ideal, it should be possible to chroot into the system and fix it. After mounting my drives, chrooting gives the following error:

chroot: failed to run command '/bin/bash': Input/output error

I spent about an hour trying to figure out how to fix this error, but couldn’t get it to work. I decided to instead reinstall the entire system. Given that this has happened before, it was much faster to do this process the second time because /home was mapped to a separate partition and I also have a script that installs and configures the system back to its original state. However, this is not an ideal solution. I would rather not have the system crash during updates. I looked at journal logs to try to understand what might be causing the issue, but found nothing unusual. If I am to take a rough guess, it may be GPU related since my laptop has intel and nvidia gpus. I am currently on a fresh install, but will be holding my breath any time I need to do a system update. What can I do to avoid such problems in the future?

Personally I found the definitive solution in BTRFS and in its snapshots.

In case of problems with an update, just select the previous snapshot directly in the Grub menu.

1 Like

I have considered BTRFS but (1) I need to understand how to use it, which takes some time, and (2) my /home is ext4, and has a lot of files that will take ages to backup.

To learn how to configure and use BTRFS just read the documentation and do some tests in a virtual machine, it is not so complicated.

EOS uses calamares installer which makes BTRFS particularly simple by creating indispensable subvolumes out of the box.

1 Like

I don’t want to use BTRFS.

It’s your choice.

There is a huge amount of learning required to understand how to use BTRFS. I don’t have the time for that.

I recently reformatted, switching from EXT4 to BTRFS. What prompted me to do this, was a system failing to start after changing how a secondary encrypted volume was auto-mounted on startup. Painful to fix, because I wasn’t using BTRFS.

It’s made pretty simple if setup via an EOS installation. It handles all the subvolumes, filesystem compression, etc.

2 Likes

Yes and no, if you use the defaults there isn’t much to learn at all

3 Likes

No you don’t actually. I went to BTRFS cause after reviewing both EXT4 and BTRFS I found BTRFS to be the better file system. There really is no need to understand the underpinnings of it unless one wants to.

It’s made pretty simple use ANY Linux Distro that either has BTRFS as the default file system for install or gives you the option to use BTRFS as the file system during install.

1 Like

Except Ubuntu 24.04.
The new Installer is limited to formatting the partition in BTRFS without creating any subvolume.

Really sad.

And yet another of over a 100 reasons not to ever install uBlunder.

2 Likes

Maybe this is not what you want to read, but

it will take many more ages to rewrite all the lost documents, take the vanished photos again and rebuild all the other stuff on your drive if a (hardware) failure knocks your device out.

5 Likes

Maybe the crash during the update broke bash on your system. But you can still fix broken packages on your system, from the live USB by running pacman from there with the flag --root /mnt so that the effect will be on the mounted system. You can find more info on how to find the broken packages and fix them on the guide in my post here

Late to the topic, I just recently set up grub-btrfs but can only boot into the snapshots in read-only mode. So no changes are permanent.

I found two solutions provided by the devs (make a writable copy of the snapshot or use overlayfs) but I don’t know how to get them to work since I don’t know the equivalent solution for dracut.

Could you tell me the workaround that you used for this? How do you make your snapshots from grub menu writable? And how do you roll back?

After booting the system from GRUB snapshot you have to restore the system using snapper or timeshift.

The convenience of grub-btrfs is that you can boot a broken system without using live system.

1 Like

Why?

Hmm… interesting. When I boot from a Grub snapshot, I always get IO errors when I try to restore with Snapper. That’s why I assumed it had something to do with it being read-only.

It looks like there’s an unrelated issue going on on my side. It’s probably Snapper-related, not grub-btrfs.
I’ll be looking into it.

I have been using brtfs with btrfs-assistant, snapper-support and btrfsmaintenance. Works extremely well without any issues.

1 Like

I can restore my snapshots with Snapper or btrfs-assistant with no issues.

The problem only happens when I boot a snapshot from Grub.
That’s why I thought the problem is related to grub-btrfs at first.