HOWTO - GPT/UEFI install with full disk encryption: BTRFSonLUKS with separate root, home and pkg subvolumes; hibernation with a swapfile; auto-snapshots with easy system rollback (GUI); boot into snapshots

Warning: Most of what is done here in this tutorial is now already implemented inside calamares installer!

but … it can be useful for pure arch install…
warning added by joekamprad 21.5.2022


Inspired by some posts from @TomZ I decided to have a (re-)go at btrfs, test its snapshot capabilities and the ease of restoring these, try different subvolume setups and check if calamares allows for an encrypted install.

As a result I have decided to make the following my new default type of install:

  • BTRFSonLUKS (fully LUKS encrypted single partition containing a btrfs filesystem with /boot, /‘root’ and /home)
  • /, /home, swapfile, pkg cache as separate subvolumes; to allow easy snapshotting of root system
  • automated creation of snapshots (on system upgrade & daily & boot)
  • ability to boot into a working snapshot from the grub menu (automated creation of boot entries) and then setting this or other snapshot as new default system (from GUI)
  • swapfile, with the ability to hibernate

With this low-maintenance type of setup it should be possible to recover/rollback from a problematic upgrade by simply booting into a working snapshot (these snapshots are created automatically) and setting this as the new default from within the booted system. You could then just wait a couple of days and then check if the upgrade problems still persist. Most of the time the problem will be fixed. (= Newb friendly :wink:) Note that the creation or restore of a snapshot takes about a second!

The following is a tutorial on setting up this type of system with the help of calamares and a little cli.



+++ QUICK (copy and paste) VERSION ++++++++++++++++++++++++++++++++

(Tested with endeavouros-2019.12.22-x86_64.iso)

The provided cli commands can be copied and pasted/run in bulk, adding only a couple of minutes to your normal install time. If you really need to know more, I have provided a step-by-step explanation of the used commands in the “lengthy version” below.


#01 - Boot into Live-CD environment & set up partitions
With the help of GParted, assuming you have an empty disk

  • create a GPT partition table
  • create sda1 with 300 to 512MiB, format as fat32 and set boot & esp flag
  • leave rest unallocated
+------------+-------------+-------------+
| 300-512MiB |      rest            xGiB |
| boot & esp |      unallocated          |
| flag set   |                           |
| /dev/sda1  |                           |
+------------+-------------+-------------+

With other storage type, dev/sda1 could also be called /dev/nvme0n1p1; in this case just substitute sda1 with nvme0n1p1.


#02 - Change default btrfs calamares settings for fstab

Open a Terminal and type (or mark and copy with ctrl-c and paste in terminal with shift-ctrl-v | you can copy&paste all the lines at once) …

sudo sed -i 's/compress=lzo/compress=zstd/' /usr/share/calamares/modules/fstab.conf
sudo sed -i 's/autodefrag/autodefrag,compress=zstd/' /usr/share/calamares/modules/fstab.conf


#03 - Install system with calamares (off- or online install)

Set up calamares like you usually would (Location, Keyboard, Users, …) but
choose Manual partioning and edit (double-click the partition(s) in the list view) …

  1. /dev/sda1
  • Mount point → /boot/efi
  • set boot flag
  • keep fat32
  1. free space
  • Mount point → /
  • File system → btrfs
  • Encrypt: choose a secure password

Let Calamares finish the installation. If you get a warning about missing esp or boot flag(s) on /dev/sda1 just add them with GParted and continue with the install.

After install, do NOT check the “Restart now” checkbox.


#04 - Choose your swapfile size

e. g. “8G” or “8192M”

swapsize=8G


#05 - Run the following (just copy&paste all the lines at once)

sudo mkdir /mnt/btrfs
btrfsonluks=`sudo blkid -o device | grep luks`
sudo mount -o subvolid=5 $btrfsonluks /mnt/btrfs
sudo sed -i 's/compress=zstd 0 1/compress=zstd 0 0/' /mnt/btrfs/@/etc/fstab
sudo sed -i 's/compress=zstd 0 2/compress=zstd 0 0/' /mnt/btrfs/@/etc/fstab
sudo btrfs subvolume create /mnt/btrfs/@var-cache-pacman-pkg
fstabstr=`grep "/home" /mnt/btrfs/@/etc/fstab`
fstabstr=${fstabstr//\/home/\/var\/cache\/pacman\/pkg}
fstabstr=${fstabstr//@home/@var-cache-pacman-pkg}
sudo sh -c "echo $fstabstr >> /mnt/btrfs/@/etc/fstab"
sudo btrfs subvolume create /mnt/btrfs/@swap
sudo mkdir /mnt/btrfs/@/swap
sudo sh -c "printf '\n$btrfsonluks /swap          btrfs   subvol=@swap,defaults,compress=no 0 0\n' >> /mnt/btrfs/@/etc/fstab"
sudo truncate -s 0 /mnt/btrfs/@swap/swapfile
sudo chattr +C /mnt/btrfs/@swap/swapfile
sudo btrfs property set /mnt/btrfs/@swap/swapfile compression none
sudo fallocate -l $swapsize /mnt/btrfs/@swap/swapfile
sudo chmod 600 /mnt/btrfs/@swap/swapfile
sudo mkswap /mnt/btrfs/@swap/swapfile
sudo swapon /mnt/btrfs/@swap/swapfile
sudo sh -c "echo '/swap/swapfile none swap defaults 0 0' >> /mnt/btrfs/@/etc/fstab"
wget https://raw.githubusercontent.com/osandov/osandov-linux/master/scripts/btrfs_map_physical.c
gcc -O2 -o btrfs_map_physical btrfs_map_physical.c
offset=`sudo ./btrfs_map_physical /mnt/btrfs/@swap/swapfile`
offset_arr=(`echo ${offset}`)
offset_pagesize=(`getconf PAGESIZE`)
offset=$(( offset_arr[25] / offset_pagesize ))
sudo sed -i "s/loglevel=3/loglevel=3 resume_offset=$offset/" /mnt/btrfs/@/etc/default/grub
sudo sed -i 's/keymap encrypt filesystems/keymap encrypt filesystems resume/' /mnt/btrfs/@/etc/mkinitcpio.conf
sudo mkdir /mnt/chroot
sudo mount -o compress=zstd,subvol=@ $btrfsonluks /mnt/chroot
efidevice=`sudo blkid -o device -l -t TYPE=vfat`
sudo mount $efidevice /mnt/chroot/boot/efi
sudo arch-chroot /mnt/chroot
sudo pacman --noconfirm -S cronie
sudo systemctl enable cronie.service
sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo mkinitcpio -p linux


#06 - REBOOT into your new system & run the following …

sudo pacman --noconfirm -Syy &&
yay --noconfirm -S timeshift timeshift-autosnap grub-btrfs

Then run …

sudo cp /etc/default/timeshift.json /etc/timeshift.json &&
fs_uuid=`sudo blkid -o device | grep luks` &&
fs_uuid=`sudo blkid -o value -s UUID $fs_uuid` &&
sudo sed -i "s/backup_device_uuid\" : \"/backup_device_uuid\" : \"$fs_uuid/" /etc/timeshift.json &&
fs_uuid=`sudo blkid -o device -l -t TYPE=crypto_LUKS` &&
fs_uuid=`sudo blkid -o value -s UUID $fs_uuid` &&
sudo sed -i "s/parent_device_uuid\" : \"/parent_device_uuid\" : \"$fs_uuid/" /etc/timeshift.json &&
sudo sed -i 's/run" : "true/run" : "false/' /etc/timeshift.json &&
sudo sed -i 's/btrfs_mode" : "false/btrfs_mode" : "true/' /etc/timeshift.json &&
sudo sed -i 's/  "include_btrfs_home" : "false",/  "include_btrfs_home_for_backup" : "false",\n  "include_btrfs_home_for_restore" : "false",/' /etc/timeshift.json &&
sudo sed -i 's/schedule_daily" : "false/schedule_daily" : "true/' /etc/timeshift.json &&
sudo sed -i 's/schedule_boot" : "false/schedule_boot" : "true/' /etc/timeshift.json &&
sudo sed -i 's/count_boot" : "5/count_boot" : "3/' /etc/timeshift.json

Choose the amount of maximum snapshots your system will allow and keep.
Timeshift cleans the automatically created snapshots but the ones you create manually aren’t handled by Timeshift and would be kept indefinetily. Just choose a high number if you prefer to delete these manually and not let the system only keep the newest (e. g.) 13 snapshots.

max_snapshots=13

Then execute …

sudo sh -c '
cat > /etc/systemd/system/limit-timeshift-snapshots-update-grub.service <<- "EOF"
[Unit]
Description=Keep only latest max_snapshots timeshift snapshots and update grub.cfg before shutdown
Before=poweroff.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecStop=

[Install]
WantedBy=multi-user.target
EOF
' &&
sudo sed -i "s#max_snapshots#$max_snapshots#" /etc/systemd/system/limit-timeshift-snapshots-update-grub.service &&
sudo sed -i "s#ExecStop=#ExecStop=/usr/bin/sh -c \"cd /run/timeshift/backup/timeshift-btrfs/snapshots/ \&\& rm -rf \`ls -1r | awk 'NR>$max_snapshots'\` \&\& /usr/bin/grub-mkconfig -o /boot/grub/grub.cfg\"#" /etc/systemd/system/limit-timeshift-snapshots-update-grub.service &&
sudo systemctl enable limit-timeshift-snapshots-update-grub.service &&
sudo systemctl start limit-timeshift-snapshots-update-grub.service &&
sudo timeshift --create --comments "1st"


#07 - OPTIONAL: to create some desktop shortcuts (to Timeshift, manually create a snapshot and hibernate) run the following …
(Note: to get rid of these later on just delete the shortcuts from your desktop)

sh -c '
cat > ~/Desktop/timeshift-gtk.desktop <<- "EOF"
[Desktop Entry]
Name=Timeshift
MimeType=
Exec=timeshift-launcher
Type=Application
GenericName=System Restore Utility
Terminal=false
Icon=timeshift
Comment=System Restore Utility
X-KDE-StartupNotify=false
Categories=System;
X-GNOME-UsesNotifications=true
EOF
chmod +x ~/Desktop/timeshift-gtk.desktop
'
sh -c '
cat > ~/Desktop/create_snapshot.desktop <<- "EOF"
[Desktop Entry]
Name=Create Snapshot
Comment=Create btrfs snapshot of root
Exec=
Icon=timeshift
StartupNotify=false
Terminal=true
Type=Application
EOF
' 
sed -i "s#Exec=#Exec=sudo sh -c \'timeshift --create --comments \"manual\" \&\& sleep 2\'#" ~/Desktop/create_snapshot.desktop
chmod +x ~/Desktop/create_snapshot.desktop
sh -c '
cat > ~/Desktop/hibernate_alt.desktop <<- "EOF"
[Desktop Entry]
Name=Hibernate
Comment=Hibernate (alternative method)
Exec=
Icon=system-shutdown
StartupNotify=false
Terminal=true
Type=Application
EOF
'
sed -i "s#Exec=#Exec=sudo sh -c \'echo disk > /sys/power/state\'#" ~/Desktop/hibernate_alt.desktop
chmod +x ~/Desktop/hibernate_alt.desktop


DONE! You should now have a fully encrypted system with the ability to rollback to automatically (and manually) created snapshots and be able to hibernate.

Note that this setup isn’t meant to be a backup strategy replacement! The snapshots are stored on the same disk as your system; if this fails the snapshots won’t be accessible. At least your /home subvolume should be backuped to another physical device. So, just apply your normal backup strategy of choice.


# - How to RESTORE a snapshot (previous system state)

A - If you ever need to reset your system to a stored state, just

  1. Launch Timeshift from the menu (or desktop shortcut)
  2. Select a snapshot and hit restore
  3. Reboot

That’s all. Takes mere seconds and doesn’t get any easier.
BTW, you could also restore a snapshot from the cli!


B - In case you can’t boot into your desktop environment try to

  1. Boot directly into a snapshot from the GRUB (boot) menu
  2. see A (above) …

C - In case you need to restore a snapshot from a live system (e. g. EndeavourOS Install medium)

  1. Boot into the live environment
  2. Install Timeshift into the live environment; run
sudo pacman --noconfirm -Syy cronie
yay --noconfirm -S timeshift
 
  1. see A (above) …

D - In case you need to chroot from a live system (e. g. EndeavourOS Install medium)

  1. Boot into the live environment
  2. Decrypt and mount your luks partition (e. g. /dev/sda2)
sudo cryptsetup luksOpen /dev/sda2 crypt &&
sudo mount -o compress=zstd,subvol=@ /dev/mapper/crypt /mnt
 
  1. Mount your EFI boot folder (e. g. /dev/sda1)
sudo mount /dev/sda1 /mnt/boot/efi
  1. chroot into your system
sudo arch-chroot /mnt
  1. Do some root stuff …

[Edit: changed “ls -t” to “ls -1r” in step #06
Because Timeshift regularly changes the modification date of all snapshots, listing by date to find the oldest snapshots didn’t work as expected. So we now list by file name, which contains the snapshot date.]
[Edit 2020-03-20: changed the path “/mnt/timeshift/backup/…” to “/run/timeshift/backup/…” in step #06
It seems that Timeshift mounts its snapshots under /run/timeshift/… . This wasn’t the case on my initial test system but two newer installs point to /run so this seems to be the (new?) timeshift behaviour.]

27 Likes

+++ VERBOSE VERSION ++++++++++++++++++++++++++++++++

If you’re installing to a dedicated hdd/sdd just copy & paste the commands provided above!
The following information is just for the technical interested and/or in the event you have a more complicated setting (e. g. dual boot and/or already have multiple vfat or luks partitions); in which case the provided commands may mess up your existing setup and need to be adjusted according to the info below.


V-01 - Boot into Live-CD environment & set up partitions

Just follow step #01 described above or adjust to your needs.


V-02 - Change default btrfs calamares settings for fstab

There won’t be any problems just following step #02 described above.

By default the Calamares installer only uses btrfs compression (lzo method) if it detects a ssd.
One can change the used compression method(s) in the installed system itself later on, but this would only compress new files or require recompression of all old files. So why not skip these extra steps and let the installer get it right in one go without adding any extra time to the install.

The following will force calamares to assign and use compression on all btrfs installation devices; regardless of being ssd or hdd. This being 2020 with 5.x kernels we’ll use zstd compression.

You can find some information about btrfs compression here.

Edit /usr/share/calamares/modules/fstab.conf by running

sudo leafpad /usr/share/calamares/modules/fstab.conf

and change compress=lzo to compress=zstd in the ssdExtraMountOptions: [...] btrfs: section.

If you want to use compression on a hdd also, just add compress=zstd after autodefrag in the mountOptions: [...] btrfs: section. Save & exit.


V-03 - Install system with calamares (off- or online install)

Just follow step #03 described above.


V-04 - Mount newly installed btrfs system to temporary directory /mnt/btrfs

sudo mkdir /mnt/btrfs

Find your still unlocked luks device which contains the newly installed system.

sudo blkid -o device | grep luks

The command should produce something like /dev/mapper/luks-4e2dc3d4-16ce-44b9-a6be-a4a73a06e854. Your luks-… device UUID will differ though.

If the command returns more than one /dev/mapper/luks-… line you may have to run lsblk and/or sudo blkid and identify the right one.

Now mount the decrypted device to /mnt/btrfs. Be sure to use your luks-… device!

sudo mount -o subvolid=5 /dev/mapper/luks-4e2dc3d4-16ce-44b9-a6be-a4a73a06e854 /mnt/btrfs


V-05 - Adjust/repair fstab for brtfs

If the root file system is btrfs, the fsck order should be set to 0 instead of 1 or higher. The standard fsck has no effect on btrfs partitions and therefore can/should be disabled. Calamares doesn’t set this up right for btrfs partitions and subvolumes so we’ll correct this.

Edit /mnt/btrfs/@/etc/fstab by running

sudo leafpad /mnt/btrfs/@/etc/fstab

Change the last number of all lines that start with your /dev/mapper/luks-… device (the one from above) to 0. Save & exit.


V-06 - Create a top-level subvolume for the pacman cache

Calamares has automatically set up / (root) and /home as seperate top-level subvolumes. This has the advantage that although /home being mounted under /, if we snapshot the root subvolume /, the /home data and all other subvolumes won’t be stored in that snapshot. We can later restore any root snapshot without affecting the /home or other subvolume data.

On Arch Linux, a subvolume for /var/cache/pacman/pkg is often recommended to make sure the large pacman cache, that often contains GB’s of changing but easily redownloadable data, is excluded from snapshots. (Other canditates for exclusion, e. g. /var/abs, /var/tmp, /srv, aren’t part of this tutorial.)
I ran into some trouble during testing when restoring snapshots while setting this up as nested subvolumes, so I decided to stick to top-level subvolumes for this. The downside is we’ll need an entry in fstab for each top-level subvolume.

Create a top-level subvolume named @var-cache-pacman-pkg that will hold all future data stored in /var/cache/pacman/pkg.

sudo btrfs subvolume create /mnt/btrfs/@var-cache-pacman-pkg

As we have a newly installed system, /var/cache/pacman/pkg is still empty, so we don’t need to move any existing data.
We just need to add an entry to /etc/fstab to mount our newly created subvolume to /var/cache/pacman/pkg on the next system start.

  1. Edit /mnt/btrfs/@/etc/fstab with sudo leafpad /mnt/btrfs/@/etc/fstab
  2. Duplicate the line starting with /dev/mapper/luks-... that contains /home

In this newly created duplicate entry

  1. replace /home with /var/cache/pacman/pkg
  2. replace @home with @var-cache-pacman-pkg

Save & exit.

ALTERNATIVELY, you can substitute steps 1. - 4. by running the following commands

fstabstr=`grep "/home" /mnt/btrfs/@/etc/fstab`
fstabstr=${fstabstr//\/home/\/var\/cache\/pacman\/pkg}
fstabstr=${fstabstr//@home/@var-cache-pacman-pkg}
sudo sh -c "echo $fstabstr >> /mnt/btrfs/@/etc/fstab"


V-07 - Optional: Make a swap file in it’s own top-level subvolume & activate hibernation

Note: If you want to use hibernation, then you must add swap because the content of the RAM will be written to the swap partition/file. This also means that the swap size should be at least the size of RAM. Read here about choosing swap sizes.

Swap files in btrfs are supported since Linux kernel 5.x but the swap file cannot be on a snapshotted subvolume. The proper procedure is to create a new subvolume to place the swap file in.

#- Create subvolume named @swap (that will be mounted as /swap in the future)

sudo btrfs subvolume create /mnt/btrfs/@swap

#- Create mountpoint /swap

sudo mkdir /mnt/btrfs/@/swap

#- Create an entry in /etc/fstab to mount the subvolume @swap to /swap on every boot

sudo leafpad /mnt/btrfs/@/etc/fstab

Add the following line …

/dev/mapper/luks-4e2dc3d4-16ce-44b9-a6be-a4a73a06e854 /swap btrfs subvol=@swap,defaults,compress=no 0 0

Make sure to use your luks-UUID! The compress=no makes sure that that there won’t be any btrfs-compression applied to this subvolume holding our swap file. Swap files and btrfs compression don’t do well together!

Save and exit.

#- Create the actual swap file

The following will create and activate a swap file of e. g. 8GB size named swapfile in the subvolume @swap. Read here about general swap file creation. Don’t forget to adjust the size to your needs.

sudo truncate -s 0 /mnt/btrfs/@swap/swapfile
sudo chattr +C /mnt/btrfs/@swap/swapfile
sudo btrfs property set /mnt/btrfs/@swap/swapfile compression none

sudo fallocate -l 8G /mnt/btrfs/@swap/swapfile

sudo chmod 600 /mnt/btrfs/@swap/swapfile
sudo mkswap /mnt/btrfs/@swap/swapfile
sudo swapon /mnt/btrfs/@swap/swapfile

#- Edit /etc/fstab to enable the swap file after reboot

sudo leafpad /mnt/btrfs/@/etc/fstab

Add the following line …

/swap/swapfile none swap defaults 0 0

Save and exit.

#- Optional: Set up Hibernation into swap file on Btrfs

Hibernation onto a swapfile is supported by recent versions of systemd. But at time of writing, since systemd 244, systemctl hibernate does not work with a swap file on Btrfs. The systemd issue 14249 was fixed on 2020-01-08, but there is not any release since then.
See below for an alternative method of invoking hibernation!

We need to let the system know the physical offset of our newly created swap file. This wiki entry describes how to do this for btrfs.

  1. Download and compile a tool to calculate the pysical offset of the swap file …
wget https://raw.githubusercontent.com/osandov/osandov-linux/master/scripts/btrfs_map_physical.c
gcc -O2 -o btrfs_map_physical btrfs_map_physical.c

  1. Now run the tool …

sudo ./btrfs_map_physical /mnt/btrfs/@swap/swapfile

  1. Note the the first physical offset returned by this tool (the last number of the first line containing numbers). In this example, we use 5749440512.

  2. Also note the pagesize (usually 4096) that can be found with

getconf PAGESIZE

  1. To compute the resume_offset value, divide the physical offset by the pagesize. In this example, it is 5749440512 / 4096 = 1403672 .

  2. We need to add the resume_offset to /etc/default/grub

sudo leafpad /mnt/btrfs/@/etc/default/grub

Add resume_offset=1403672 (be surre to use your calculated offset!) after loglevel=3 on the line starting with GRUB_CMDLINE_LINUX_DEFAULT= . Save & exit.

  1. Configure the initramfs
    When an initramfs with the base hook is used, which is the default, the resume hook is required in /etc/mkinitcpio.conf . Whether by label or by UUID, the swap partition is referred to with a udev device node, so the resume hook must go after the udev hook.

sudo leafpad /mnt/btrfs/@/etc/mkinitcpio.conf

Add resume after filesystems on the line starting with HOOKS= . Save & exit.

You will need to let the system know that your grub and initramfs settings have changed. This is covered in the next section.


V-08 - Let the system know that some things have changed & install some needed software

#- chroot into your system

  1. make a temporary folder and mount your root system and efi folder

sudo mkdir /mnt/chroot

Mount the root system to the newly created folder. Use your luks-… device (see V-04).

sudo mount -o compress=zstd,subvol=@ /dev/mapper/luks-4e2dc3d4-16ce-44b9-a6be-a4a73a06e854 /mnt/chroot

Mount the partition containing your EFI data (e. g. /dev/sda1) to /mnt/chroot/boot/efi

sudo mount /dev/sda1 /mnt/chroot/boot/efi

  1. now chroot into the mounted system

sudo arch-chroot /mnt/chroot

#- While we’re in the chroot environment we’ll just install some software we’ll need later on to our newly installed system.

sudo pacman --noconfirm -S cronie
sudo systemctl enable cronie.service

#- Now we’ll update Grub and generate updated kernel images.

sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo mkinitcpio -p linux

V-09 - REBOOT into your new system & run the following …

sudo pacman --noconfirm -Syy &&
yay --noconfirm -S timeshift timeshift-autosnap grub-btrfs

This will install the following:

  • timeshift => A system restore utility for Linux
  • timeshift-autosnap => Timeshift auto-snapshot script which runs before package upgrade using Pacman hook.
  • grub-btrfs => Include btrfs snapshots in GRUB boot options

V-10 - Configure Timeshift (/etc/timeshift.json)

  1. Open Timeshift from the menu
  2. Select “BTRFS” as the “Snapshot Type”; continue with “Next”
  3. Choose your btrfs system partition as “Snapshot Location”; continue with “Next”
  4. “Select Snapshot Levels” (type and number of snaphots that will be automatically created and managed/deleted by Timeshift)
    Recommendation:
  • Keep “Daily” at 5
  • Activate “Boot”, but change to 3; continue with “Finish”
  1. Optional: “Create” a manual first snapshot & exit Timeshift

Timeshift will now check on every full hour if snapshots (hourly, daily, weekly, monthly) need to be created or deleted. Note that boot snapshots will actually be created about 10 minutes after boot, not directly at system startup!


V-11 - Configure timeshift-autosnap (/etc/timeshift-autosnap.conf)

Timeshift-autosnap creates timeshift snapshots whenever it detects a pacman (and yay invoked) upgrade. This way you should be able to rollback to a state right before the package upgrades.

By default only the last 3 snapshots are kept. I recommend leaving it at that value.


V-12 - Configure grub-btrfs (/etc/default/grub-btrfs/config)

Grub-btrfs creates Grub boot entries for your snapshots. I recommend leaving the configuration as is.

Grub-btrfs should be able to monitor the snapshot folder for changes and automatically update grub if it detects such a change. I wasn’t able to get this functionality to work reliably and therefore decided to just update the grub boot entries on every system shutdown (see V-13).


V-13 - Limit the maximum amount of snapshots & update grub boot entries

Timeshift cleans the automatically created snapshots and timeshift-autosnap has a default limit of 3 snapshots after upgrades, BUT the ones you create manually aren’t handled by Timeshift and would be kept indefinetily. High amounts of old snapshots could potentially cause slowdowns, so we’re going to limit the amount of maximum snapshots your system will allow and keep.

For this we will create a systemd service that runs right before you shutdown or restart the system and

  • keeps only the newest (e. g. 13) snapshots and deletes all the older ones
  • updates grub to contain only these snapshots as boot entries

I recommend to limit the amount of snapshots to about 13 to 15.

Create a systemd service called “limit-timeshift-snapshots-update-grub”:

sudo leafpad /etc/systemd/system/limit-timeshift-snapshots-update-grub.service

Now add the following content (you can adjust the number of snapshots to keep by changing the number from 13 to whatever in the ExecStop=... line; just choose a high number if you prefer to delete these manually):

[Unit]
Description=Keep only latest 13 timeshift snapshots and update grub.cfg before shutdown
Before=poweroff.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecStop=/usr/bin/sh -c "cd /run/timeshift/backup/timeshift-btrfs/snapshots/ && rm -rf `ls -1r | awk 'NR>13'` && /usr/bin/grub-mkconfig -o /boot/grub/grub.cfg"


[Install]
WantedBy=multi-user.target

Save & exit.

Next we’ll enable and start the newly created service.

sudo systemctl enable limit-timeshift-snapshots-update-grub.service &&
sudo systemctl start limit-timeshift-snapshots-update-grub.service

V-14 - Activate/Invoke Hibernation

The normal way of invoking hibernation would be to run/call systemctl hibernate . But, as stated above, there seems to be a bug in the current systemd that causes this to fail.

e. g. Failed to hibernate system via logind: Not enough swap space for hibernation

Until this is fixed, here is a workaround; activate hibernation by running:

sudo sh -c 'echo disk > /sys/power/state'


V-15 - Restore a snapshot (rollback the system)

See the # - How to RESTORE a snapshot (previous system state) section in the previous post.


8 Likes

@2000
Thanks! This is something that would be great to have as a wiki article on EndeavourOS.
What do you think about it?

7 Likes

Sure, if you think you have use for it just go ahead. Didn’t post it to keep it to myself :wink: . Thanks

5 Likes

Again, thank you for your great tutorial for the second time. Like your other manual, I will turn it into a wiki article later this week.

5 Likes

Hi @2000,

since there is no separate boot partition there is the limitation that the password can only be typed in once before falling to grub shell when the entered password is incorrect, am I right?

Thanks in advance,

Christoph

I’ve read about a patch handling this (forgot where), but out of the box, sadly you’re correct. There is a way to open a LUKS volume from grub rescue mode and continue booting without rebooting, though (see here for example). But I personally don’t bother with that and just do a quick reboot if I mistyped.

Hmmm…to bad. I already followed your great tutorial here, so would it be possible to add an unencrypted boot partition (ext4) to the above setup? My fingers are itching to try this out…

Regards,

Christoph

Possible, yes - but I wouldn’t recommend it; you’d lose the whole easy rollback feature.

The way the system is set up above insures that you can rollback the complete system to a stored state (kernel and all). Having a separate boot partition thwarts this strategy.

A snapshot is a special type of subvolume. It is a copy of a subvolume, without the parent or child subvolumes and without mounted external data. The snapshot does not make copies of files, but shares the data and metadata of the original subvolume, so it’s space-efficient and extremely fast to create.

/boot needs to be a part of the root subvolume, not just mounted under it, for the stored kernels and initrd/initramfs that reside in /boot to be included in a root snapshot.
If they weren’t included in the snapshot you could potentially rollback the system to a state incompatible with the newer untouched kernel; this could render your system unbootable and you’d have to manually downgrade/rebuild the kernel.

2 Likes

Now I understand! I tried this a couple of months ago, and I wondered why everything was borked when I rolled back to a snapshot.

Considering a multiboot system (EOS daily driver, Manjaro somehow secondary system, WIndows also there) I would install Manjaro with a separate boot partition, put the Manjaro grub/efi entry in charge so I can reach Windows without a password. With a custom entry for EOS like this

menuentry "EOS" --class arch {
        insmod part_gpt
        insmod chain
        set root='(hd0,gpt2)'
        chainloader /EFI/EOS/grubx64.efi
}

so I can switch to the fully encrypted setup. This may be a feasible setup, do you think?

Regards,

Christoph

That’s the way I would try to set it up too. If you don’t want to have to unlock your system everytime just to boot into Windows, you will need an unencrypted /boot partition. Either EOS’s or that of a 3rd system.

I must confess, the only multiboot system I have is a LVMonLUKS Arch install with an unencrypted boot partition and Windows. So no problem there.
All my other computers now run a single Linux system with an encrypted /boot. I really dig the btrfs snapshot rollback feature and don’t want to miss it anymore (I also use lvm-snapshots on my ext4 setup, but the restoring/rollback process takes more time and in my case always requires it being done from a live environment; it also has the described potential kernel rollback problem).
I guess my next project will be thinking about how to multiboot “efficiently” without sacrificing an encrypted /boot and the btrfs snapshot feature.

Please report back, if you happen to set this up properly.

Hi,

tank you for your tutorial.
I have got an error on Step 5:

[root@archiso liveuser]# btrfsonluks=sudo blkid -o device | grep luks
[root@archiso liveuser]# sudo mount -o subvolid=5 $btrfsonluks /mnt/btrfs
mount: /mnt/btrfs: can’t find in /etc/fstab.
[root@archiso liveuser]#

Installation just finished before, did everything like you said. Any advice?

Thank you!

Edit: Tried again. Reason is: I am Stuck on Antergos and have a similar laptop around, just for testing purposes. So I followed your tutorial again. After partitioning with gparted I run calamares and as soon as is change the filesystem for the unallocated space to “btrfs” and enable “Encryption” it switches to LUKS. Even if I set it to btrfs again afterwards the encryption is still missing…So I am stuck at this part actually :wink:

Oh, I see the “problem”.

You actually did everything right by setting the following for the unnallocated space …

  • Mount point → /
  • File system → btrfs
  • Encrypt: choose a secure password

but were irritated after hitting “OK”.

It’s wanted behaviour for the new partition’s file system to change to LUKS! Just ignore this and continue with “Next”. Then you’ll have a luks-partition after install that the command you posted can actually find.

1 Like

OK, will try this and report back to you.
Thank you!

Hi!

Worked like a charm - thank you.
I am totally new into btrfs and endeavour os, like I said before. I am not that linux-geek type, but I’ll try my best. For what I understand, btrfs does NOT copy any files for a snapshot!? I understand, that it copies the data and metadata - but at some point there has to be a real copy of an existing file, right? If there is a textfile in the snapshot and I change this textfile, will btrfs do a complete copy of the changed file?

I set the Maximum-Snapshots amount at 13, just for testing. A few questions regarding timeshift:

  • Timeshift lists jobs for daily and “on boot” snapshots, and will keep 5 (daily) and 3 (on boot). I can change this value to anything, but it will not exceed 13, right?

  • Snapshots will include anything, even my home?

  • I read on another article, that you could set to do a snapshot every 10 minutes so if you accidentally delete a file or something you could just roll back immediately. I cant find an option for that in time-shift

  • Your desktop icons are not working on gnome (understandable). I will figure out a way to add them into gnome - any suggestions? would be nice to have an hibernate button in the menu.

  • is there any chance to guess how much space a snapshot is blocking? Just for curiosity…

Thank you for your great work, I really (really really!) appreciate it!

Yes, a snapshot in Btrfs is a special type of subvolume - one which seems to contain a copy of the current state of some other subvolume. But, of course, Btrfs is a copy-on-write filesystem, so there is no need to actually copy all of that data; the snapshot simply has a reference to the current root of the filesystem. If anything is changed - in either the main volume or the snapshot - a copy of the relevant data will then be made, so the other copy will remain unchanged. So basically the snapshot has no size at all unless you actually start changing files.

Yes, on shutdown/restart all snapshots exceeding the chosen amount will be deleted.

BTW, I made a small change to the command that gets called on shutdown; could you please check and make sure you don’t still have the old value:

  1. sudo leafpad /etc/systemd/system/limit-timeshift-snapshots-update-grub.service
  2. Please make sure you have […] ls -1r | awk […] and not the old code […] ls -t | awk […]. If not, change it; save, exit and then run …
  3. systemctl daemon-reload
  4. systemctl restart limit-timeshift-snapshots-update-grub.service

NO! This isn’t the default behavior.

You can configure Timeshift to include your /home subvolume whenever a snapshot is created, though. On restoring, Timeshift will ask if you also want to restore your /home subvolume. Afaik the Timeshift GUI allows to restore /home AND root OR just root, but not just /home. So this may not be what you’re looking for.

Depending on the amount of changing data, a lot of snapshots can/will cause serious system slowdowns. Btrfs does have scaling issues due to too many snapshots and single to low double-digits of snapshots per snapshotted subvolume remains the strong recommendation for that reason.
You could write a small script that creates a snapshot every 10m (see timeshift --help ) but I would advise to only keep a handful of those snapshots and regularly delete old ones.

Gosh, actually not that easy; standard system tools installed don’t get the job done.
You could try something like …

The reported sizes would probably be the uncompressed sizes - but don’t forget that the system uses zstd compression. To see the actual effective compression ratio (the actual used space) you could try out a tool like

@2000 I got a message on step #6 stating something about missing quota. Did I mess something up here?

I know nothing about BTRFS but maybe you are looking for this filesystem usage?
(Include /@home)

~ >>> sudo btrfs filesystem usage /                                                                                 
[sudo] Passwort für sgs: 
Overall:
    Device size:                 111.79GiB
    Device allocated:             55.02GiB
    Device unallocated:           56.77GiB
    Device missing:                  0.00B
    Used:                         33.20GiB
    Free (estimated):             76.38GiB      (min: 76.38GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               94.97MiB      (used: 0.00B)

Data,single: Size:52.01GiB, Used:32.40GiB (62.30%)
   /dev/sdb1      52.01GiB

Metadata,single: Size:3.01GiB, Used:816.22MiB (26.50%)
   /dev/sdb1       3.01GiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/sdb1       4.00MiB

Unallocated:
   /dev/sdb1      56.77GiB
~ >>> sudo timeshift --list
/dev/sdb1 is mounted at: /run/timeshift/backup, options: rw,relatime,compress=zstd:3,ssd_spread,space_cache,subvolid=5,subvol=/

Device : /dev/sdb1
UUID   : fe060551-6a03-4d1a-9f8b-996ff8288fe3
Path   : /run/timeshift/backup
Mode   : BTRFS
Status : OK
3 snapshots, 84.3 GB free

Num     Name                 Tags  Description                           
------------------------------------------------------------------------------
0    >  2020-03-04_17-52-05  O     {TS-autosnap created before upgrade}  
1    >  2020-03-04_23-29-02  O     {TS-autosnap created before upgrade}  
2    >  2020-03-05_12-43-21  O     {TS-autosnap created before upgrade} 

Difference after “big” update :wink:

Gesamtgröße des Downloads:           424,53 MiB
Gesamtgröße der installierten Pakete:  942,80 MiB
Größendifferenz der Aktualisierung:    0,32 MiB
~ >>> sudo btrfs filesystem usage /                                                                                 
[sudo] Passwort für sgs: 
Overall:
    Device size:                 111.79GiB
    Device allocated:             55.02GiB
    Device unallocated:           56.77GiB
    Device missing:                  0.00B
    Used:                         34.69GiB
    Free (estimated):             74.98GiB      (min: 74.98GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              100.67MiB      (used: 0.00B)

Data,single: Size:52.01GiB, Used:33.80GiB (64.98%)
   /dev/sdb1      52.01GiB

Metadata,single: Size:3.01GiB, Used:916.33MiB (29.75%)
   /dev/sdb1       3.01GiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/sdb1       4.00MiB

Unallocated:
   /dev/sdb1      56.77GiB
~ >>> sudo timeshift --list                                                                                         

/dev/sdb1 is mounted at: /run/timeshift/backup, options: rw,relatime,compress=zstd:3,ssd_spread,space_cache,subvolid=5,subvol=/

Device : /dev/sdb1
UUID   : fe060551-6a03-4d1a-9f8b-996ff8288fe3
Path   : /run/timeshift/backup
Mode   : BTRFS
Status : OK
4 snapshots, 82.7 GB free

Num     Name                 Tags  Description                           
------------------------------------------------------------------------------
0    >  2020-03-04_17-52-05  O     {TS-autosnap created before upgrade}  
1    >  2020-03-04_23-29-02  O     {TS-autosnap created before upgrade}  
2    >  2020-03-05_12-43-21  O     {TS-autosnap created before upgrade}  
3    >  2020-03-06_00-22-06  O     {TS-autosnap created before upgrade}  


1 Like

Here is the exact message:

Tagged snapshot '2020-03-06_06-45-29': ondemand
------------------------------------------------------------------------------
E: ERROR: can't list qgroups: quotas not enabled

E: btrfs returned an error: 256
E: Failed to query subvolume quota
Enabled subvolume quota support

It’s just Timeshift telling you it has checked and NOW enabled quota. All good!