HOWTO - GPT/UEFI install with full disk encryption: BTRFSonLUKS with separate root, home and pkg subvolumes; hibernation with a swapfile; auto-snapshots with easy system rollback (GUI); boot into snapshots

@2000: Referring on your post: Can the verbose setup be used for a dual boot with Windows 10 Pro where secure boot and bitlocker is disabled?
Or should I use a modified installation with a unencrypted /boot-partitition with LVMonLUKS for the rest of the installation?

Shouldn’t be a problem. See here and here.

You’d end up with rEFInd as the boot-loader which let’s you choose between Windows or your fully encrypted EOS. This way you would only have to enter your luks password once if you really want to load EOS. Tested this myself with Win8 and EOS.

1 Like

Thank you for the links, which I did not found.
I have to read them and then decide how to procede.

Thank you for this comprehensive reply, which is a guide in itself! :pray: :slight_smile: I wanted to try what you have suggested before reporting back.

I̶ h̶a̶v̶e̶ ̶j̶u̶s̶t̶ ̶t̶e̶s̶t̶e̶d̶ ̶d̶e̶l̶e̶t̶i̶n̶g̶ ̶t̶h̶e̶ ̶m̶o̶s̶t̶ ̶r̶e̶c̶e̶n̶t̶ ̶s̶n̶a̶p̶s̶h̶o̶t̶ ̶a̶n̶d̶ ̶t̶h̶e̶r̶e̶ ̶w̶a̶s̶ ̶n̶o̶ ̶f̶r̶e̶e̶z̶i̶n̶g̶ but literally as I typed this perhaps 45 seconds later one occurred (a bad one where everything was frozen bar the cursor and nothing responded. Mostly I will be in Brave (but not exclusively just in Brave) and when I try to see how the rest of the system is responding by opening Nautilus it will open but it will not display any files like it cannot load them). How can I check if they occur when snapshots are deleted automatically please, perhaps we could setup some kind of notification on autodelete and then it would be clear?

Okay so I have now tried several days with @home and @ set to no commit argument, commit=90, and now I am on commit=60 but unfortunately the issue is ongoing. Would it be worth breaking this down further, perhaps at 15 increments from 45, 75, 105 or do my tests suggest we look elsewhere?

This makes sense, and low disk space was the problem that brought me initially back to the thread but at the time of my last post when I had already freed up a lot of space, I think ~60GiB, now 43GiB. Also, the same problem, albeit less frequently, with the same setup on my Dell XPS 9360 although this has much more free space and the disk capacity ~500GiB is twice what my desktop has. I say less frequently but I do not use my laptop as much as desktop, I use it as a secondary machine next to my desktop. I also just expect it to perform better as it has an m.2 SSD.

Low disk space →

  1. General linux userspace tools such as df will inaccurately report free space on a Btrfs partition. It is recommended to use btrfs filesystem usage to query Btrfs partitions. For example:
    btrfs filesystem usage /

Desktop:

Overall:
    Device size:		 232.38GiB
    Device allocated:		 216.37GiB
    Device unallocated:		  16.01GiB
    Device missing:		     0.00B
    Used:			 190.52GiB
    Free (estimated):		  40.35GiB	(min: 40.35GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 338.55MiB	(used: 0.00B)

Data,single: Size:213.36GiB, Used:189.02GiB (88.59%)
   /dev/mapper/cryptdata	 213.36GiB

Metadata,single: Size:3.01GiB, Used:1.50GiB (50.02%)
   /dev/mapper/cryptdata	   3.01GiB

System,single: Size:4.00MiB, Used:48.00KiB (1.17%)
   /dev/mapper/cryptdata	   4.00MiB

Unallocated:
   /dev/mapper/cryptdata	  16.01GiB

Laptop:

Overall:
    Device size:		 476.44GiB
    Device allocated:		 324.01GiB
    Device unallocated:		 152.42GiB
    Device missing:		     0.00B
    Used:			 282.01GiB
    Free (estimated):		 192.46GiB	(min: 192.46GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 402.86MiB	(used: 0.00B)

Data,single: Size:320.00GiB, Used:279.96GiB (87.49%)
   /dev/mapper/cryptdata	 320.00GiB

Metadata,single: Size:4.01GiB, Used:2.05GiB (51.19%)
   /dev/mapper/cryptdata	   4.01GiB

System,single: Size:4.00MiB, Used:64.00KiB (1.56%)
   /dev/mapper/cryptdata	   4.00MiB

Unallocated:
   /dev/mapper/cryptdata	 152.42GiB

I have run this twice since you posted, a cronjob might be nice for this.

I have not run this but I will keep it in mind if I completely run out of space again.

I should mention that your setup is not the only encryption I employ. I have free unlimited Google Drive storage and I have used Insync with this for a few years before my most recent install. This install I updated to their latest version 3.x when before I think I was on 1.5.x or 2.x, but in addition to this I have setup Cryptomator for my Google Drive which contains most of my personal files, code and documents etc are automounted and minimised on start with a kdocker command, Insync then syncs to the unlocked vault. Insync syncs to a directory in ~ and cryptomator mounts the decrypted vault to another in ~, they are 68GiB each. I cannot say the system is more likely to freeze during the initial Insync sync on login or as it periodically updates, unless something in the background is causing it. I do not store config files that normally are located in the home directory there - do you think this could be an issue?

I also use Dejadup for additional encrypted daily autobackups of /home to Google Drive, could that cause some conflict?

I am very happy with my setup and I would hate to lose any part of it, BTRSFSonLUKS, or Cryptomator working with Insync. I have ordered a SB-ROCKET-2TB m.2 2280 SSD which I will install my desktop on fresh on over the holidays, and I expect to see less freezing but like I said unfortunately the same issue occurs on my XPS 13 9360 which has an m.2. It would after I set everything up on the new m.2 to put Windows on the current SATA SSD for emergencies, you have mentioned how this is possible with rEFInd and I see it is an active topic in the thread :slight_smile:

Is there any other information I can provide or tests that I can run?

There always is … :wink: .

I don’t know where anymore but I remember reading about reports of significant slow downs being caused by quota groups.
These are enabled by default by Timeshift but fortunately timeshift has a setting to disable them.

The downsides I have noticed when disabling quotas (so far) are:

  • Timeshift won’t be able to show shared and unshared snapshot sizes in the main window.
  • If you run timeshift from cli you will see a lot of warnings and errors about quotas not being enabled.
  • If you (or hooks like timeshot-autosnap etc.) try to delete a snapshot, Timeshift will report it couldn’t delete it. The snapshot will only be successfully deleted on the second try.

If you can live with this, try the following and check if you still experience slowdowns:

  1. Disable (uncheck) quota groups in Timeshifts settings
    Timeshift > Settings > Users > Enable BTRFS qgroups

  2. -and- disable quota groups in general
    sudo btrfs quota disable /

If you ever want the old behaviour back, just enable the qgroups option in Timeshift and quota groups will be reenabled.

1 Like

@2000: Sorry for the - maybe - dumb question:
I do the install of BTRFSonLUKS as written in the wiki tutorial. Afterwards I install rEFInd and change in the UEFI-BIOS the boot standard to rEFInd, right?

It usually suffices to (1) install and then (2) run rEFInd’s install script.

You shouldn’t have to adjust the bios settings, but you can of course. If you need to adjust the boot sequence efibootmgr is also an option.
Maybe also check @freebird54’s Wiki article.

1 Like

@marteng69
I do have triple boot setup using @2000 wiki tutorial and i am using rEFInd. The instructions given are correct and if you have any issue i may be able to guide you if the wiki doesn’t. It took me a while to understand how to use rEFInd and i won’t say i understand everything that it is capable of but, i do use it with a unique setup that has both encrypted and unencrypted multiboot Btrfs.

2 Likes

@2000 You have fixed the issues with freezing, thank you for that but the problems with space being used up is still occurring, and it happened right before an exam during the week so I just deleted some snapshots from the Timeshift GUI - I did not risk running the command you said would take a long time, because I am unsure how long it is going to take.

That allowed me to log back in for a few days and I thought that it would be enough to get me through until my new m.2 2TB arrives, which has been delayed in transit, but today after I installed the Discord flatpak and rebooted after entering my passprase and selecting Ubuntu from GRUB the Gigabyte logo would just stay on the screen.

From Ubuntu recovery I tried the usual autoremove, autoclean etc, but it reported issues with network access, despite enabling it. I then noticed it was giving me some no space left on device errors, so I manually removed some video files from my Downloads folder, which I thought would be enough but it has not been.

There have been many reboots into recovery and I notice now that I have errors about a read-only file system. I have tried fixing this with: mount -o remount,rw / which I think at one point was working but now outputs ‘mount: /: mount point not mounted or bad option.’

Please find screenshots of some commands I have tried and information that may help identify the issue:

noboot1 noboot2 noboot3

I have tried restoring the various remaining snapshots since after I deleted but error with:

noboot4

I am hoping that when I set my system up like this again on the 2TB drive this will mean this problem will not recur as it never happens on my XPS 13 which has twice the drive capacity.

Can you please tell me if there is something I can do to get this running, and perhaps disable Timeshift until I can reinstall on a larger drive?

It say’s wrong file system type on swap?

I am not sure why exactly, the only changes I made to fstab were to the commit argument some weeks ago. Despite this, please find attached new screenshots which include the fstab, and for context the lines around where the btrfs, swap errors, and other errors start, and some other things I have tried. It seems whatever I try it results in errors either about no space left on device or a read only file system, and when I try to fix one of those the other prevents it.

nobootA nobootB nobootC nobootD

I have chrooted in from a live USB and encounter the same issue, the btrfs fs immediately goes ro, even when I explicitly specify rw in the mount comand.

The closest I have come to solving this is mounting with:

sudo mount -o rw,skip_balance /dev/mapper/cryptdata /mnt

I was then able to remove 6 video files of ~ 1.5GiB each just using rm -rf (this would fail on first attempt, succeed on second attempt every time) and in Nautilus when I check properties of the home directory with a right click it still shows ‘free space 0 bytes’.

After using umount -f /mnt I have also tried: btrfs check /mnt and again with the --repair flag and read that for free space cache and super generation do not match. I then did it again with a --clear-space-cache v1 and ran both the check and check with --repair again (as I read it is important to do them both). After the error is no longer reported.

On reboot I get the same error for both types of balancing, the one liner and the other with the for loop.

@2000 I have one question: If I want to disable btrfs compression altogether, how should I do that? Will removing compress=lzo from /usr/share/calamares/modules/fstab.conf do the trick? I will be installing on an SSD.
Thank you! :pray:

Maybe better to change to
compress=no
in the conf file.

1 Like

@baremetaloctopus
Please copy (Strg+Shift+c) the terminal output of the commands listed below and paste (Strg+v) the information between “collapsing text blocks” like the following:

cat /proc/cmdline

… pasted info …

[details="cat /proc/cmdline"]
... pasted info ...
[/details]
  1. cat /proc/cmdline
  2. findmnt -D -t btrfs,vfat,ext4
  3. sudo btrfs sub list /
  4. sudo btrfs filesystem usage /
  5. sudo timeshift --list
  6. cat /etc/fstab | grep -v "#"
  7. whatever else you may find useful :wink:
1 Like

So I have learned that strg is ctrl in Germany, danke for the reply Herr! :slightly_smiling_face:

As I can only directly access the system from a recovery root shell that will not allow me bring up networking, I can only copy and paste the input and output using the live USB with chroot, so after doing so I also carefully manually transcribed the output for cat /proc/cmdline regular Ubuntu recovery - everything else is from chroot.

cat /proc/cmdline

BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity quiet splash nomodeset —

cat /proc/cmdline

BOOT_IMAGE=/@/boot/vmlinuz-5.4.0-58-generic root=UUID=1b793f77-ed39-405a-ba74-91d898adf313 ro recovery nomodeset dis_ucode_ldr rootflags=subvol=@

findmnt -D -t btrfs,vfat,ext4

SOURCE FSTYPE SIZE USED AVAIL USE% TARGET
/dev/mapper/cryptdata[/@] btrfs 232.4G 203.1G 0 87% /
/dev/sda1 vfat 598.7M 56.4M 542.3M 9% /boot/efi

sudo btrfs sub list /

ID 269 gen 359810 top level 5 path @swap
ID 493 gen 361262 top level 5 path timeshift-btrfs/snapshots/2020-09-19_20-34-57/@
ID 494 gen 360144 top level 5 path timeshift-btrfs/snapshots/2020-09-19_20-34-57/@home
ID 960 gen 361261 top level 5 path @home
ID 961 gen 361523 top level 5 path @
ID 1969 gen 360515 top level 5 path timeshift-btrfs/snapshots/2020-12-10_15-10-40/@
ID 1970 gen 359813 top level 5 path timeshift-btrfs/snapshots/2020-12-10_15-10-40/@home
ID 1971 gen 359813 top level 5 path timeshift-btrfs/snapshots/2020-12-10_17-00-01/@
ID 1972 gen 359814 top level 5 path timeshift-btrfs/snapshots/2020-12-10_17-00-01/@home
ID 1985 gen 361262 top level 5 path timeshift-btrfs/snapshots/2020-12-11_16-44-59/@
ID 1986 gen 359814 top level 5 path timeshift-btrfs/snapshots/2020-12-11_16-44-59/@home
ID 2001 gen 361264 top level 5 path timeshift-btrfs/snapshots/2020-12-12_18-00-01/@
ID 2002 gen 359814 top level 5 path timeshift-btrfs/snapshots/2020-12-12_18-00-01/@home
ID 2003 gen 359814 top level 5 path timeshift-btrfs/snapshots/2020-12-13_00-17-47/@
ID 2004 gen 359814 top level 5 path timeshift-btrfs/snapshots/2020-12-13_00-17-47/@home
ID 2005 gen 360142 top level 5 path timeshift-btrfs/snapshots/2020-12-13_11-59-53/@
ID 2006 gen 359815 top level 5 path timeshift-btrfs/snapshots/2020-12-13_11-59-53/@home
ID 2011 gen 359816 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-43/@
ID 2013 gen 359816 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-46/@
ID 2014 gen 359817 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-46/@home
ID 2015 gen 359818 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-53/@
ID 2016 gen 360137 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-53/@home
ID 2017 gen 361266 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-57/@
ID 2018 gen 359827 top level 5 path timeshift-btrfs/snapshots/2020-12-13_12-45-57/@home
ID 2019 gen 360106 top level 5 path timeshift-btrfs/snapshots/2020-12-13_15-59-15/@
ID 2020 gen 360134 top level 5 path timeshift-btrfs/snapshots/2020-12-13_15-59-15/@home

sudo btrfs filesystem usage /

Overall:
Device size: 232.38GiB
Device allocated: 232.38GiB
Device unallocated: 1.00MiB
Device missing: 0.00B
Used: 202.75GiB
Free (estimated): 29.28GiB (min: 29.28GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 357.97MiB (used: 160.00KiB)

Data,single: Size:229.36GiB, Used:200.08GiB (87.23%)
/dev/mapper/cryptdata 229.36GiB

Metadata,single: Size:3.02GiB, Used:2.67GiB (88.42%)
/dev/mapper/cryptdata 3.02GiB

System,single: Size:4.00MiB, Used:48.00KiB (1.17%)
/dev/mapper/cryptdata 4.00MiB

Unallocated:
/dev/mapper/cryptdata 1.00MiB

sudo timeshift --list

E: Error opening file “/var/log/timeshift/2020-12-14_12-59-34_list-snapshots.log”: Read-only file system
E: Error creating directory /root/.temp/FjI8AixN: Read-only file system
E: Failed to create directory: /root/.temp/FjI8AixN/2020-12-14_12-59-34
E: Error creating directory /root/.temp/FjI8AixN: Read-only file system
E: Failed to create directory: /root/.temp/FjI8AixN/2020-12-14_12-59-34
E: Failed to mount device ‘/dev/dm-0’ at mount point ‘/run/timeshift/backup’
E: mount: /run/timeshift/backup: wrong fs type, bad option, bad superblock on /dev/mapper/cryptdata, missing codepage or helper program, or other error.

** (process:7997): CRITICAL **: 12:59:34.730: tee_jee_file_system_path_combine: assertion ‘path1 != NULL’ failed

** (process:7997): CRITICAL **: 12:59:34.730: tee_jee_file_system_dir_exists: assertion ‘dir_path != NULL’ failed

** (process:7997): CRITICAL **: 12:59:34.730: tee_jee_file_system_path_combine: assertion ‘path1 != NULL’ failed

** (process:7997): CRITICAL **: 12:59:34.730: tee_jee_file_system_dir_exists: assertion ‘dir_path != NULL’ failed
Device : /dev/dm-0 (sda2)
UUID : 1b793f77-ed39-405a-ba74-91d898adf313
Path :
Mode : BTRFS
Status : Selected snapshot device is not a system disk
Select BTRFS system disk with root subvolume (@)

No snapshots found

cat /etc/fstab | grep -v "#"

root@ubuntu:/# cat /etc/fstab | grep -v “#”
/dev/mapper/cryptdata / btrfs defaults,subvol=@,ssd,noatime,space_cache,commit=60,compress=zstd 0 0
UUID=E625-C979 /boot/efi vfat umask=0077 0 1
/dev/mapper/cryptdata /home btrfs defaults,subvol=@home,ssd,noatime,space_cache,commit=60,compress=zstd 0 0
UUID=1b793f77-ed39-405a-ba74-91d898adf313 /swap btrfs subvol=@swap,compress=no 0 0
/swap/swapfile none swap defaults 0 0
root@ubuntu:/#

For reference, the full terminal input and output after booting the live USB: https://pastebin.ubuntu.com/p/7GNMqzYmrz/

My new 2TB m.2 SSD appears to be lost somewhere between England and Ireland, and with Christmas, Brexit, and Covid-19, it is likely going to be some time until I can get a new one for the same value (New Sabrent ROCKET 2TB NVMe PCIe M.2 2280 Internal SSD @ €180), so perhaps if we can fix this we should also either disable timeshift completely or reduce the number of snapshots taken, or try something else to stop this recurring?

OK, your device is filled to the brim. If a btrfs device runs out of space and there’s nowhere to write new metadata, it will switch to read-only.

You need to re-balance the btrfs filesystem. The problem is - balancing requires some free disk space :frowning_face: .

You could try two things:

  1. If you happen to be able to mount the filesystem writable, you could try to gradually free up some chunks by running something like this …
    sudo sh -c 'for i in 0 1 2 3 5 10 15 20 25 30 40 50 60 70 80 90 100; do echo "${0}: Running with ${i}%"; sudo btrfs balance start -dusage=$i -musage=$i /; done'

If this fails and the device switches to read-only you simply don’t have enough free space available. You will then have to …

  1. add a new temporary disk (device) to the pool (btrfs filesystem), run the balance command and finally remove the temporary device.
    Read here on how to add/remove devices to a mounted filesystem.
1 Like

I was able to mount it writable last night - kind of, it was very strange, every video file I deleted took two attempts and made no difference to the free space, and unless that command you just posted is different than the original one you posted to me before - I already tried it and it errored on every iteration of the loop. Last night I tried doing it - but it was 4am and I cannot remember if that was from the live USB with chroot or a normal boot with recovery, but I think I typed it which would mean recovery. I can try whatever steps I posted I tired here last night when I did this and see if it will work from either recovery or chroot - or is it just necessary to try one of those? I think there may be an issue with doing it from recovery because at that point the filesystem is already mounted and I am logged into it - whereas with a live USB and chroot I can remount forcing the option ‘rw,skip_balance’ which make it writable and prevent auto-balancing from immediately setting the disk RO on mount.

Failing that, I can try to add a disk to the btrfs filesystem. Last night I tried adding a ramdisk, which is supposed to work much faster for this, I think from the the live USB, as described at the very end of this article under ‘Update’ https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/. I can try it again, do you think if I just use those commands - edited for my drive etc - under the update section from it should work for me?

If I can use either chroot live USB or normal boot with recovery I would prefer chroot as I can copy and paste commands and the output in and out of terminal?

Your filesystem doesn’t even have enough free space left to register the deletion.
You could try to delete the snapshots still residing on your system and then try to run the balance command again. If the following still happens …

… you have no choice but to add some external storage to the filesystem. The link you provided is excellent (thanks for that)!

It shouldn’t be a problem doing this from a live environment.


Once you’re up and running again, don’t forget to regularly balance the filesystem. This is even more important the less space you have to begin with. I run the following every three days:
btrfs balance start -dusage=50 -dlimit=2 -musage=50 -mlimit=4 /

1 Like

I have been able to log in to my desktop but both reblance commands output errors, when run from the live USB even with / mounted rw, and now on my desktop after several reboots and deleting all snap shots to make space. The steps I I followed are the output is outlined below

sudo cryptsetup luksOpen /dev/sda2 cryptdata
sudo mount -o subvol=@,ssd,rw,skip_balance,noatime,space_cache,commit=120,compress=zstd /dev/mapper/cryptdata /mnt
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
sudo mount /dev/sda1 /mnt/boot/efi
sudo cp /etc/resolv.conf /mnt/etc/
sudo chroot /mnt
cat /proc/mounts

/dev/mapper/cryptdata / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,skip_balance,commit=120,subvolid=961,subvol=/@ 0 0
udev /dev devtmpfs rw,nosuid,noexec,relatime,size=8079640k,nr_inodes=2019910,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=1625596k,mode=755 0 0
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

mount -av

/ : ignored
/boot/efi : already mounted
/home : successfully mounted
/swap : successfully mounted
none : ignored

I noticed that Nautlis was showing 8GiB free and I was able to delete some files first time.

timeshift --list

E: Error opening file “/var/log/timeshift/2020-12-14_22-46-27_list-snapshots.log”: No space left on device
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
Mounted ‘/dev/dm-0 (sda2)’ at ‘/run/timeshift/backup’
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: Error creating directory /root/.temp/ppeZcwm0: No space left on device
E: Failed to create directory: /root/.temp/ppeZcwm0/2020-12-14_22-46-27
E: ERROR: can’t list qgroups: quotas not enabled

E: btrfs returned an error: 256
E: Failed to query subvolume quota
Device : /dev/dm-0 (sda2)
UUID : 1b793f77-ed39-405a-ba74-91d898adf313
Path : /run/timeshift/backup
Mode : BTRFS
Status : OK
14 snapshots, 31.4 GB free

Num Name Tags Description

0 > 2020-09-19_20-34-57 O BEFORE KODI RESTORE
1 > 2020-12-10_15-10-40 D M
2 > 2020-12-10_17-00-01 W
3 > 2020-12-11_16-44-59 D
4 > 2020-12-12_14-44-52
5 > 2020-12-12_18-00-01 D
6 > 2020-12-13_00-17-47 B
7 > 2020-12-13_11-59-53 B
8 > 2020-12-13_12-45-31 O {timeshift-autosnap-apt} {created before call to APT}
9 > 2020-12-13_12-45-43 O {timeshift-autosnap-apt} {created before call to APT}
10 > 2020-12-13_12-45-46 O {timeshift-autosnap-apt} {created before call to APT}
11 > 2020-12-13_12-45-53 O {timeshift-autosnap-apt} {created before call to APT}
12 > 2020-12-13_12-45-57 O {timeshift-autosnap-apt} {created before call to APT}
13 > 2020-12-13_15-59-15 B

When deleting the snapshots sometimes it took several attempts or there was a delay between when I would input the command and they would disappear from the --list. Also, despite going through all 13, when I was able to reboot into my normal desktop I saw in the Timeshift GTK many of the snapshots still remnaining. Sample output when trying to delete:

timeshift --delete --snapshot '2020-12-13_15-59-15

E: Error opening file “/var/log/timeshift/2020-12-14_22-49-45_delete.log”: No space left on device
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45

/dev/dm-0 is mounted at: /run/timeshift/backup, options: rw,relatime,compress=zstd:3,ssd,space_cache,skip_balance,commit=120,subvolid=5,subvol=/

E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: Error creating directory /root/.temp/3B4CdgIn: No space left on device
E: Failed to create directory: /root/.temp/3B4CdgIn/2020-12-14_22-49-45
E: ERROR: can’t list qgroups: quotas not enabled

E: btrfs returned an error: 256
E: Failed to query subvolume quota

Removing snapshot: 2020-12-13_15-59-15
Deleting subvolume: @ (Id:2019)
Deleted subvolume: @ (Id:2019)

Destroying qgroup: 0/2019
E: Failed to destroy qgroup: ‘0/2019’
E: Failed to remove snapshot: 2020-12-13_15-59-15

df -k

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/cryptdata 243671040 212862092 30743540 88% /
udev 8079640 0 8079640 0% /dev
tmpfs 1625596 2000 1623596 1% /run
/dev/sda1 613036 57748 555288 10% /boot/efi
/dev/mapper/cryptdata 243671040 212862092 30743540 88% /home
/dev/mapper/cryptdata 243671040 212862092 30743540 88% /swap
/dev/mapper/cryptdata 243671040 212862092 30743540 88% /run/timeshift/backup

With the errors deleting timeshifts I tried a rm -rf of some small deb files in my home direcory and there were no errors, and the files were actually deleted, unlike last night when it would take several attempts.

Next I tried both rebalancing commands to receive error output:

sudo btrfs balance start -dusage=50 -dlimit=2 -musage=50 -mlimit=4 /

ERROR: error during balancing ‘/’: Operation now in progress

sudo sh -c 'for i in 0 1 5 10 15 20 25 30 40 50 60 70 80 90 100; do echo "${0}: Running with ${i}%"; sudo btrfs balance start -dusage=$i -musage=$i /; done'

sh: Running with 0%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 1%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 5%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 10%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 15%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 20%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 25%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 30%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 40%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 50%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 60%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 70%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 80%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 90%
ERROR: error during balancing ‘/’: Operation now in progress
sh: Running with 100%
ERROR: error during balancing ‘/’: Operation now in progress

With timeshift --list now showing 12 instead of 13, after many more attempts deleting different snapshots I was able to bring that number down to 8.

And although space in use is observed decreasing:

df -k

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/cryptdata 243671040 203188056 39555480 84% /
udev 8079640 0 8079640 0% /dev
tmpfs 1625596 1992 1623604 1% /run
/dev/sda1 613036 57748 555288 10% /boot/efi
/dev/mapper/cryptdata 243671040 203188056 39555480 84% /home
/dev/mapper/cryptdata 243671040 203188056 39555480 84% /swap
/dev/mapper/cryptdata 243671040 203188056 39555480 84% /run/timeshift/backup

Both rebalancing comands fail with the same errors as before.

From here I begin the procedure to add a ram disk to the file system in an attempt to allow rebalancing work: https://pastebin.com/3NpEM7qs

sudo mknod -m 660 /dev/ram0 b 1 0
sudo chown root:disk /dev/ram0
sudo mkdir /mnt/ramdisk
sudo mount -t ramfs -o size=4G,maxsize=4G /dev/ram0 /mnt/ramdisk
sudo dd if=/dev/zero of=/mnt/ramdisk/extend.img bs=4M count=1000

1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB, 3.9 GiB) copied, 1.25105 s, 3.4 GB/s

sudo losetup -fP /mnt/ramdisk/extend.img

sudo losetup -a |grep extend.img

/dev/loop6: [0065]:895566 (/mnt/ramdisk/extend.img)

sudo btrfs device add /dev/loop6 /

sudo btrfs fi usage / |head -n 6

Overall:
Device size: 232.38GiB
Device allocated: 232.38GiB
Device unallocated: 1.00MiB
Device missing: 0.00B
Used: 187.10GiB

echo 'scale=2;187/232' |bc

.80

Same error trying to balance with the ramdisk as without:

sudo btrfs balance start -dusage=80 /

ERROR: error during balancing ‘/’: Operation now in progress

I also tried the two different balance commands you gave me and they also error.

I rebooted, and thankfully I was able to normal boot to a login screen! On logging in everything seems sluggish so I left the system alone and tried several reboots. Despite this both balancing command still give the same errors!

As a precaution I have removed all of the remaining Timeshifts for the GUI program and - for the moment - disabled all scheduling for snapshots. I also deleted the one game I had installed about a week ago to allow more free up another 8 GiB of space.

Is there anything else I can try to get balancing working please?

Also, do you just use a cronjob to run it every 3 days?