No space left on device for rescue

Brolands · September 17, 2023, 6:05pm

Hello,

Several days ago I managed to break the system with an update - launched the update, left PC running, returned and without looking shut PC down. Now I can’t turn it on, it gives some errors - don’t remember which, as I can’t boot into my SSD anymore, because I tried to clean efibootmgr entries as rescue operation.

Here is output of several commands and methods that I’ve tried (arch-chrooted):

[root@EndeavourOS /]# btrfs subvolume list /
ID 257 gen 318432 top level 5 path @home
ID 258 gen 126654 top level 5 path @cache
ID 259 gen 129896 top level 5 path @log
ID 483 gen 309712 top level 5 path timeshift-btrfs/snapshots/2023-08-09_18-43-09/@
ID 505 gen 290498 top level 5 path @
ID 508 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-08-09_19-14-32/@
ID 569 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-09-02_17-36-42/@
ID 570 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-09-03_09-57-52/@
ID 571 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-03_18-34-58/@
ID 572 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-06_17-19-14/@
ID 573 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-10_14-26-24/@
ID 574 gen 129057 top level 5 path timeshift-btrfs/snapshots/2023-09-12_19-29-12/@
[root@EndeavourOS /]# findmnt /
TARGET
  SOURCE             FSTYPE OPTIONS
/ /dev/nvme0n1p1[/@] btrfs  rw,relatime,ssd,discard=async,space_cache=v2,subvolid=505,subvol=/@
[root@EndeavourOS /]# sudo timeshift --restore
E: Error opening file “/var/log/timeshift/2023-09-17_20-34-28_restore.log”: No space left on device

Select backup device:

Num    Device  Size  Type  Label  
------------------------------------------------------------------------------

Enter device name or number (a=Abort): 
a
Aborted.


[root@EndeavourOS Documents]# sudo df /
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme0n1p1 209729536 135615200  73835552  65% /

[root@EndeavourOS Documents]# sudo echo "1" > test
bash: test: No space left on device

[root@EndeavourOS Documents]# sudo pacman -Syu
:: Synchronizing package databases...
error: failed to synchronize all databases (unable to lock database)

How I’ve arch-chrooted:

[liveuser@eos-2023.05.28 ~]$ sudo mount -o subvol=@ /dev/nvme0n1p1 /mnt
[liveuser@eos-2023.05.28 ~]$ sudo mount -o subvol=@home /dev/nvme0n1p1 /mnt/home
[liveuser@eos-2023.05.28 ~]$ sudo mount -o subvol=@log /dev/nvme0n1p1 /mnt/var/log
[liveuser@eos-2023.05.28 ~]$ sudo mount -o subvol=@cache /dev/nvme0n1p1 /mnt/var/cache
[liveuser@eos-2023.05.28 ~]$ sudo mount  /dev/nvme0n1p5 /mnt/boot/efi
[liveuser@eos-2023.05.28 ~]$ sudo arch-chroot /mnt

I don’t understand why it tells me that I don’t have space left on device. Same with GRUB - when I try to run grub-install, it tells me that no space left on device.

Oh, and system time looks to be incorrect - 3 hours behind.

Thank you!

dalto · September 17, 2023, 6:07pm

Is it read-only? What does findmnt / return?

Brolands · September 17, 2023, 6:09pm

Looks like read-write:

[root@EndeavourOS /]# findmnt /
TARGET
  SOURCE             FSTYPE OPTIONS
/ /dev/nvme0n1p1[/@] btrfs  rw,relatime,ssd,discard=async,space_cache=v2,subvolid=505,subvol=/@

dalto · September 17, 2023, 6:11pm

Actually. In that test, you were in Documents. What does df -h show?

Brolands · September 17, 2023, 6:13pm

df -h from root:

[root@EndeavourOS /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  201G  130G   71G  65% /
/dev/nvme0n1p1  201G  130G   71G  65% /home
/dev/nvme0n1p1  201G  130G   71G  65% /var/log
/dev/nvme0n1p1  201G  130G   71G  65% /var/cache
/dev/nvme0n1p5  300M  864K  299M   1% /boot/efi
udev            7.5G     0  7.5G   0% /dev
shm             7.6G     0  7.6G   0% /dev/shm
run             7.6G     0  7.6G   0% /run
tmp             7.6G     0  7.6G   0% /tmp
airootfs         10G   87M   10G   1% /etc/resolv.conf

dalto · September 17, 2023, 6:13pm

Interesting. Does touch /testfile work?

Brolands · September 17, 2023, 6:14pm

Unfortunately not

[root@EndeavourOS /]# sudo touch /testfile
touch: cannot touch '/testfile': No space left on device

dalto · September 17, 2023, 6:16pm

Oh. It is probably because df isn’t accurate for btrfs volumes.

Try deleting some of the older snapshots you don’t need anymore.

Also, rm /var/lib/pacman/db.lck might help with pacman.

Brolands · September 17, 2023, 6:24pm

Just to make sure, I would do it like this: btrfs subvolume delete -i 483?

Brolands · September 17, 2023, 6:32pm

BTW, looks like Timeshift snapshots should be under /run/timeshift folder, but I have nothing in there. Maybe I didn’t mount some subvolume?
Added my mounting information to post description.

Brolands · September 17, 2023, 6:42pm

Deleted a subvolume, but still no space:

[root@EndeavourOS /]# [root@EndeavourOS /]# sudo btrfs subvolume delete -i 483 /
Delete subvolume (no-commit): '//timeshift-btrfs/snapshots/2023-08-09_18-43-09/@'
[root@EndeavourOS /]# sudo btrfs subvolume list /
ID 257 gen 318432 top level 5 path @home
ID 258 gen 126654 top level 5 path @cache
ID 259 gen 129896 top level 5 path @log
ID 505 gen 290498 top level 5 path @
ID 508 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-08-09_19-14-32/@
ID 569 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-09-02_17-36-42/@
ID 570 gen 128977 top level 5 path timeshift-btrfs/snapshots/2023-09-03_09-57-52/@
ID 571 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-03_18-34-58/@
ID 572 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-06_17-19-14/@
ID 573 gen 128978 top level 5 path timeshift-btrfs/snapshots/2023-09-10_14-26-24/@
ID 574 gen 129057 top level 5 path timeshift-btrfs/snapshots/2023-09-12_19-29-12/@
[root@EndeavourOS /]# sudo touch /testfile
touch: cannot touch '/testfile': No space left on device

I tried to delete some files as well, and then create file immediately after that, no luck.

[root@EndeavourOS Documents]# ls GuacameleeSTCE_EGS/
SAVE.DAT
[root@EndeavourOS Documents]# rm Guacamelee2_EGS/ -r
[root@EndeavourOS Documents]# rm GuacameleeSTCE_EGS/ -r
[root@EndeavourOS Documents]# touch /testfile
touch: cannot touch '/testfile': No space left on device

Brolands · September 17, 2023, 6:48pm

This might be useful as well

dalto · September 17, 2023, 7:11pm

Delete this snapshot: timeshift-btrfs/snapshots/2023-08-09_19-14-32/@

Brolands · September 17, 2023, 7:19pm

I deleted ‘timeshift-btrfs/snapshots/2023-08-09_18-43-09/@’, still no space.

dalto · September 17, 2023, 7:22pm

What does btrfs filesystem usage / show?

Brolands · September 17, 2023, 7:38pm

Filesystem usage result:

[root@EndeavourOS /]# sudo btrfs filesystem usage /
Overall:
    Device size:		 200.01GiB
    Device allocated:		 200.01GiB
    Device unallocated:		   1.00MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 126.85GiB
    Free (estimated):		  72.45GiB	(min: 72.45GiB)
    Free (statfs, df):		  72.45GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 288.06MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:196.00GiB, Used:123.54GiB (63.03%)
   /dev/nvme0n1p1	 196.00GiB

Metadata,DUP: Size:2.00GiB, Used:1.65GiB (82.66%)
   /dev/nvme0n1p1	   4.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%)
   /dev/nvme0n1p1	  16.00MiB

Unallocated:
   /dev/nvme0n1p1	   1.00MiB

manuel · September 17, 2023, 7:40pm

Could the problem be the EFI partition size is only 300 MiB?

dalto · September 17, 2023, 7:45pm

It is fully allocated. Try running a balance on that filesystem.

btrfs balance start /

No, it is almost totally empty.

Brolands · September 18, 2023, 4:31am

Wow! Thank you very much!
Didn’t know that it really was memory issue. Thought it could’ve been consequence, rather than issue

GjMan78 · September 18, 2023, 5:24am

I don’t know if you know it but I recommend you install btrfs-assistant to set automatic scrub and balance.

yay -S btrfs-assistant btrfsmaintenance