No space left, despite having 200GB free space (BTRFS install)

franticfranny · July 5, 2023, 6:55am

Hi there,
I am experiencing an issue where my system says the disk drive is full, when it is not. Therefore stopping me from booting, and editing files.

Filesystem: BTRFS, encrypted with Luks with 1TB storage. ~800GB used.
System broke while selfhosted AI was generating a response. My hardware being insufficient for model used.
Everything froze and computer told me there’s no space.
I rebooted. System can no longer boot since no space.

I can access TTY, and browse filesystem. I can delete files. Deleting files shows more available space, yet system still says no free space when booting, or trying to create a file or folder.

On boot, it goes to the loading screen, but stays there due to errors. I can go to TTY from there and access file system.

As shown here, the SSD is /dev/mapper/nvme0n1p1_crypt and it 180GB free space.

Does anyone have any ideas on how to solve the issue?

Edit:
Tried btrfs check but there were no errors.

mbod · July 5, 2023, 7:39am

Please provide output of :

sudo btrfs fi show

and

btrfs fi usage /mnt
btrfs fi usage /mnt/home
... etc.

franticfranny · July 5, 2023, 7:48am

here you go! I hope that tells you more than it tells me.

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi show

Label: none  uuid: 76e04d9a-f1ea-42b1-ba9f-e4f59262d4ab
	Total devices 1 FS bytes used 749.73GiB
	devid    1 size 931.51GiB used 931.51GiB path /dev/mapper/nvme0n1p1_crypt

[liveuser@eos-2022.09.10 ~]$ sudo mount -o subvol=@ /dev/mapper/nvme0n1p1_crypt /mnt
[liveuser@eos-2022.09.10 ~]$ sudo mount -o subvol=@home /dev/mapper/nvme0n1p1_crypt /mnt/home
[liveuser@eos-2022.09.10 ~]$ sudo mount -o subvol=@cache /dev/mapper/nvme0n1p1_crypt /mnt/var/cache
[liveuser@eos-2022.09.10 ~]$ sudo mount -o subvol=@log /dev/mapper/nvme0n1p1_crypt /mnt/var/log
[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt/home

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt/home

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt/var/log

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

[liveuser@eos-2022.09.10 ~]$ sudo btrfs fi usage /mnt/var/cache

Overall:
    Device size:		 931.51GiB
    Device allocated:		 931.51GiB
    Device unallocated:		   1.03MiB
    Device missing:		     0.00B
    Used:			 753.31GiB
    Free (estimated):		 177.34GiB	(min: 177.34GiB)
    Free (statfs, df):		 177.34GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:923.49GiB, Used:746.15GiB (80.80%)
   /dev/mapper/nvme0n1p1_crypt	 923.49GiB

Metadata,DUP: Size:4.00GiB, Used:3.58GiB (89.42%)
   /dev/mapper/nvme0n1p1_crypt	   8.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/nvme0n1p1_crypt	  16.00MiB

Unallocated:
   /dev/mapper/nvme0n1p1_crypt	   1.03MiB

joekamprad · July 5, 2023, 7:58am

welcome @franticfranny i am sure we can fix this here better as on telegram.
may @dalto have some time to check in now or later

mbod · July 5, 2023, 10:56am

This confirms that the file system is full

But interesting that it also says:

    Free (estimated):		 177.34GiB	(min: 177.34GiB)

Do you have snapshots on the file system? If yes, can you delete some and see if that improves the situation?

PS
By the way, I am not using brtfs myself except for a 500GB test drive with btrfs. This drive has only one folder with 97.6 GiB of data and btrfs usage says:

    Device size:		 465.75GiB
    Device allocated:		  95.02GiB
    Device unallocated:		 370.72GiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			  92.74GiB
    Free (estimated):		 372.21GiB	(min: 186.84GiB)

It clearly shows that only 95 GiB are allocated. Which is what I would expect. So this allocation number seems to be odd in your case.

dalto · July 5, 2023, 11:09am

Run btrfs balance on that filesystem.

franticfranny · July 5, 2023, 11:16am

Thank you for this. Unfortunately it tells me that it cannot balance as there’s no space left on device.

dalto · July 5, 2023, 11:16am

Do you have any snapshots you can delete?

franticfranny · July 5, 2023, 11:17am

Is it ok to delete the snapshots directly from the timeshift-btrfs folder? There’s a few snapshots I can delete and given time, 100GB of files give or take if need be.

dalto · July 5, 2023, 11:19am

Use the command sudo btrfs subvolume delete /path/to/snapshot.

Deleting older snapshots should free up more space than more recent snapshots.

franticfranny · July 5, 2023, 11:42am

I have successfully deleted all snapshots using sudo btrfs subvolume delete /mnt/timeshift-btrfs/snapshots/
And running that for each snapshot in the folder, there were around 8 in total. Does not seem like my system gained any space.
However booting into TTY and running sudo btrfs balance start / no longer instantly failed and seems to be running now. Will report back once it’s finished!

Thanks to you all for the help thus far. It truly is appreciated.

franticfranny · July 5, 2023, 4:47pm

It’s still running, was unaware the btrfs balance would take as this long on a single ssd!

So if I understand correctly, the unallocated is the “unused” and the free space is… in use but not actively storing files?

mbod · July 5, 2023, 8:49pm

Was this caused by a bug in btrfs? What was the root cause for btrfs filling up ike this?

@dalto: Do you have any idea?

dalto · July 5, 2023, 9:29pm

Depending on your usage, if you don’t run balance operations periodically this can happen. We will see if that is really the issue once @franticfranny finishes the balance operation.

I have never seen it happen on one of my installs but it isn’t all that uncommon for others to report it.

franticfranny · July 6, 2023, 5:08am

I am happy to report I woke up this morning to the balancing being complete. I believe it said it had to move/relocate 750 of 931 chunks/sectors.
I rebooted my PC and it booted like normal to my desktop environment, I now have 370GB of free space.

[strangeh21@strangeh21-eos ~]$ sudo btrfs fi usage /

Overall:
    Device size:                 931.51GiB
    Device allocated:            563.06GiB
    Device unallocated:          368.45GiB
    Device missing:                  0.00B
    Device slack:                  512.00B
    Used:                        559.08GiB
    Free (estimated):            370.11GiB      (min: 185.89GiB)
    Free (statfs, df):           370.11GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:556.00GiB, Used:554.33GiB (99.70%)
   /dev/mapper/luks-6b7a0064-ce68-4fa6-a463-5ebc4554d228         556.00GiB

Metadata,DUP: Size:3.50GiB, Used:2.38GiB (67.88%)
   /dev/mapper/luks-6b7a0064-ce68-4fa6-a463-5ebc4554d228           7.00GiB

System,DUP: Size:32.00MiB, Used:96.00KiB (0.29%)
   /dev/mapper/luks-6b7a0064-ce68-4fa6-a463-5ebc4554d228          64.00MiB

Unallocated:
   /dev/mapper/luks-6b7a0064-ce68-4fa6-a463-5ebc4554d228         368.45GiB

If there’s anything else you’d like me to check please do so, otherwise I’m happy to say this issue is resolved.
Thanks for the help to both of you!

dalto · July 6, 2023, 11:36am

Glad to here it worked for you. I would recommend scheduling a balance to run monthly. You can do this easily with the package btrfsmaintenance. If you would like a gui to manage that, btrfs-assistant provides a gui front-end to btrfs-maintenance and other btrfs functions.

system · July 8, 2023, 11:37am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.