SYSTEM TOASTED - Help much needed - "No space left" messages?

linuxislife · January 24, 2026, 1:18am

Hey

So, an update when wrong…
I don’t know what happen and if there is a log I can retrieve or if that would help ?

longer story...

What I know is, while the update was probably still running, I did in another terminal (because I saw the prompt on the update):

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=EndeavourOS
grub-mkconfig -o /boot/grub/grub.cfg

it kind of told me no kernel were found… that’s when I realized the update might still be running… so I just wanted to wait and redo it after.

I don’t think that has to do with the rest of the problem (beside the fact I don’t have a bootloader anymore of course )

But at one point the update got stuck, the progression percentages weren’t moving. I checked my connection, everything was fine. I had many browser pages open and thought than could be the problem. A moment later the percentage moved, but 1% more… so I decided to leave it there and come back later.

When I came back many errors of “couldn’t extract” the packages (I think…) were there, so I decided to just yay again… it started but at one point the same behavior occurred. I had no more stuff in the background running but the computer was very slow responsive. htop wasn’t showing anything wrong though.

It shows some more errors (now I realize how bad it sound to just says “some errors”.. sorry… again, if there logs to be found please tell me, I couldn’t find it), and something about mirrors, something about space… I decided to clear pacman cahe to free space paccache -rk2… but I had space, like 40Go…

System was so unresponsive I decided to reboot… knowing I would have to chroot my way in… I even had to REISUB …

Well here we are… booting from live iso :

lsblk

[liveuser@eos-2025.02.08 ~]$ lsblk -f

NAME        FSTYPE     FSVER      LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0       squashfs   4.0                                                               0   100% /run/archiso/airootfs
sda         iso9660    Joliet Ext EOS_202502  2025-02-08-08-06-14-00                              
├─sda1      iso9660    Joliet Ext EOS_202502  2025-02-08-08-06-14-00                     0   100% /run/archiso/bootmnt
└─sda2      vfat       FAT32      ARCHISO_EFI 67A7-1076                                           
nvme0n1                                                                                           
├─nvme0n1p1 vfat       FAT32                  D476-FF9D                                           
├─nvme0n1p2                                                                                       
├─nvme0n1p3                                                                                       
├─nvme0n1p4 ntfs                              0E30873F30872CB1                                    
├─nvme0n1p5 ext4       1.0                    f3496a94-266c-4596-a437-da3530ad8f7e                
└─nvme0n1p6 crypto_LUK 1                      f6540be3-39bf-401d-8c00-5603b407b4cd                


[liveuser@eos-2025.02.08 ~]$ lsblk -l
NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0       7:0    0   2.6G  1 loop /run/archiso/airootfs
sda         8:0    1   7.5G  0 disk 
sda1        8:1    1   2.8G  0 part /run/archiso/bootmnt
sda2        8:2    1   177M  0 part 
nvme0n1   259:0    0 465.8G  0 disk 
nvme0n1p1 259:1    0   100M  0 part 
nvme0n1p2 259:2    0    16M  0 part 
nvme0n1p3 259:3    0 106.4G  0 part 
nvme0n1p4 259:4    0   824M  0 part 
nvme0n1p5 259:5    0   512M  0 part 
nvme0n1p6 259:6    0 357.9G  0 part

/ partition is encrypted, BTRFS on LUKS so:

Decrypt and mount luks partition

[liveuser@eos-2025.02.08 ~]$ sudo cryptsetup luksOpen /dev/nvme0n1p6 crypt &&
sudo mount -o compress=zstd,subvol=@ /dev/mapper/crypt /mnt
Enter passphrase for /dev/nvme0n1p6:

Mount boot folder

[liveuser@eos-2025.02.08 ~]$ sudo mount /dev/nvme0n1p5 /mnt/boot

Mount EFI boot folder

[liveuser@eos-2025.02.08 ~]$ sudo mount /dev/nvme0n1p1 /mnt/boot/efi

`chroot` into system

[liveuser@eos-2025.02.08 ~]$ sudo arch-chroot /mnt

fstab

[root@EndeavourOS /]# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=f3496a94-266c-4596-a437-da3530ad8f7e /boot          ext4    defaults,noatime 0 2
UUID=D476-FF9D                            /boot/efi      vfat    umask=0077 0 2
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /              btrfs   subvol=/@,defaults,noatime,space_cache=v2,compress-force=zstd 0 0
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /home          btrfs   subvol=/@home,defaults,noatime,space_cache=v2,compress-force=zstd 0 0
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /var/cache     btrfs   subvol=/@cache,defaults,noatime,space_cache=v2,compress-force=zstd 0 0
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /var/cache/pacman/pkg          btrfs   subvol=/@var-cache-pacman-pkg,defaults,noatime,space_cache=v2,compress-force=zstd 0 0
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /var/log       btrfs   subvol=/@log,defaults,noatime,space_cache=v2,compress-force=zstd 0 0
/dev/mapper/luks-f6540be3-39bf-401d-8c00-5603b407b4cd /swap btrfs subvol=@swap,defaults,noatime,compress=no,nodatacow,nodatasum 0 0
/swap/swapfile none swap defaults 0 0

Here comes the real problem

[root@EndeavourOS /]# yay -Syyuu
 -> Avoid running yay as root/sudo.
 -> exit status 1
error: config file /etc/pacman.d/endeavouros-mirrorlist could not be read: No such file or directory
error parsing '/etc/pacman.conf'

ls /etc/pacman.d

[root@EndeavourOS /]# ls /etc/pacman.d
endeavouros-mirrorlist.2022.11.28-19.37.26  endeavouros-mirrorlist.2023.03.21-11.18.35  endeavouros-mirrorlist.2023.11.23-10.14.54  endeavouros-mirrorlist.2024.03.27-12.10.35
endeavouros-mirrorlist.2022.12.19-12.18.08  endeavouros-mirrorlist.2023.04.14-10.01.34  endeavouros-mirrorlist.2023.12.18-23.04.06  endeavouros-mirrorlist.2024.03.27-13.05.21
endeavouros-mirrorlist.2022.12.28-12.47.58  endeavouros-mirrorlist.2023.06.25-15.12.12  endeavouros-mirrorlist.2024.01.03-20.35.30  endeavouros-mirrorlist.2024.03.30-17.06.09
endeavouros-mirrorlist.2022.12.29-17.25.49  endeavouros-mirrorlist.2023.08.22-15.37.37  endeavouros-mirrorlist.2024.01.16-15.00.12  endeavouros-mirrorlist.bak
endeavouros-mirrorlist.2023.01.03-20.58.39  endeavouros-mirrorlist.2023.09.01-14.13.34  endeavouros-mirrorlist.2024.01.28-14.54.47  endeavoursos-mirrorlist
endeavouros-mirrorlist.2023.01.17-18.15.35  endeavouros-mirrorlist.2023.09.29-11.23.42  endeavouros-mirrorlist.2024.02.07-11.10.01  gnupg
endeavouros-mirrorlist.2023.01.30-11.21.27  endeavouros-mirrorlist.2023.10.05-22.47.10  endeavouros-mirrorlist.2024.03.07-16.20.11  mirrorlist
endeavouros-mirrorlist.2023.02.28-20.11.50  endeavouros-mirrorlist.2023.10.18-16.30.57  endeavouros-mirrorlist.2024.03.07-21.30.27  mirrorlist.bak
endeavouros-mirrorlist.2023.03.15-11.37.51  endeavouros-mirrorlist.2023.11.07-15.30.53  endeavouros-mirrorlist.2024.03.14-20.28.22  mirrorlist.pacnew
[root@EndeavourOS /]#

endeavouros-mirrorlist is named endeavoursos-mirrorlist somehow…!

So i was happy to find the beginning of a solution and wanted to rename its backup:

[root@EndeavourOS /]# cd /etc/pacman.d
[root@EndeavourOS pacman.d]# mv endeavouros-mirrorlist.bak endeavouros-mirrorlistBAKBAK.bak
mv: cannot overwrite 'endeavouros-mirrorlistBAKBAK.bak': No space left on device

Here is my disk usage:

[root@EndeavourOS /]# btrfs filesystem usage /
Overall:
    Device size:                 357.90GiB
    Device allocated:            357.90GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Device slack:                  3.50KiB
    Used:                        312.49GiB
    Free (estimated):             44.92GiB      (min: 44.92GiB)
    Free (statfs, df):               0.00B
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 112.00KiB)
    Multiple profiles:                  no

Data,single: Size:354.88GiB, Used:309.96GiB (87.34%)
   /dev/mapper/crypt     354.88GiB

Metadata,single: Size:3.01GiB, Used:2.53GiB (84.10%)
   /dev/mapper/crypt       3.01GiB

System,single: Size:4.00MiB, Used:64.00KiB (1.56%)
   /dev/mapper/crypt       4.00MiB

Unallocated:
   /dev/mapper/crypt       1.00MiB

My first question is: could you please help me?

I'm an *****

And how can I access my /home directory , which is on the encrypted partition? I thought decrypting and mounting the partition was enough, but I assume I have to mount /home too probably (or do i? )
I can’t figure out how to access /home… (Or is it nuked??)
sudo mount -o compress=zstd,subvol=@home /dev/mapper/crypt /mnt/home
…indeed

Any help would be extremely appreciated, I tried other files, renaming, moving, creating new ones, same message all the time No space left on device even in nano

ddnn · January 24, 2026, 1:27am

Please ignore. Just here to see how this unfolds.

ricklinux · January 24, 2026, 1:55am

Georges Geórgos:

What I know is, while the update was probably still running, I did in another terminal (because I saw the prompt on the update):
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=EndeavourOS
grub-mkconfig -o /boot/grub/grub.cfg

Don’t tell me you ran this from another terminal when the system was still updating. Not supposed to do that. So you booted on the live ISO and arch-chroot. Did you try running an update? Then fix the bootloader? It looks like you messed up the mirror lists with pacnew files? Something i never bother with.

I don’t use encryption but i do use btrfs with the standard setup that is provided by EOS. I’m not an expert on btrfs but when i arch-chroot. I use the following method

sudo mount -o subvol=@ /dev/sdxn /mnt

I’m not familiar with using luks and arch-chroot on btrfs. Sorry!

If it were me i would fix the mirror lists and try updating and fixing the bootloader.

Edit: This is what I would try myself first or maybe you have and the problem is more that that.

linuxislife · January 24, 2026, 1:58am

Thanks Rick for your reply.

It is exactly what I wanted to do in chroot, but the everything is full id going on, can’t do anything.

(please let’s not talk about the grub part ), and no i didn’t mess with pacnew files… not that am aware of..

ricklinux · January 24, 2026, 2:01am

I did look at the wiki for unlocking when arch-chroot if using luks. Did you accomplish that properly?

https://discovery.endeavouros.com/system-rescue/arch-chroot/2022/12/

ricklinux · January 24, 2026, 2:03am

Grub is a nothing burger … it’s simple. That’s all i use. But i won’t talk about it.

Edit: Did you try removing your paccache?

linuxislife · January 24, 2026, 2:04am

yes i’m pretty sure i got the chroot right and in order at that point.

the problem “full no space” was what prevented me to update in the first place when the system was still running… and it’s still there in chroot

[root@EndeavourOS /]# pacman -Syu
error: config file /etc/pacman.d/endeavouros-mirrorlist could not be read: No such file or directory

and even if I just try to remove endeavouros-mirrorlist from pacman.conf I cannot edit the file in any way, nano tells me disk full !

ricklinux · January 24, 2026, 2:10am

You can’t use the welcome app and update the arch mirror list?

ricklinux · January 24, 2026, 2:13am

Looks like this is the issue. Look at mine.

[ricklinux@rick-ms7c91 ~]$ sudo btrfs filesystem usage /
Overall:
    Device size:                 929.51GiB
    Device allocated:            109.07GiB
    Device unallocated:          820.44GiB
    Device missing:                  0.00B
    Device slack:                  3.00KiB
    Used:                         94.17GiB
    Free (estimated):            832.73GiB      (min: 422.51GiB)
    Free (statfs, df):           832.73GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              146.03MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:105.01GiB, Used:92.72GiB (88.30%)
   /dev/nvme0n1p2        105.01GiB

Metadata,DUP: Size:2.00GiB, Used:740.95MiB (36.18%)
   /dev/nvme0n1p2          4.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
   /dev/nvme0n1p2         64.00MiB

Unallocated:
   /dev/nvme0n1p2        820.44GiB

linuxislife · January 24, 2026, 2:14am

when I was still on the system running from the machine I couldn’t lauch Welcome at the end…

Now I’m on the live iso, I don’t think launching Welcome will update those on my machine, or will it?

linuxislife · January 24, 2026, 2:16am

I dunno i will try and erase a big file, but I have like 40Gi left, and went way fuller than that before…

edit:

removed 15Go, nothing changes, can’t modify any files

ricklinux · January 24, 2026, 2:31am

I really don’t know what the issue might be.

ricklinux · January 24, 2026, 2:38am

Have you tried rebooting again with the live ISO and arch-chroot again just to see?

linuxislife · January 24, 2026, 2:43am

yes just now

and whatever I do disk full, no mater what I wanna do, or where I am (/home, /)

and btw

[liveuser@eos-2025.02.08 ~]$ sudo arch-chroot /mnt

[root@EndeavourOS /]# grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=EndeavourOS
Installing for x86_64-efi platform.
Installation finished. No error reported.

[root@EndeavourOS /]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found theme: /boot/grub/themes/EndeavourOS/theme.txt
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
/usr/share/os-prober/common.sh: line 42: /var/lib/os-prober/labels: No space left on device
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No space left on device
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No space left on device
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No space left on device
Found Windows Boot Manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Kernels not found.
If you think an error has occurred, please file a bug report at "https://github.com/Antynea/grub-btrfs"
Unmount /tmp/grub-btrfs.9Zho8a9Vyz .. Success
done

linuxislife · January 24, 2026, 2:46am

No matter how much I delete…!

[root@EndeavourOS /]# df -h
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/crypt  358G  303G     0 100% /
/dev/nvme0n1p5     488M   33M  420M   8% /boot
/dev/nvme0n1p1      96M   74M   23M  77% /boot/efi
/dev/mapper/crypt  358G  303G     0 100% /home
efivarfs           150K   89K   57K  61% /sys/firmware/efi/efivars
udev                31G     0   31G   0% /dev
shm                 32G     0   32G   0% /dev/shm
run                 32G   10M   32G   1% /run
tmp                 32G     0   32G   0% /tmp
airootfs            10G  154M  9.9G   2% /etc/resolv.conf

just heard about inodes… are those 0 supposed to be there?

[root@EndeavourOS /]# df -i
Filesystem         Inodes IUsed   IFree IUse% Mounted on
/dev/mapper/crypt       0     0       0     - /
/dev/nvme0n1p5      32768   461   32307    2% /boot
/dev/nvme0n1p1          0     0       0     - /boot/efi
/dev/mapper/crypt       0     0       0     - /home
efivarfs                0     0       0     - /sys/firmware/efi/efivars
udev              8125060   749 8124311    1% /dev
shm               8145797     1 8145796    1% /dev/shm
run               8145797  1278 8144519    1% /run
tmp               8145797     1 8145796    1% /tmp
airootfs          8145797  3374 8142423    1% /etc/resolv.conf

_Six · January 24, 2026, 3:56am

I am not entirely sure how to fix this particular issue but I don’ think those inodes should be 0.

Filesystems have an inode table and create an allocated inode table, if there are lots of small files on the system then the inode table can be used up before the actual space on the drive is used up and will prevent the creation of new files. And can be caused by lots of log files being generated over time. Inodes are data objects that store data such as size of the file, who owns it, metadata, permissions strings etc, but the name of the file and the content of the file is not stored in inodes, so inodes are everything other than file name and the content of the file.

The reason I don’t know how to fix it is because the video I was writing the notes from didn’t mention how to fix it as they mentioned this issue can happen on something like an email server (and you will know since the server will stop sending emails) but next to no chance of ever running out of inodes on a desktop PC so it was beyond the scope of that video. Not only that but I am a bit suspicious of the “inodes” column showing as 0 in particular as this would suggest there wasn’t any inodes to run out in the first place which shouldn’t be the case. When I use df -i all 16 entries have numbers in all all three columns for me, “Inodes”, “IUsed”, and “IFree”.

Unfortunately this is all I can say about this particular issue due to all those columns being 0 and I haven’t read anything about that happening in particular otherwise removing small files should reduce the usage of inodes. If it was my own system and data I would probably try various things due to backups etc but in this case I would be messing with someone else’s data without knowing what would happen and I am not qualified to tackle this specific issue but only really know about inodes and seeing them in the filesystem and not so much about fixing them.

ricklinux · January 24, 2026, 2:19pm

@linuxislife
I thinks you broke it!

linuxislife · January 24, 2026, 2:20pm

I know, right

linuxislife · January 24, 2026, 5:18pm

thanks for your time @ricklinux @_Six (and @ddnn for the popcorn )

So, I will start to try and understand more the btrfs filesystem as I secretly hope it is were the problem might be… from what I see so far

In Btrfs, df -i reporting 0 for Inodes is normal because Btrfs allocates inodes dynamically.

Well, so there’s that… and also I will look further this way:

Read this: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

It seems that you are suffering the “metadata is full case” of the article, because Btrfs reserved 0.5 GB for itself, meaning it is full and therefore preventing any new writes. The article suggests that you should do a full rebalance on it.

Keep also on mind that normal df lies, it is never going to tell you the real btrfs usage stats.

# btrfs filesystem usage /
Overall:
    Device size:                 357.90GiB
    Device allocated:            357.90GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Device slack:                  3.50KiB
    Used:                        302.31GiB
    Free (estimated):             55.09GiB      (min: 55.09GiB)
    Free (statfs, df):               0.00B
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 144.00KiB)
    Multiple profiles:                  no

Data,single: Size:354.88GiB, Used:299.79GiB (84.48%)
   /dev/mapper/crypt     354.88GiB

Metadata,single: Size:3.01GiB, Used:2.51GiB (83.62%)
   /dev/mapper/crypt       3.01GiB

System,single: Size:4.00MiB, Used:64.00KiB (1.56%)
   /dev/mapper/crypt       4.00MiB

Unallocated:
   /dev/mapper/crypt       1.00MiB

maybe @dalto knows if all this is btrfs related…

dalto · January 24, 2026, 5:26pm

A couple of questions.

Do you have snapshots? If so, deleting data won’t free up much space since that data will still be held by your snapshots
It looks like you may need to run a btrfs balance. Are you running those regularly?