/ become magically full

hi, I’m facing a problem I can’t manage by myself :frowning: , today / become magically full.

/usr/bin/df tell me 97% full :

# df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/dm-0          882G    811G   27G  97% /

but when i check with /usr/bin/du and /usr/bin/ncdu there is another story :face_with_raised_eyebrow:

# /usr/bin/du -d1 -h /
541G	/home
du: impossible d'accéder à '/run/user/1231/doc': Permission non accordée
du: impossible d'accéder à '/run/user/1231/gvfs': Permission non accordée
2,4M	/run
12K	/srv
16K	/lost+found
309M	/efi
16G	/usr
528K	/root
11G	/var
20K	/tmp
15M	/boot
1,3T	/data1
0	/sys
55M	/dev
4,0K	/mnt
du: impossible d'accéder à '/proc/4473/task/20074/fdinfo/67': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/4473/task/20083/fd/206': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/20105/task/20105/fd/4': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/20105/task/20105/fdinfo/4': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/20105/fd/3': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/20105/fdinfo/3': Aucun fichier ou dossier de ce nom
0	/proc
12M	/etc
1,3T	/data0
887M	/opt
3,2T	/




ncdu 2.9.2 ~ Use the arrow keys to navigate, press ? for help                                                                                                       
--- / --------------------------------------------------------------------------------------------------------------------------------------------------------------
    1.3 TiB [#######################] /data0                                                                                                                        
  540.4 GiB [#########              ] /home
   15.9 GiB [                       ] /usr
   10.4 GiB [                       ] /var
  886.7 MiB [                       ] /opt
  308.9 MiB [                       ] /efi
   14.2 MiB [                       ] /boot
   11.5 MiB [                       ] /etc
.   2.3 MiB [                       ] /run
  528.0 KiB [                       ] /root
e  16.0 KiB [                       ] /lost+found
   16.0 KiB [                       ] /tmp
   12.0 KiB [                       ] /srv
e   4.0 KiB [                       ] /mnt
e   4.0 KiB [                       ] /data1
    0.0   B [                       ] /proc
    0.0   B [                       ] /sys
    0.0   B [                       ] /dev
@   0.0   B [                       ]  sbin
@   0.0   B [                       ]  lib64
@   0.0   B [                       ]  lib
@   0.0   B [                       ]  bin

(/data0 & /data1 are on separate drive)

We are far from 811G cumulative for /

I’m sure few weeks ago all was fine ( / around 60% occupy), my guess is my NVME is dying but, what do you think ? if not, where are those missing gigas ?

Thanks for reading and for helping, sorry for my bad english.

Edit 1 : I’ve run fsck at boot without positive issue :frowning:

Which filesystem is that?

ext4

I do not understand your partition scheme. Is it correct that you only have one partition / and that include /home? Where is /data0 coming from?

Please share the output of ncdu -x /
The option -x is important to tell ncdu to stick to this one file system.

yep all in one for / /home /usr /var /tmp…on the encrypted nvme except /data0 & /data1 who come from separates drives

nvme0n1                                       259:0   0 931,5G 0 disk  
├─nvme0n1p1                                   259:1   0     1G 0 part  /efi
├─nvme0n1p2                                   259:2   0 896,2G 0 part  
│ └─luks-ebdf8437-0ee9-455a-abdd-33ba37eb65a0 253:0   0 896,2G 0 crypt /
└─nvme0n1p3                                   259:3   0  34,3G 0 part  
  └─luks-5b2b5ca3-98f6-4e77-936a-ce305fb88f64 253:1   0  34,3G 0 crypt [SWAP]

# ncdu -x /

ncdu 2.9.2 ~ Use the arrow keys to navigate, press ? for help                                                                                                       
--- / --------------------------------------------------------------------------------------------------------------------------------------------------------------
  540.5 GiB [#######################] /home                                                                                                                         
   15.9 GiB [                       ] /usr
   10.4 GiB [                       ] /var
  886.7 MiB [                       ] /opt
   14.2 MiB [                       ] /boot
   11.5 MiB [                       ] /etc
  528.0 KiB [                       ] /root
e  16.0 KiB [                       ] /lost+found
   12.0 KiB [                       ] /srv
e   4.0 KiB [                       ] /mnt
@   0.0   B [                       ]  sbin
@   0.0   B [                       ]  lib64
@   0.0   B [                       ]  lib
@   0.0   B [                       ]  bin
>   0.0   B [                       ] /tmp
>   0.0   B [                       ] /sys
>   0.0   B [                       ] /run
>   0.0   B [                       ] /proc
>   0.0   B [                       ] /efi
>   0.0   B [                       ] /dev
>   0.0   B [                       ] /data1
>   0.0   B [                       ] /data0

Maybe these commands could clarify the situation a bit:

LANG=C df -hT
LANG=C lsblk -fm

Is /data0 not related to zswap or zram? I have seen this before in my old Pop OS installation.

The output is missing the last line. Like this here

*Total disk usage: 37.3 GiB Apparent size: 35.3 GiB Items: 1.291.265

Stupid question: Have you rebooted?

# LANG=C df -hT
Sys. de fichiers Type     Taille Utilisé Dispo Uti% Monté sur
/dev/dm-0        ext4       882G    811G   26G  97% /
devtmpfs         devtmpfs    16G       0   16G   0% /dev
tmpfs            tmpfs       16G    244M   16G   2% /dev/shm
efivarfs         efivarfs   192K    157K   31K  84% /sys/firmware/efi/efivars
tmpfs            tmpfs      6,3G    2,3M  6,3G   1% /run
none             tmpfs      1,0M       0  1,0M   0% /run/credentials/systemd-cryptsetup@luks\x2d5b2b5ca3\x2d98f6\x2d4e77\x2d936a\x2dce305fb88f64.service
none             tmpfs      1,0M       0  1,0M   0% /run/credentials/systemd-cryptsetup@luks\x2debdf8437\x2d0ee9\x2d455a\x2dabdd\x2d33ba37eb65a0.service
none             tmpfs      1,0M       0  1,0M   0% /run/credentials/systemd-journald.service
none             tmpfs      1,0M       0  1,0M   0% /run/credentials/systemd-cryptsetup@luks\x2d32f17c7d\x2d9aff\x2d44ff\x2da8ab\x2dff3eb498ed5b.service
none             tmpfs      1,0M       0  1,0M   0% /run/credentials/systemd-cryptsetup@luks\x2d347d3455\x2da7b9\x2d479e\x2d8194\x2d975a96379db4.service
tmpfs            tmpfs       16G     68M   16G   1% /tmp
/dev/nvme0n1p1   vfat      1022M    309M  714M  31% /efi
/dev/dm-2        ext4       3,6T    1,3T  2,2T  38% /data1
/dev/dm-3        ext4       3,6T    1,3T  2,2T  38% /data0
tmpfs            tmpfs      3,2G    140K  3,2G   1% /run/user/1231

# LANG=C lsblk -fm
NAME FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS   SIZE OWNER GROUP MODE
sda                                                                                             3,6T root  disk  brw-rw----
└─sda1
  │  crypto 2                 347d3455-a7b9-479e-8194-975a96379db4                              3,6T root  disk  brw-rw----
  └─luks-347d3455-a7b9-479e-8194-975a96379db4
     ext4   1.0   data        12fddb77-8b96-4216-8d48-4520c1cf196a    2,1T    36% /data0        3,6T root  disk  brw-rw----
sdb                                                                                           447,1G root  disk  brw-rw----
├─sdb1
│    vfat   FAT32             DCA2-6DE8                                                         100M root  disk  brw-rw----
├─sdb2
│                                                                                                16M root  disk  brw-rw----
├─sdb3
│    ntfs                     DE78BB7478BB49D3                                                446,3G root  disk  brw-rw----
└─sdb4
     ntfs                     3290C30690C2CF91                                                  781M root  disk  brw-rw----
sdc                                                                                             3,6T root  disk  brw-rw----
└─sdc1
  │  crypto 2                 32f17c7d-9aff-44ff-a8ab-ff3eb498ed5b                              3,6T root  disk  brw-rw----
  └─luks-32f17c7d-9aff-44ff-a8ab-ff3eb498ed5b
     ext4   1.0   data-sync   2e119cc3-d447-4ff3-85d2-d9ade5e6860c    2,1T    36% /data1        3,6T root  disk  brw-rw----
nvme0n1
│                                                                                             931,5G root  disk  brw-rw----
├─nvme0n1p1
│    vfat   FAT32             26C6-2337                               713M    30% /efi            1G root  disk  brw-rw----
├─nvme0n1p2
│ │  crypto 2                 ebdf8437-0ee9-455a-abdd-33ba37eb65a0                            896,2G root  disk  brw-rw----
│ └─luks-ebdf8437-0ee9-455a-abdd-33ba37eb65a0
│    ext4   1.0   endeavouros bd3a0634-aad2-4e7c-a00e-34de1ca9bfa7   25,4G    92% /           896,2G root  disk  brw-rw----
└─nvme0n1p3
  │  crypto 2                 5b2b5ca3-98f6-4e77-936a-ce305fb88f64                             34,3G root  disk  brw-rw----
  └─luks-5b2b5ca3-98f6-4e77-936a-ce305fb88f64
     swap   1     swap        1fab2519-1bdf-474d-9149-c6e375d5eb4a                [SWAP]       34,3G root  disk  brw-rw----

ncdu 2.9.2 ~ Use the arrow keys to navigate, press ? for help                                                 
--- / --------------------------------------------------------------------------------------------------------
  540.7 GiB [###############] /home                                                                           
   15.9 GiB [               ] /usr
   10.8 GiB [               ] /var
  886.4 MiB [               ] /opt
   14.2 MiB [               ] /boot
   11.5 MiB [               ] /etc
  528.0 KiB [               ] /root
e  16.0 KiB [               ] /lost+found
   12.0 KiB [               ] /srv
e   4.0 KiB [               ] /mnt
@   0.0   B [               ]  sbin
@   0.0   B [               ]  lib64
@   0.0   B [               ]  lib
@   0.0   B [               ]  bin
>   0.0   B [               ] /tmp
>   0.0   B [               ] /sys
>   0.0   B [               ] /run
>   0.0   B [               ] /proc
>   0.0   B [               ] /efi
>   0.0   B [               ] /dev
>   0.0   B [               ] /data1
>   0.0   B [               ] /data0




*Total disk usage: 568.2 GiB   Apparent size: 566.8 GiB   Items: 1 022 928                                    


yep, multiple time. nothing has changed

those drives have been in a zpool before, but ZFS is not installed on this system.
plus, the problem show up suddenly. I’m sure few weeks ago / was around 60% occupy. :confused:

well you have 540 in your /home, which is the bulk of the usage, but 300 gigs is somewhere else… looks weird, do you have some files or . directories in your / ?

you could run this sudo du -sh * .* | sort -h in your / to get a fuller picture, it’s going to run for a while, but it should list everything sorted

I have similarly weird ext4 multiple partition setup, it ran for almost 3 minutes, listed the dotfiles too

is docker running on this system?

nope, no Docker here, I play with some Virtualbox guest sometimes but not 24/7

thank you !! you made me found those missing gigas !!
As you said it will take long time because it will run for a while, I prefer to unmount /data0 & /data1, and what a surprise :

# du -sh * .* | sort -h
du: impossible d'accéder à 'proc/4351/task/4354/fd/165': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/4351/task/4356/fd/165': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/4351/task/4365/fdinfo/96': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/4351/task/163388/fdinfo/165': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/4351/task/163388/fdinfo/166': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/163444/task/163444/fd/4': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/163444/task/163444/fdinfo/4': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/163444/fd/3': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'proc/163444/fdinfo/3': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à 'run/user/1231/doc': Permission non accordée
du: impossible d'accéder à 'run/user/1231/gvfs': Permission non accordée
du: impossible d'accéder à '.*': Aucun fichier ou dossier de ce nom
0	bin
0	lib
0	lib64
0	proc
0	sbin
0	sys
4,0K	data0
4,0K	mnt
12K	srv
16K	lost+found
100K	tmp
528K	root
2,4M	run
12M	etc
15M	boot
55M	dev
309M	efi
887M	opt
11G	var
16G	usr
243G	data1
541G	home

4,0K data0
243G data1

Looks like my backup run one time when /data1 was not mounted.

Thanks a lot to everyone who help me on this one, I feel a bit dumb :clown_face:

Huh I didn’t know you can mount something to a non empty directory… What happens if you mount something to a non empty directory? The content just disappears?

There’s a command to check if something is mounted so you don’t fill your /

Well, obviously it doesn’t really disappear, but the path to reach it is no longer traversable.

It still takes up space on the hard disk, but you can no longer see it.