Btrfs write about 300-500GB/day

General system Newbie
1

As the title says, my endeavourOS using btrfs have enormously high write amplication. Why?
My specs: Asrock Deskmini X300 with ryzen 7 5700G, 32GB RAM, new 1TB Samsung 980.

The kernel is 6.5.8-arch1-1, installed btrfs-assistant, btrfsmaintenance, snapper. The PCs is my daily driver for browsing web and web dev (php+mysql). Very little database activity.

My subvolumes are (with cow, compression enabled except @portables, @machines, @mysql):

  • ID 256 gen 75525 top level 5 path @
  • ID 257 gen 75523 top level 5 path @snapshots
  • ID 258 gen 75525 top level 5 path @home
  • ID 259 gen 75523 top level 5 path @root
  • ID 260 gen 74355 top level 5 path @cache
  • ID 261 gen 75525 top level 5 path @log
  • ID 262 gen 73347 top level 5 path @mysql
  • ID 263 gen 47365 top level 256 path var/lib/portables
  • ID 264 gen 47365 top level 256 path var/lib/machines
  • ID 265 gen 47374 top level 257 path @snapshots/1/snapshot
  • ID 267 gen 47365 top level 5 path @local

At first, I’m using snapper timeline snapshots to backup @ and @home, with max number save 20 for @ and 10 for @home, then reduce it to 5 for @ and 5 for @home. And then I disable it 2 days ago. Enable btrfs quotas, set weekly balancing for @snapshots, @home, @. Monthly scrub for @.

And all those time, it’ll write about ~300-500GB per day. After about ~40 days using eOS + btrfs, I’m now at 1.71TBW (about 1>x>2 years if I use ext4).

So, can someone point to me, what is the problem? Thanks!.

2

snapshots don’t cause much writing.

The weekly balancing might cause a bit.

400GB/day on a 1TB drive is way too high obviously.

I would try installing iotop and see if anything is writing lots of data. If you don’t see anything obvious, try installing the LTS kernel and see if the behavior continues.

Is there are reason you haven’t updated your system in so long?

3

Thanks for the reply @dalto. I’ve installed iotop. From brief observation, looks like the excessive write mainly from kworker/u64-btrfs-endif-write, kworker/u64-event-unbound that erratic when I open browser like chrome; then upowerd, and btrfs-transaction.

I’ve updated to newest kernel, 6.6.3 and the problem still persist. Oh, and I’m not updated for so long because I’m new to btrfs and want to monitor how the system behave.

4

Can you see if you see the same issues on the LTS kernel?

5

I’ve tried using linux-lts. But the problem persist. At 1.72TBW now :sweat_smile:

6

Are these two subvolumes set to nodatacow ? Also are you running qemu check /var/lib/libvirt/images as well. If not your VM’s and Containers are likely merrily writing away, and trashing your drive.

7

Things will jump to the top a lot but sometimes they are writing tiny amounts of data.

If you run sudo -iotop -ao and leave it running it will show accumulated usage and give you a better idea if there is something writing tons of data.

8

Yes, @portables and @machines ought to set to NO_COW; If I remember correctly, it’s created either by btrfs-assistant or btrfsmaintenance. Or maybe snapper.
And yes, I’m using qemu, but all my images is on different disk (hdd). No image at all at /var/lib/libvirt/images.
My guess is google-chrome or firefox catching trasing my drive.

9

For comparative info, I am writing about 150GB/day not using btrfs. Of course, our usage is probably totally different so that is just a point of reference.

My browsers alone are on pace to write about 15GB/day. That is when they are mostly idle with a lot of tabs open. The chromium browsers seem to be writing more.

I believe those are created by systemd.

10

Noted. This is screenshoot from running sudo iotop -ao for about ~5 minutes. Just opening google chrome, two konsole, and dolphine.

Screenshot_20231203_233432
Screenshot_20231203_233432689×847 152 KB

Thanks for clearing the @portables and @machines.
I’m using google chrome a lot. So I expect around ~10GB/day from there. But 400GB/day is quite insane for my use cases.

I’m planning to go back to ext4. Last time using ext4, to used up ~1.7TB take close to 2 years IIRC.

11

I would manually verify with the lsattr command as different distros have been known to have different defaults.

Also for comparison the nvme drive in my laptop is just over 1 year old and sitting at total 12.54 TB written.

12

It’s not required to check that directories as it’s empty.
Well, 12.54TB is quite normal I think. If I’m still using btrfs, I think I’ll be at ~110TBW at 1 years at minimum and ~185TBW at maximum.

13

I’ve back to ext4 for now, and all looks good. Use about ~10GB per day now. I’ll experiment with above setup using my laptop. Curious as why it behave like that.
Thanks everyone.