"PSA: Linux 5.16 has major regression in btrfs causing extreme IO load"

[mshmm]

Welcome @mil

[jonathon]

@pebcak

That is exactly what is happening. The manual defrag getting stuck on a tiny file of ~ 1 byte.

Heh.

@jonathon

There’s some good detail about the increased IO in the kernel mailing list thread.

There are two main issues:

1. Defragmenting 1 byte files will cause an infinite loop;

[pebcak]

Yeah! But seeing that with my own eyes was awesome!

And sharing the experience

[mil]

I agree, this is exactly what we were told all these years so I do not see how btrfs will need defraging on ssds.

Additionally in the manual it says:

autodefrag, noautodefrag
(since: 3.0, default: off)

which is even more strange, so if the default is off, why does the installer defaults to enabling it on ssds.

Btw…thank you @mshmm and @pebcak for the welcome…

[magtuired]

Ah man. I was blissfully unaware of all this. I have to admit I don’t really monitor my stats unless something is obviously up…maybe that’s something I should do more often?

I have no idea whether the impact of this for me was as catastrophic as the Reddit poster implies it can be, but I had a look at my SSD with smartctl. For some reason I couldn’t get the Total Bytes Written value (TBW), so I had to convert the ‘Total LBAs Written’ value via a calculator.

Commands I needed to get data for the calculation:

sudo smartctl -A /dev/sdX --all | grep "Sector Size"
sudo smartctl -A /dev/sdX --all | grep "Total_LBAs_Written"

Looks like I’m at at 60.65 TB. Had this drive for around…I don’t know…5 years? Don’t know if that’s good or bad. My SSD’s warranty covers 150TB for this model (250GB) so it looks like it should be alright for a while yet.

[dalto]

@mil

which is even more strange, so if the default is off, why does the installer defaults to enabling it on ssds

Calamares, the installer we use doesn’t have the option to specify HDD-only options so there isn’t currently a way to say enable that option on HDDs but not SSDs. That functionality has been built and should be available in a future release.

[mil]

@dalto

Calamares, the installer we use doesn’t have the option to specify HDD-only options so there isn’t currently a way to say enable that option on HDDs but not SSDs. That functionality has been built and should be available in a future release.

Thanks that is good to know. The reason I was questioning the “wisdom” of the installer is just in case “it knows something more than I do” and I shouldn’t argue with its decisions. Obviously that is not that case, so I am fine with removing autodefrag from the fstab config

[anon49550872]

Well, to get a more concrete baseline, I’m now running 5.16 again, but without the autodefrag option, and it seems a lot better than running LTS with the autodefrag enabled.
If defrag for SSD is ‘controversial’, I personally suggest users that are using SSD to remove that option completely, or to set the noautodefrag option instead. But I’m no specialist, this is just my observation.

[Trekkie00]

Maybe Arch News?
I never digged into how this works and who maintains it.

[ajgringo619]

On my newest drive - SanDisk SDSSDH3 2T00 - smartctl uses the label "Total_Writes_GiB` instead of “Total_LBAs_Written”.

[anon49550872]

There is something wrong with my smartcl…

Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       50378
Error 52032 occurred at disk power-on lifetime: 38473 hours (1603 days + 1 hours)

This is not possible, 4,3 years and only that LBAs_Written?

[ajgringo619]

My SanDisk SDSSDH3512G, which is at least 3 years old now, is sitting at 16419. It’s been my main OS drive - Linux and Windows/Linux - since I bought it. Now my data drive (20 months old) is at 62993; to be expected, certainly.

[anon49550872]

uow, so my drive will probably die due to lifespan than to writing cycles…

[ajgringo619]

SSD technology has come a long way in a relatively short period of time.

[Stephane]

Btrfs was rewritted for this kernel series ,
there is still patches coming for btrfs
see https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.16.3
and find Btrfs …

[jonathon]

Fixes for this hit 5.16.5-rc; they are coming.

[damian101]

I can reproduce this issue with option autodefrag, but not with options autodefrag,ssd.
In my case, multiple Gigabytes were silently written per minute, so this is definitely a very serious issue.
Removing autodefrag solved it, but adding ssd seems to do it as well for me.
I’m using a SATA SSD.

Is someone absolutely sure that autodefrag,ssd caused the issue for them?

I confirmed the issue by looking at smart data changes.

[dalto]

btrfs autodetects ssd even if you don’t set it explicitly.

[petsam]

@dalto

btrfs autodetects ssd

It says so, but does it?

@damian101

adding ssd seems to do it as well for me.

Then @damian101 confirm it’s not an illusion, please.
When testing btrfs options, check active flags with

mount | grep btrfs

[damian101]

The ssd mount option is indeed set automatically, so my assumption that it fixes the issue makes no sense. @dalto
For some reason I can’t reproduce the issue anymore, but it definitely was there. Multiple gigabytes were written to my SSD per minute, while the system monitor showed no such write activity for any process.

Before I rebooted with autodefrag enabled I had not used defrag on my SSD ever, maybe that plays a role. Perhaps defragmentation has to be triggered for the issue to appear. That would be my guess. The ssd mount option making a difference was very likely just a false assumption on my part.