Problem with the systemd fstrim service

I have a 1TB SSD in my main laptop. When I installed EOS, I enabled the fstrim timer. That was several months ago. It obviously worked - first time the laptop was booted on a Monday morning, it would run the discard on the SSD. Took 3 or 4 minutes to complete and I got into the habit of doing something else while it ran. As a result, I didn’t notice that sometime recently, it started taking no time at all to run - there would be a few flickers from the HDD access LED and it would be done.

When I noticed the sudden speed up, I checked the journal, which seems to show the command completing without any problems:

Mar 30 07:32:50 rigel kernel: audit: type=1130 audit(1585549970.836:24): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 30 07:32:50 rigel kernel: audit: type=1131 audit(1585549970.836:25): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 30 07:32:50 rigel audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 30 07:32:50 rigel audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 30 07:32:50 rigel systemd[1]: fstrim.service: Succeeded.
Mar 30 07:32:50 rigel systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.

Being suspicious, I ran the command from terminal. Which had quite different results. Instead of the near instantaneous completion, it ran for around 3 minutes 40 seconds, and produced the following output:

[paulm@rigel ~]$ sudo /usr/bin/fstrim --fstab --verbose --quiet
[sudo] password for paulm: 
/mnt/win: 207.4 GiB (222639628288 bytes) trimmed on /dev/sda2
/mnt/sda08: 15 GiB (16045195264 bytes) trimmed on /dev/sda8
/mnt/sda07: 14.4 GiB (15471661056 bytes) trimmed on /dev/sda7
/home: 45.3 GiB (48599416832 bytes) trimmed on /dev/sda5
/: 12 GiB (12908900352 bytes) trimmed on /dev/sda6

So it seems that the timer controlled systemd service isn’t doing much, certainly not really doing what it is supposed to.

Has anyone else seen this problem? Any suggestions as to what the problem is?

Interesting, I had similar results. Seems that this fstrim systemd service is not really working.

And command

   systemctl status fstrim.timer

quite different schedule than what it is configured to be…

1 Like

What does systemctl status fstrim.service show

I had changed the schedule to monhly instead of the default that is weekly.
The status shows it will run it about daily…
So there must be some other setting somewhere that affects this.

But now I already stopped and disabled the timer. It is easy to run manually, for example the first day of each month.

1 Like

Well, that narrows it down a bit - I wasn’t sure if I’d done something to screw it up. Trouble is, since I didn’t notice when the problem started, I don’t know what update upset it (I’m assuming that the problem has been caused by an update).

In my case, systemctl status fstrim.timer is correct:

[paulm@rigel ~]$ systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
     Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: disabled)
     Active: active (waiting) since Tue 2020-03-31 16:15:42 BST; 4h 47min ago
    Trigger: Mon 2020-04-06 00:00:00 BST; 5 days left
   Triggers: ● fstrim.service
       Docs: man:fstrim

I have no idea how to trace the problem.

:slight_smile: I was thinking of setting it up as a cron job for startup on Monday mornings… Just lazy I guess.

It could be one of the systemd updates, but that’s just a guess.
They’ve had 7 updates only in this month.

If you will, package pahis can easily show the package history.
E.g.

  pahis systemd  # shows all systemd package events
  pahis          # shows all package events
2 Likes

There certainly have been a number of systemd upgrades recently. I should have the last two or three in one of my systems at least. I can downgrade and see if I can work out where the problem occurred. Only trouble is the time it would take to test.

I’ve not encountered pahis. Is it part of another package - haven’t found it so far…

It is one of Endeavour packages, can be installed with
sudo pacman -S pahis

Odd. I assumed that was the case, but attempting to download it resulted in a target not found error. I’ll have to have a look at the mirrolist setup - on a different (pure Arch) system at the moment.

Same here. Now this is intriguing. I wonder if it’s working as it should, but just something we don’t understand. At least I don’t understand.

[ben@brick ~]$  systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
     Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: disabled)
     Active: active (waiting) since Tue 2020-03-31 10:43:10 EDT; 8h ago
    Trigger: Mon 2020-04-06 00:00:00 EDT; 5 days left
   Triggers: ● fstrim.service
       Docs: man:fstrim

Mar 31 10:43:10 brick systemd[1]: Started Discard unused blocks once a week.

[ben@brick ~]$ sudo /usr/bin/fstrim --fstab --verbose --quiet
[sudo] password for ben: 
/mnt/data: 446 GiB (478829420544 bytes) trimmed on /dev/sdb1
/home: 681.8 GiB (732082323456 bytes) trimmed on /dev/sda3
/: 37.4 GiB (40137379840 bytes) trimmed on /dev/sda2
/boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sda1
[ben@brick ~]$ 

@toothandnail do the following and see what it gives.

journalctl -u fstrim

I think mine may have quit sometime after Jan 20, 2020.

[ben@brick ~]$ journalctl -u fstrim
-- Logs begin at Sat 2019-12-14 16:15:21 EST, end at Tue 2020-03-31 20:57:44 EDT. --
Dec 16 08:39:56 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Dec 16 08:40:06 brick fstrim[672]: /mnt/data: 448.3 GiB (481376591872 bytes) trimmed on /dev/sd>
Dec 16 08:40:06 brick fstrim[672]: /: 40.5 GiB (43524288512 bytes) trimmed on /dev/sda2
Dec 16 08:40:06 brick fstrim[672]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sda1
Dec 16 08:40:06 brick systemd[1]: fstrim.service: Succeeded.
Dec 16 08:40:06 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Dec 23 00:00:01 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Dec 23 00:00:12 brick fstrim[3687]: /mnt/data: 435.4 GiB (467453173760 bytes) trimmed on /dev/s>
Dec 23 00:00:12 brick fstrim[3687]: /: 40.1 GiB (43100913664 bytes) trimmed on /dev/sda2
Dec 23 00:00:12 brick fstrim[3687]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sda1
Dec 23 00:00:12 brick systemd[1]: fstrim.service: Succeeded.
Dec 23 00:00:12 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Dec 30 08:19:29 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Dec 30 08:19:43 brick fstrim[690]: /mnt/data: 448.3 GiB (481376587776 bytes) trimmed on /dev/sd>
Dec 30 08:19:43 brick fstrim[690]: /: 39.9 GiB (42862817280 bytes) trimmed on /dev/sdb2
Dec 30 08:19:43 brick fstrim[690]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sdb1
Dec 30 08:19:43 brick systemd[1]: fstrim.service: Succeeded.
Dec 30 08:19:43 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Jan 06 08:33:19 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Jan 06 08:33:30 brick fstrim[697]: /mnt/data: 448.3 GiB (481376587776 bytes) trimmed on /dev/sd>
Jan 06 08:33:30 brick fstrim[697]: /: 39.6 GiB (42458206208 bytes) trimmed on /dev/sdb2
Jan 06 08:33:30 brick fstrim[697]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sdb1
Jan 06 08:33:30 brick systemd[1]: fstrim.service: Succeeded.
Jan 06 08:33:30 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Jan 13 08:47:00 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Jan 13 08:47:11 brick fstrim[687]: /mnt/data: 446 GiB (478831054848 bytes) trimmed on /dev/sdb1
Jan 13 08:47:11 brick fstrim[687]: /: 39.2 GiB (42039775232 bytes) trimmed on /dev/sda2
Jan 13 08:47:11 brick fstrim[687]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sda1
Jan 13 08:47:11 brick systemd[1]: fstrim.service: Succeeded.
Jan 13 08:47:11 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Jan 20 08:57:08 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Jan 20 08:57:19 brick fstrim[709]: /mnt/data: 446 GiB (478831054848 bytes) trimmed on /dev/sdb1
Jan 20 08:57:19 brick fstrim[709]: /: 38.7 GiB (41492598784 bytes) trimmed on /dev/sda2
Jan 20 08:57:19 brick fstrim[709]: /boot/efi: 498.8 MiB (522969088 bytes) trimmed on /dev/sda1
Jan 20 08:57:19 brick systemd[1]: fstrim.service: Succeeded.
Jan 20 08:57:19 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Jan 27 08:40:40 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Jan 27 08:40:40 brick systemd[1]: fstrim.service: Succeeded.
Jan 27 08:40:40 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Feb 03 08:26:44 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Feb 03 08:26:44 brick systemd[1]: fstrim.service: Succeeded.
Feb 03 08:26:44 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Feb 10 06:39:05 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Feb 10 06:39:05 brick systemd[1]: fstrim.service: Succeeded.
Feb 10 06:39:05 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Feb 17 07:39:52 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Feb 17 07:39:52 brick systemd[1]: fstrim.service: Succeeded.
Feb 17 07:39:52 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Feb 24 09:29:40 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Feb 24 09:29:40 brick systemd[1]: fstrim.service: Succeeded.
Feb 24 09:29:40 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Mar 02 09:46:43 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Mar 02 09:46:44 brick systemd[1]: fstrim.service: Succeeded.
Mar 02 09:46:44 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Mar 09 08:38:43 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Mar 09 08:38:44 brick systemd[1]: fstrim.service: Succeeded.
Mar 09 08:38:44 brick systemd[1]: Started Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Mar 16 00:00:16 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Mar 16 00:00:17 brick systemd[1]: fstrim.service: Succeeded.
Mar 16 00:00:17 brick systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Mar 23 10:29:52 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Mar 23 10:29:52 brick systemd[1]: fstrim.service: Succeeded.
Mar 23 10:29:52 brick systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
-- Reboot --
Mar 30 09:20:59 brick systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab>
Mar 30 09:20:59 brick systemd[1]: fstrim.service: Succeeded.
Mar 30 09:20:59 brick systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
lines 51-82/82 (END)

Also see:

and

https://bbs.archlinux.org/viewtopic.php?id=252451

EDIT: The suggestion is to edit /usr/lib/systemd/system/fstrim.service
and comment out

The suggestion is to copy the file /usr/lib/systemd/system/fstrim.service to /etc/systemd/system and comment out

ProtectSystem=strict

and

ProtectHome=read-only

Commenting out ProtectHome may be necessary if you have a separate /home partition.

Thanks @dalto for the copy suggestion. See his suggestion later in this thread.

Thanks for the tip! I’ll try it later today.

EDIT: it seems to be a tricky thing! More info:

   man systemd.exec

I see what you mean. Back to the books for a more secure answer. In the meantime I’m leaving ProtectSystem=strict commented out on one of my computers to see what happens.

Edit:I may comment out ProtectHome=read-only too.

Accoding to:

both were removed in a commit on Feb 4. But I still have both in the current fstrim.service. Now I’m really confused.

1 Like

Instead of editing that file, it may be better to copy it to /etc/systemd/system/fstrim.service and modify it there. Otherwise, every time the package updates, it will overwrite your changes.

3 Likes

Good point. I had overlooked that fact. I’ll also edit my original note above to reflect this.