How to best notify yourself about failing systemd services?

General system Newbie
1

What’s the best way to notify yourself about failing systemd user and system services running a desktop environment (e.g. plasma)?

Currently I have “systemctl list-units --failed” as my console greater, which is sufficient, but also very clumsy and unreliant.

2

That’s an xyproblem. Systemd’s reports about something is wrong (failing may not be crucial) are there for when you want to troubleshoot something, or you suspect something goes wrong.
So, depending on what you need, you have different ways to look for info. :person_shrugging:

That’s an adequate report for very worried users, IMHO :wink: :rofl: .

3

Yes and no. Of course I don’t care about systemd failing, but the job that systemd was supposed to facilitate. Systemd is the messenger and manger of those jobs. It starts the jobs and possesses all the information about the outcome.

If a systemd job fails then something that was supposed to happen didn’t. This isn’t suspected troubleshooting but being informed about an actual failure. The decision if it was a crucial failure should then be up to the user. But the user has to know to make that decision.

That said, I don’t really care about the philosophical discussion. Let’s assume that a user’s system is sound and shouldn’t have any systemd started processes running into an error. What is the most practical way to be informed that an error occurred?

4

There is journalctl-desktop-notification-git from AUR, it notifies you when any error message appears in journalctl log.

5

Thanks, I’ll check it out. :+1:

6

I’ve implemented a tool specifically for that: Salmon: a tray icon and desktop notifications for failing systemd services and anything else.

I wanted something much simpler than a traditional monitoring stack: if an important service fails, make a tray icon blink and annoy me until I deal with it.

salmon_demo

Originally implemented the bare minimum in 2021 and sat on it for a few years using it just for myself, now finally polished it enough for v1.0.0, and so releasing it.

Salmon can:

  • Monitor systemd services
  • Run arbitrary command-based health checks
  • Combine incidents from multiple machines into one desktop app
  • Show tray icon, desktop notifications and a small local web UI
  • Snooze incidents temporarily

It consists of a small monitoring service installed on each Linux machine and a desktop app that collects their incidents.

For remote machines, it supports managed SSH tunnels or TLS+bearer authentication.

GitHub repo: https://github.com/dimonomid/salmon