Possible Kernel/Grub/systemd Shutdown Error Message

Stagger_Lee · November 13, 2021, 4:18pm

You do realize that this is a service that runs at shutdown?

Description=Remove nvidia modules before shutdown

Manually starting it isn’t going to remove modules from the system that is still in use. It will be triggered at shutdown.

The service is enabled; it is currently inactive because you’re not shutting down.

As to this:

The unit file, source configuration file or drop-ins of nvidia-unload-shutdown.service changed on disk. Run 'systemctl daemon-reload' to reload units.

That’s because you created a drop-in file that changed the configuration of the unit; if you don’t reboot, you can load the new configuration by this command.

Scotty_Trees · November 13, 2021, 4:29pm

I’ve run sudo nano /etc/systemd/system/nvidia-unload-shutdown.service

and added:

  GNU nano 5.9   /etc/systemd/system/nvidia-unload-shutdown.service              
# /etc/systemd/system/nvidia-unload-shutdown.service
[Unit]
Description=Remove nvidia modules before shutdown
DefaultDependencies=no
After=gdm.service

[Service]
Type=oneshot
ExecStart=/usr/bin/modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia

[Install]
WantedBy=shutdown.target

Then I’ve run:
sudo systemctl enable nvidia-unload-shutdown.service

Follwed by:
systemctl daemon-reload

And did a few restarts after that, but I still get the FAILED error when I shutdown:

[FAILED] Failed to start Remove nvidia modules before shutdown.
[     40.550004] watchdog: watchdog0: watchdog did not stop!
[     40.719409] watchdog: watchdog0: watchdog did not stop!
[     40.769894] sd-unmoun[1704]: Failed to unmount /oldroot: Device or resource busy
[     40.770754] sd-unomun[1785]: Failed to unmount /oldroot/sys: Device or resource busy
[     40.771918] shutdown[1]: Failed to finalize file systems, ignoring.

That’s just up to speed of where I’m currently at.

Stagger_Lee · November 13, 2021, 4:45pm

Since it would appear that the steps you have taken aren’t helping in any way (was there any indication that this was a problem with nvidia modules?), you might want to consider undoing what you’ve done, reboot twice, and then check your journal from the previous boot to try and identify what your issue actually is:

journalctl -e -b-1

If you want to just get the last 100 lines, you can use the -n option with a number:

journalctl -e -n100 -b-1

Scotty_Trees · November 13, 2021, 9:55pm

What’s the proper way to go about that? Never had to do something like this before

Stagger_Lee · November 13, 2021, 10:04pm

Just do what you have done, only backwards .

Stop and disable the service:

systemctl disable --now nvidia-unload-shutdown.service

Remove the file you created (or remove the lines you added to an existing file, if that was the case).

Then you can either use systemctl daemon-reload to reload the original (default) config, or reboot. Personally I would reboot. Then reboot again, just to be sure that no config changes remained lingering after you stopped/disabled them, and then check your journal.

Scotty_Trees · November 13, 2021, 10:19pm

[scott@endeavourOS ~]$ systemctl disable --now nvidia-unload-shutdown.service
Removed /etc/systemd/system/shutdown.target.wants/nvidia-unload-shutdown.service.
[scott@endeavourOS ~]$

I created this file sudo nano /etc/systemd/system/nvidia-unload-shutdown.service so is it better to delete the lines in the file and save it so it’s basically just blank or is there a certain remove command to remove the entire file itself? Thanks for the help, btw

Stagger_Lee · November 13, 2021, 10:22pm

I would delete it, myself, but you could leave it blank if you want.

fbodymechanic · November 13, 2021, 10:22pm

man rm

The answer to Arch knowledge is a path walked.

Scotty_Trees · November 13, 2021, 10:27pm

Thank you, I just checked man rm (and added it to my common cmd list), I see there’s a few options to choose from but I think it I’d probably be just fine with rm for this.

So sudo rm /etc/systemd/system/nvidia-unload-shutdown.service would be acceptable?

fbodymechanic · November 13, 2021, 10:31pm

man _________

Man = manual. It’s like passing the help flag and works with all sorts of commands to bring up their how to manual.

Post the terminal output. Did it work for you or does it throw an error? It looks like it should work, but does it?

Scotty_Trees · November 13, 2021, 10:33pm

[scott@endeavourOS ~]$ sudo rm /etc/systemd/system/nvidia-unload-shutdown.service
[sudo] password for scott: 
[scott@endeavourOS ~]$

Edit: Hands Bender more beer

fbodymechanic · November 13, 2021, 10:35pm

Looks good then.

If there’s only one file it should work, if you have others in it as a directory, I don’t believe it will rm it without passing the -r flag as well.

You’re on the next step to self sufficiency.

Scotty_Trees · November 13, 2021, 10:42pm

OK so I’ve done:

systemctl disable --now nvidia-unload-shutdown.service
sudo rm /etc/systemd/system/nvidia-unload-shutdown.service

I didn’t run systemctl daemon-reload but instead rebooted twice, then:

[scott@endeavourOS ~]$ journalctl -e -n100 -b-1
Nov 13 17:38:13 endeavourOS systemd[1]: Unmounting /tmp...
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS systemd[1]: boot-efi.mount: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Unmounted /boot/efi.
Nov 13 17:38:13 endeavourOS systemd[1]: systemd-fsck@dev-disk-by\x2duuid-BB5D\x2d4A9F.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped File System Check on /dev/disk/by-uuid/BB5D-4A9F.
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-BB5D\x2d4A9F comm="systemd" exe="/usr/lib/systemd/sys>
Nov 13 17:38:13 endeavourOS systemd[1]: Removed slice Slice /system/systemd-fsck.
Nov 13 17:38:13 endeavourOS systemd[1]: run-credentials-systemd\x2dsysusers.service.mount: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Unmounted /run/credentials/systemd-sysusers.service.
Nov 13 17:38:13 endeavourOS systemd[1]: tmp.mount: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Unmounted /tmp.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped target Preparation for Local File Systems.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped target Swaps.
Nov 13 17:38:13 endeavourOS systemd[1]: Deactivating swap /swapfile...
Nov 13 17:38:13 endeavourOS systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Nov 13 17:38:13 endeavourOS systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped Create Static Device Nodes in /dev.
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? ad>
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-sysusers comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? termi>
Nov 13 17:38:13 endeavourOS systemd[1]: systemd-sysusers.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped Create System Users.
Nov 13 17:38:13 endeavourOS systemd[1]: swapfile.swap: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Deactivated swap /swapfile.
Nov 13 17:38:13 endeavourOS systemd[1]: Reached target Unmount All Filesystems.
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? ter>
Nov 13 17:38:13 endeavourOS systemd[1]: systemd-remount-fs.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped Remount Root and Kernel File Systems.
Nov 13 17:38:13 endeavourOS systemd[1]: lvm2-monitor.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lvm2-monitor comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=>
Nov 13 17:38:13 endeavourOS systemd[1]: Reached target System Shutdown.
Nov 13 17:38:13 endeavourOS systemd[1]: Reached target Late Shutdown Services.
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? termin>
Nov 13 17:38:13 endeavourOS audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? termina>
Nov 13 17:38:13 endeavourOS systemd[1]: systemd-reboot.service: Deactivated successfully.
Nov 13 17:38:13 endeavourOS systemd[1]: Finished System Reboot.
Nov 13 17:38:13 endeavourOS systemd[1]: Reached target System Reboot.
Nov 13 17:38:13 endeavourOS systemd[1]: Shutting down.
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS audit: BPF prog-id=0 op=UNLOAD
Nov 13 17:38:13 endeavourOS systemd[1]: Using hardware watchdog 'iTCO_wdt', version 0, device /dev/watchdog
Nov 13 17:38:13 endeavourOS systemd[1]: Set hardware watchdog to 10min.
Nov 13 17:38:13 endeavourOS kernel: watchdog: watchdog0: watchdog did not stop!
Nov 13 17:38:13 endeavourOS systemd-shutdown[1]: Syncing filesystems and block devices.
Nov 13 17:38:13 endeavourOS systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Nov 13 17:38:13 endeavourOS systemd-journald[254]: Journal stopped
lines 51-101/101 (END)

Edit: I run my terminal in window mode most times, I noticed the output cut a bunch of messages off, so I went full screen, ran the command again, so it’s easier to read now.

Stagger_Lee · November 13, 2021, 10:49pm

Did your reboot hang?

I am not seeing any references to the errors you first mentioned in the OP related to /oldroot not unmounting.

Scotty_Trees · November 13, 2021, 10:53pm

I’m still able to reboot just fine, but I still see these messages show, specifically the last 3 lines only started to display after a systemd update. The watchdog messages I’m not concerned about. It’s a random issues I’ve seen people report earlier this year and last year and ever before that, but for whatever reason it only started to show up for me this year.

[     40.550004] watchdog: watchdog0: watchdog did not stop!
[     40.719409] watchdog: watchdog0: watchdog did not stop!
[     40.769894] sd-unmoun[1704]: Failed to unmount /oldroot: Device or resource busy
[     40.770754] sd-unomun[1785]: Failed to unmount /oldroot/sys: Device or resource busy
[     40.771918] shutdown[1]: Failed to finalize file systems, ignoring.

Stagger_Lee · November 13, 2021, 10:59pm

Yeah, systemd just updated yesterday, and /oldroot is a systemd thing. There have been several times in the past few years that systemd has had bugs involving the unmounting of /oldroot.

If you’re not seeing any specific issues, if it were me, I would wait and see if there is a systemd update in the next few days. I would suspect a bug that has been around in the past has shown its head again. That’s just what I would do, feel free to do whatever you feel is best for your system.

If it persists, and there are no bug reports on systemd’s github site, then you might want to dig in deeper.

Scotty_Trees · November 13, 2021, 11:30pm

Yeah that’s some pretty solid advice, thank you. There’s been maybe one or two systemd updates since then, but it has’t removed these messages. Luckily I haven’t noticed any issues at all other than these shutdown messages.

Just checked out the systemd github, they’ve got over 1.5k issues at the moment. I’ll see if my issue has already been reported recently and if not I’ll create a new issue if need be and just go from there. In the meantime, thanks again for all your help, very much appreciate it and I learned a little bit more today so thank you

Stagger_Lee · November 13, 2021, 11:38pm

The messages suggest that your / partition may not be unmounted cleanly before shutdown. This could prove problematic, causing corruption; assuming it is a temporary issue, though, I personally wouldn’t sweat it. If you get fsck warnings, or you see a list of orphaned inodes on your boot, then you may want to dig deeper, like, right away .

I have learned that systemd bugs generally tend to get sorted out pretty quickly, and again, if it were me, I would assume that the fact that these warnings first appeared after a systemd update means it’s probably a systemd bug.

Scotty_Trees · November 13, 2021, 11:45pm

I’ll for sure keep an eye on any boot messages like that and let y’all know. I think I’ll wait for the next systemd update and if I still see it I’ll report it to them upstream. Thanks again!

kagetora13 · November 17, 2021, 8:23pm

@Stagger_Lee

Hi Guys,

I did an update last night of my system and my system is showing two symptoms which maybe simlar to yours:

Kernel sporadically taking over 20 seconds to loads (vs under 5s normally).
I’m also seeing “Watchdog did not stop” messages on shutdown if I have a slow boot up time.

I posted my notes here just in case you guys hear anything new: