Boot Times sporatically changing when loadinig Kernel AND system hangs with Watchdog error on shutdown

Hi,

I’ve been trying to determine why boot times have been sporadically changing (from cold boot) after I changed:

  • Upgraded my NVME from the original Samsung 256 GB to a WD 750 SE 1 Gb, where I cloned the original drive to the new one.
  • Updated the system with all updates on 11-16-2021 including the LTS and ZEN kernel.

Symptoms:

  • The Kernel from cold boot will appear to pause after OS is chosen in GRUB Menu sporadically for upwards of 24 seconds compared to normally completing in 4 seconds.
  • The same symptoms occur whether I use the latest LTS or ZEN Kernels.
  • The system on shutdown will show it has reached the shutdown state but then does not fully power stopping at Watchdog did not stop message. I have to force it down with the power button (usually wait 5 minutes to see if it will shutdown itself.
  • The system bios is at its latest version. This has not changed for months.

The below timings I captured after shutting down the system about 6 times in an hour trying to troubleheshoot the cause.

Slow:
systemd-analyze
Startup finished in 12.335s (firmware) + 6.167s (loader) + 22.418s (kernel) + 6.105s (userspace) = 47.027s
graphical.target reached after 5.404s in userspace

Fast:
systemd-analyze
Startup finished in 10.699s (firmware) + 3.265s (loader) + 1.621s (kernel) + 5.018s (userspace) = 20.605s
graphical.target reached after 4.386s in userspace

Example of the Watchdog error when shutting down and system hangs:

watchdog error

Example of a Fast boot LTS:

systemd-analyze
Startup finished in 10.699s (firmware) + 3.265s (loader) + 1.621s (kernel) + 5.018s (userspace) = 20.605s
graphical.target reached after 4.386s in userspace

systemd-analyze blame
3.629s apparmor.service
2.784s dev-nvme0n1p5.device
2.675s systemd-random-seed.service
 625ms tlp.service
 289ms ufw.service
 259ms unbound.service
 169ms systemd-journal-flush.service
 168ms systemd-udev-trigger.service
 156ms systemd-binfmt.service
 149ms lvm2-monitor.service
 141ms accounts-daemon.service
 133ms systemd-modules-load.service
 117ms upower.service
 116ms systemd-logind.service
 114ms user@1000.service
 114ms systemd-hostnamed.service
 106ms polkit.service
 103ms systemd-udevd.service
 100ms systemd-tmpfiles-setup-dev.service
  82ms modprobe@fuse.service
  80ms modprobe@drm.service
  78ms modprobe@configfs.service
  72ms systemd-remount-fs.service
  71ms usbguard.service
  64ms systemd-localed.service
  64ms systemd-tmpfiles-setup.service
  61ms sys-kernel-config.mount
  61ms systemd-journald.service
  60ms kmod-static-nodes.service
  59ms sys-fs-fuse-connections.mount
  57ms sys-kernel-tracing.mount
  56ms sys-kernel-debug.mount
  52ms proc-sys-fs-binfmt_misc.mount
  52ms NetworkManager.service
  51ms dev-mqueue.mount
  50ms udisks2.service
  47ms swapfile.swap
  44ms power-profiles-daemon.service
  40ms boot-efi.mount
  33ms dev-hugepages.mount
  33ms gdm.service
  32ms alsa-restore.service
  29ms systemd-fsck@dev-disk-by\x2duuid-E028\x2d05A2.service
  24ms bluetooth.service
  19ms colord.service
  12ms systemd-user-sessions.service
  11ms wpa_supplicant.service
   9ms tmp.mount
   8ms systemd-sysctl.service
   7ms systemd-update-utmp.service
   7ms systemd-backlight@backlight:intel_backlight.service
   6ms systemd-backlight@leds:tpacpi::kbd_backlight.service
   6ms user-runtime-dir@1000.service
   6ms rtkit-daemon.service
   2ms snapd.socket
   1ms nordvpnd.socket

Example of a Slow Boot with LTS:

systemd-analyze
Startup finished in 12.335s (firmware) + 6.167s (loader) + 22.418s (kernel) + 6.105s (userspace) = 47.027s
graphical.target reached after 5.404s in userspace

systemd-analyze blame
4.528s apparmor.service
2.357s dev-nvme0n1p5.device
2.318s systemd-random-seed.service
 664ms tlp.service
 314ms ufw.service
 311ms bolt.service
 268ms unbound.service
 176ms upower.service
 158ms user@1000.service
 156ms systemd-hostnamed.service
 144ms systemd-journal-flush.service
 140ms systemd-modules-load.service
 139ms systemd-logind.service
 131ms boot-efi.mount
 126ms systemd-udev-trigger.service
 117ms usbguard.service
 116ms systemd-tmpfiles-setup.service
 113ms systemd-journald.service
 111ms polkit.service
 107ms lvm2-monitor.service
  93ms systemd-udevd.service
  84ms systemd-binfmt.service
  83ms power-profiles-daemon.service
  82ms swapfile.swap
  78ms modprobe@drm.service
  76ms systemd-backlight@backlight:intel_backlight.service
  72ms accounts-daemon.service
  65ms udisks2.service
  64ms gdm.service
  61ms NetworkManager.service
  60ms systemd-localed.service
  51ms systemd-fsck@dev-disk-by\x2duuid-E028\x2d05A2.service
  49ms systemd-tmpfiles-setup-dev.service
  40ms modprobe@fuse.service
  39ms modprobe@configfs.service
  38ms kmod-static-nodes.service
  37ms sys-fs-fuse-connections.mount
  36ms sys-kernel-tracing.mount
  35ms sys-kernel-debug.mount
  33ms dev-mqueue.mount
  31ms proc-sys-fs-binfmt_misc.mount
  31ms dev-hugepages.mount
  28ms systemd-remount-fs.service
  26ms sys-kernel-config.mount
  21ms tmp.mount
  21ms systemd-user-sessions.service
  17ms colord.service
  17ms wpa_supplicant.service
  16ms bluetooth.service
  14ms snapd.socket
  14ms systemd-backlight@leds:tpacpi::kbd_backlight.service
  12ms user-runtime-dir@1000.service
  10ms systemd-sysctl.service
   9ms systemd-update-utmp.service
   5ms rtkit-daemon.service
   5ms alsa-restore.service
   1ms nordvpnd.socket

My systems configuration:

inxi -b
System:
  Host: sol Kernel: 5.10.79-1-lts x86_64 bits: 64 Desktop: GNOME 41.1
  Distro: EndeavourOS
Machine:
  Type: Laptop System: LENOVO product: 20HG**** v: ThinkPad T470s
  serial: <superuser required>
  Mobo: LENOVO model: 20HGS**** v: ******** WIN serial: <superuser required>
  UEFI: LENOVO v: N1WET62W (1.41 ) date: 04/27/2021
Battery:
  ID-1: BAT0 charge: 7.0 Wh (34.8%) condition: 20.1/21.3 Wh (94.4%) volts: 11.4
  min: 11.4
  ID-2: BAT1 charge: 18.8 Wh (99.5%) condition: 18.9/26.1 Wh (72.4%)
CPU:
  Info: Dual Core Intel Core i7-7600U [MT MCP] speed: 1100 MHz min/max: 400/3900 MHz
Graphics:
  Device-1: Intel HD Graphics 620 driver: i915 v: kernel
  Device-2: Acer Integrated Camera type: USB driver: uvcvideo
  Display: wayland server: X.Org 1.21.1.3 driver: loaded: intel
  unloaded: fbdev,modesetting,vesa resolution: 2048x1152~60Hz
  OpenGL: renderer: Mesa Intel HD Graphics 620 (KBL GT2) v: 4.6 Mesa 21.2.5
Network:
  Device-1: Intel Wireless 8265 / 8275 driver: iwlwifi
Drives:
  Local Storage: total: 931.51 GiB used: 83.4 GiB (9.0%)
Info:
  Processes: 192 Uptime: 35m Memory: 15.51 GiB used: 1.96 GiB (12.7%) Shell: Zsh
  inxi: 3.3.07