Not quite sure what started this but it was around when glibc updated about a week ago (probably not the actual cause but it started around then) my system all of a sudden is sometimes getting timed out waiting on start jobs for 4 drive UUIDs.
I’ve checked my fstab and everything seems fine. I haven’t even changed it at all recently and it’s been working for months.
I tried booting into a live environment and running fsck on all of my partitions and they all come back clean. I can also boot into my backup Linux Mint install and it boots perfectly every time with a very similar fstab setup.
I’ve reinstalled my kernel and ran dracut-rebuild, as I saw people online saying this worked, but it didn’t help in my case.
The main thing I don’t understand is that this issue only occurs maybe 50% of the time when I boot my Endeavour system. It’s pretty much a coin toss whether it will boot perfectly or if it will get stuck waiting.
Any help with this would be appreciated because I’m stumped as to what’s causing it. I can provide any extra info if needed.
I doubt it’s hardware. In the post I wrote about how I’m able to use other distros like Mint and get reliable boots with a similar fstab and all. I’ll figure out the journal and try to get it posted.
After skimming through it, it seems that my /efi (C729-BEEE) is the first partition to fail. I’m not sure if that’s whats causing the other drives to fail as well, because there’s a lot of errors about dependencies. /efi is the first partition listed in my fstab, so I don’t know if there’s a problem with the efi partition itself or something else causing it to fail. I still have no idea why it’s so inconsistent.
Inconsistent problem is the hardest to diagnose. Inconsistency usually points to hardware problems since the software does not just change and fail on its own, then miraculously fix itself and work at other times.
I actually just noticed that in KDE Partition Manager it complains every time that my 3 host drive partitions aren’t properly aligned. Doing an offline fsck on the 3 partitions doesn’t report any issues and it doesn’t fix it either. Could this be part of the issue?
Maybe, I don’t rightly know about improperly aligned partitions. I have not seen that type of issue in years. I have used gparted, cfdisk, and gdisk and I have not had to worry about partition alignment in ages as those tools just do the proper alignment in my experience. What tool did you use to create the partitions?
May 02 15:09:55 EOS-Desktop systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
May 02 15:09:55 EOS-Desktop systemd[1]: Expecting device /dev/disk/by-uuid/0158000c-99aa-4244-ae45-319d371654e8...
May 02 15:09:55 EOS-Desktop systemd[1]: Expecting device /dev/disk/by-uuid/07fbb757-2065-43c7-958e-f47f868103e8...
May 02 15:09:55 EOS-Desktop systemd[1]: Expecting device /dev/disk/by-uuid/558fb03b-855c-4286-b19c-0777a0ba875b...
May 02 15:09:55 EOS-Desktop systemd[1]: Expecting device /dev/disk/by-uuid/C729-BEEE...
May 02 15:09:55 EOS-Desktop systemd[1]: Expecting device /dev/disk/by-uuid/b999f1d5-4da9-40af-98a7-71953710814c...
May 02 15:09:55 EOS-Desktop systemd[1]: Reached target Local Encrypted Volumes.
...
May 02 15:09:55 EOS-Desktop systemd[1]: Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 02 15:09:55 EOS-Desktop systemd[1]: Reached target Preparation for Local File Systems.
May 02 15:09:55 EOS-Desktop systemd[1]: Virtual Machine and Container Storage (Compatibility) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/machines.raw).
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: Using default interface naming scheme 'v255'.
May 02 15:09:55 EOS-Desktop systemd[1]: Finished Flush Journal to Persistent Storage.
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: /etc/udev/rules.d/51-trezor.rules:11 Unknown group 'plugdev', ignoring.
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: /etc/udev/rules.d/51-trezor.rules:12 Unknown group 'plugdev', ignoring.
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: /etc/udev/rules.d/51-trezor.rules:15 Unknown group 'plugdev', ignoring.
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: /etc/udev/rules.d/51-trezor.rules:16 Unknown group 'plugdev', ignoring.
May 02 15:09:55 EOS-Desktop systemd-udevd[599]: /etc/udev/rules.d/51-trezor.rules:17 Unknown group 'plugdev', ignoring.
...
May 02 15:09:55 EOS-Desktop systemd[1]: Mounting /mnt/Game-NVME-01...
May 02 15:09:55 EOS-Desktop systemd[1]: Repartition Root Disk was skipped because no trigger condition checks were met.
May 02 15:09:55 EOS-Desktop kernel: BTRFS: device label Game NVME 01 devid 1 transid 80074 /dev/nvme0n1p1 scanned by mount (773)
May 02 15:09:55 EOS-Desktop systemd[1]: Dispatch Password Requests to Console Directory Watch was skipped because of an unmet condition check (ConditionPathExists=!/run/plymouth/pid).
May 02 15:09:55 EOS-Desktop systemd[1]: Starting Load Kernel Module dm_mod...
May 02 15:09:55 EOS-Desktop systemd[1]: Starting Load Kernel Module loop...
May 02 15:09:55 EOS-Desktop systemd[1]: Rebuild Hardware Database was skipped because no trigger condition checks were met.
May 02 15:09:55 EOS-Desktop kernel: BTRFS info (device nvme0n1p1): first mount of filesystem 0158000c-99aa-4244-ae45-319d371654e8
May 02 15:09:55 EOS-Desktop kernel: BTRFS info (device nvme0n1p1): using crc32c (crc32c-intel) checksum algorithm
May 02 15:09:55 EOS-Desktop kernel: BTRFS info (device nvme0n1p1): using free-space-tree
May 02 15:09:55 EOS-Desktop systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
May 02 15:09:55 EOS-Desktop systemd[1]: Finished Load Kernel Module dm_mod.
May 02 15:09:55 EOS-Desktop systemd[1]: modprobe@loop.service: Deactivated successfully.
May 02 15:09:55 EOS-Desktop systemd[1]: Finished Load Kernel Module loop.
May 02 15:09:55 EOS-Desktop systemd[1]: Repartition Root Disk was skipped because no trigger condition checks were met.
May 02 15:09:55 EOS-Desktop kernel: intel_rapl_common: Found RAPL domain package
May 02 15:09:55 EOS-Desktop kernel: intel_rapl_common: Found RAPL domain core
May 02 15:09:55 EOS-Desktop systemd[1]: mullvad-early-boot-blocking.service: Deactivated successfully.
May 02 15:09:55 EOS-Desktop systemd[1]: Finished Mullvad early boot network blocker.
May 02 15:09:55 EOS-Desktop systemd[1]: Mounted /mnt/Game-NVME-01.
May 02 15:10:55 EOS-Desktop (udev-worker)[611]: hiddev1: Spawned process '/usr/bin/ddcutil chkusbmon /dev/usb/hiddev1 -v' [616] is taking longer than 59s to complete
May 02 15:10:55 EOS-Desktop systemd-udevd[599]: hiddev1: Worker [611] processing SEQNUM=2871 is taking a long time
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-C729\x2dBEEE.device: Job dev-disk-by\x2duuid-C729\x2dBEEE.device/start timed out.
May 02 15:11:25 EOS-Desktop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/C729-BEEE.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for /efi.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for Local File Systems.
May 02 15:11:25 EOS-Desktop systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
May 02 15:11:25 EOS-Desktop systemd[1]: efi.mount: Job efi.mount/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/C729-BEEE.
May 02 15:11:25 EOS-Desktop systemd[1]: systemd-fsck@dev-disk-by\x2duuid-C729\x2dBEEE.service: Job systemd-fsck@dev-disk-by\x2duuid-C729\x2dBEEE.service/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-C729\x2dBEEE.device: Job dev-disk-by\x2duuid-C729\x2dBEEE.device/start failed with result 'timeout'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.device: Job dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.device/start timed out.
May 02 15:11:25 EOS-Desktop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/b999f1d5-4da9-40af-98a7-71953710814c.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/b999f1d5-4da9-40af-98a7-71953710814c.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for /mnt/Storage-02.
May 02 15:11:25 EOS-Desktop systemd[1]: mnt-Storage\x2d02.mount: Job mnt-Storage\x2d02.mount/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: systemd-fsck@dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.service: Job systemd-fsck@dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.service/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.device: Job dev-disk-by\x2duuid-b999f1d5\x2d4da9\x2d40af\x2d98a7\x2d71953710814c.device/start failed with result 'timeout'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.device: Job dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.device/start timed out.
May 02 15:11:25 EOS-Desktop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/558fb03b-855c-4286-b19c-0777a0ba875b.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for /mnt/Storage-01.
May 02 15:11:25 EOS-Desktop systemd[1]: mnt-Storage\x2d01.mount: Job mnt-Storage\x2d01.mount/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/558fb03b-855c-4286-b19c-0777a0ba875b.
May 02 15:11:25 EOS-Desktop systemd[1]: systemd-fsck@dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.service: Job systemd-fsck@dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.service/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.device: Job dev-disk-by\x2duuid-558fb03b\x2d855c\x2d4286\x2db19c\x2d0777a0ba875b.device/start failed with result 'timeout'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.device: Job dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.device/start timed out.
May 02 15:11:25 EOS-Desktop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/07fbb757-2065-43c7-958e-f47f868103e8.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for /dev/disk/by-uuid/07fbb757-2065-43c7-958e-f47f868103e8.
May 02 15:11:25 EOS-Desktop systemd[1]: Dependency failed for Swaps.
May 02 15:11:25 EOS-Desktop systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.swap: Job dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.swap/start failed with result 'dependency'.
May 02 15:11:25 EOS-Desktop systemd[1]: dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.device: Job dev-disk-by\x2duuid-07fbb757\x2d2065\x2d43c7\x2d958e\x2df47f868103e8.device/start failed with result 'timeout'.
...
May 02 15:11:25 EOS-Desktop systemd-coredump[833]: Process 616 (ddcutil) of user 0 dumped core.
Stack trace of thread 616:
#0 0x00007e7dc7d8809d n/a (libc.so.6 + 0x16309d)
#1 0x0000587907da8031 strsplit (ddcutil + 0x8b031)
#2 0x0000587907dba4ba n/a (ddcutil + 0x9d4ba)
#3 0x0000587907dba778 is_possible_monitor_by_hiddev_name (ddcutil + 0x9d778)
#4 0x0000587907dba8a9 check_usb_monitor (ddcutil + 0x9d8a9)
#5 0x0000587907d38752 main (ddcutil + 0x1b752)
#6 0x00007e7dc7c4ad4a n/a (libc.so.6 + 0x25d4a)
#7 0x00007e7dc7c4ae0c __libc_start_main (libc.so.6 + 0x25e0c)
#8 0x0000587907d38895 _start (ddcutil + 0x1b895)
ELF object binary architecture: AMD x86-64
May 02 15:11:25 EOS-Desktop (udev-worker)[611]: hiddev1: Process '/usr/bin/ddcutil chkusbmon /dev/usb/hiddev1 -v' terminated by signal SEGV.
May 02 15:11:25 EOS-Desktop (udev-worker)[611]: hiddev1: Failed to wait for spawned command '/usr/bin/ddcutil chkusbmon /dev/usb/hiddev1 -v': Input/output error
May 02 15:11:25 EOS-Desktop (udev-worker)[611]: hiddev1: /usr/lib/udev/rules.d/60-ddcutil-usb.rules:8 Failed to execute "/usr/bin/ddcutil chkusbmon /dev/usb/hiddev1 -v": Input/output error
Storage1/2 and Game seem to hang because the dccutil chkusbmon udev rule times out. We even have dccutil dumping core.
Should probably reported upstream. As a quick workaround you could remove /usr/lib/udev/rules.d/60-ddcutil-usb.rules. Something is up with the USB chain.
Interesting. I pasted one of my errors into the ddcutils github issues page and I think I may have found some similar reports, but I’m not 100% sure if they’re the same. I’m using Nvidia proprietary drivers and #406 in particular looks a bit similar to some of my errors, right?
Here’s what I pasted into Github:
Summary
/usr/lib/udev/rules.d/60-ddcutil-usb.rules:8 Failed to execute “/usr/bin/ddcutil chkusbmon /dev/usb/hiddev1 -v”: Input/output error
