Boot stuck at "A start job is running for /dev/disk/by-uuid", UUID not familiar

General system Newbie
1

Hello super users,

After a 2+ week hiatus on system maintenance, I returned to my computer and attempted a standard yay update. After updating, I was no longer able to boot. I suspect there may have been a partial update due to lack of root partition memory. Hoping to address this in the future with a new hard drive dedicated to Endeavour this year.

My computer hangs with the command “A start job is running for /dev/disk/by-uuid/xxxxxxxx” indefinitely. However, the UUID presented during boot does not match with any UUIDs present in the system identified with ls -l /dev/disk/by-uuid and lsblk.

Boot: af16de27-957f-4d22-842a-48d32ed20617

Output of ls -l /dev/disk/by-uuid:

lrwxrwxrwx 1 root root 10 Jun 10 21:27 01DB89190D896F20 → ../../sda3
lrwxrwxrwx 1 root root 10 Jun 10 21:28 055681aa-77af-4ac5-a4a8-94c6cad069d6 → ../../sda8
lrwxrwxrwx 1 root root 10 Jun 10 21:28 4ECD-2C00 → ../../sda5
lrwxrwxrwx 1 root root 10 Jun 10 21:27 5259-2C08 → ../../sda1
lrwxrwxrwx 1 root root 10 Jun 10 21:28 62f8aaef-f122-49f3-89b0-1cf01a70f16d → ../../sda6
lrwxrwxrwx 1 root root 10 Jun 10 21:27 7A821266821226E3 → ../../sda4
lrwxrwxrwx 1 root root 10 Jun 10 21:28 A207-F98C → ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 10 21:28 ebc86489-dcfb-4215-985a-c4bac50f4be1 → ../../sda7

My boot and data partitions are sda7 and sda8 respectively, the other ones are used by a depreciated Windows 10 installation that I have not removed.

I am able to boot using USB media (Mercury, not Ganymede) and arch-chroot into my boot partition. I have run sudo pacman -Syu to no avail. Potentially relevant detail, I am running on a NVIDIA 1080 with 580 series of drivers as I had issues with Nouveau. I am using the AUR-maintained drivers and can not update them with pacman and cannot run yay using arch-chroot.

Any recommendations before I reinstall my OS (again) would be appreciated, hoping to learn something new about maintaining my computer. Thank you!

2

From the live session:

lsblk -f

or

blkid

Mount your root partition ( or subvolume if using btrfs) and post the contents of /etc/fstab

Depending on bootloader, post:

grub: /etc/default/grub

systemd-boot: /etc/kernel/cmdline

3

Thanks, Cactux. See below:

UUID present when boot hangs
af16de27-957f-4d22-842a-48d32ed20617

Output of lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /run/archiso/airootfs
sda
├─sda1 vfat FAT32 5259-2C08
├─sda2
├─sda3 ntfs 01DB89190D896F20
├─sda4 ntfs 7A821266821226E3
├─sda5 vfat FAT32 4ECD-2C00
├─sda6 swap 1 62f8aaef-f122-49f3-89b0-1cf01a70f16d
├─sda7 ext4 1.0 endeavouros ebc86489-dcfb-4215-985a-c4bac50f4be1
└─sda8 ext4 1.0 055681aa-77af-4ac5-a4a8-94c6cad069d6
sdb
└─sdb1 vfat FAT32 ENDEAVOUROS A207-F98C 25.9G 10% /run/archiso/bootmnt

Contents of fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=4ECD-2C00 /efi vfat fmask=0137,dmask=0027 0 2
UUID=ebc86489-dcfb-4215-985a-c4bac50f4be1 / ext4 noatime 0 1
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
UUID=055681aa-77af-4ac5-a4a8-94c6cad069d6 /home ext4 defaults 0 2

Contents of cmdline
nvme_load=YES nowatchdog rw root=UUID=ebc86489-dcfb-4215-985a-c4bac50f4be1

I do not see the UUID listed in boot in any of these locations

4

If you were able to arch chroot did you then run an update?

5

Do you use grub or systemd-boot, mkinitcpio or dracut?

6

Systemd-boot

7

Hi Rick, thanks for your time. Yes, I ran pacman -Syu but it did not fix the issue. I’ll note my graphics drivers were not able to be updated since they are AUR files. I still run a 1080 so I use the 580 series of NVIDIA drivers. My (newbish) understanding is the issue does not lie with the GPU, please re-educate me if I am wrong

8

############### Ignore this part, see below #########

To update your AUR packages inside chroot (#) you could use the su command first to run a command with substitute user, in this case your regular user, so:

# su your_username (I think su $USER should work too)

Then you can run you AUR updates as your user.

#####################################

Also, inside chroot, you could try to regenerate your initrds:

# reinstall-kernels

EDIT:
what I said above about switching user inside chroot is not exactly like that it seems. Please refer to

EDIT 2:

# su user_name seems to be working in my testing. Not sure if you need to cd into your user’s home directory or not. Anyways, I did and I could git clone and build an AUR package.

9

I’m not as familiar with systemd-boot issues as i use grub mainly. I wonder if in arch-chroot you reinstall the kernels would fix it?

10

After arch-chroot, su user_name yay was not working on my system, even sudo’d. Running it outside of $HOMEresponded “cannot execute binary file”. Once I changed into $HOME it said Permission denied, even if sudo’d. I tried changing directory into $HOME, which was /root. I wonder if it’s not possible on the Mercury live disk? My USB is outdated and I plan to fix that once I restore my system.

I was able to reinstall-kernels and will return with more feedback after I attempt to reboot.

EDIT: reinstalling kernel did not fix the issue, boot still stalls at the same place.

11

Have you reformat one of your partitions, swap perhaps?

If dracut, have you configured hibernation/resume?

Also, perhaps checking the logs may give some more information. In the live session, mount your system partition at /mnt and do:

journalctl --directory=/mnt/var/log/journal --list-boots

Then investigate:

journalctl --directory=/mnt/var/log/journal -b -1 ( or -b -2 and so on)

You could also use some filters:

journalctl --directory=/mnt/var/log/journal -p err
journalctl --directory=/mnt/var/log/journal -u '*.swap'
journalctl --directory=/mnt/var/log/journal | grep "resume"
12

Thanks for the continued support.

I have not knowingly reformatted any partitions. Swap still shows up when I run lsblk -f as seen here in sda6. My assumption is that it is still correctly formatted, but I could be wrong:

[liveuser@eos-2025.02.08 ~]$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /run/archiso/airootfs
sda
├─sda1 vfat FAT32 5259-2C08
├─sda2
├─sda3 ntfs 01DB89190D896F20
├─sda4 ntfs 7A821266821226E3
├─sda5 vfat FAT32 4ECD-2C00
├─sda6 swap 1 62f8aaef-f122-49f3-89b0-1cf01a70f16d
├─sda7 ext4 1.0 endeavouros ebc86489-dcfb-4215-985a-c4bac50f4be1
└─sda8 ext4 1.0 055681aa-77af-4ac5-a4a8-94c6cad069d6
sdb
└─sdb1 vfat FAT32 ENDEAVOUROS A207-F98C 25.9G 10% /run/archiso/bootmnt

I run dracut and have not configured anything special with hibernation/resume. However, I have successfully put the computer to sleep in the past. Let me know if there are any specific settings that I should research.

Here’s an interesting finding - when I examine the journal with --list-boots, the last entry is from my last successful boot in May, despite attempting to boot a few times over the last week or so. Perhaps this is unsurprising because I have not been able to fully boot and everything seems to get stuck. Even if I just call journalctl without any arguments (other than --directory), there are no entries since my last successful shutdown.

However, if I arch-chrootinto my mounted partition, then examine journal, I do see data from the last attempted boot. I posted it here: https://dpaste.com/CJCN7T88L

I would assume the last few lines would contain some sort of error. If there is one, I’m not skilled enough to see it.

13

You could try going back to the May successful boot using eos-shifttime

14

I used arch-chroot to access the boot partition, but receive the following errors running eos-shifttime, regardless of whether I use sudo:

[root@EndeavourOS /]# sudo eos-shifttime  
Authorization required, but no authorization protocol specified 


(yad:17): Gtk-WARNING **: 07:45:58.391: cannot open display: :0

I checked the forum and online but was not seeing much documentation on how to use the function. So, I entered the core commands of the tool directly into my terminal, thanks to these helpful Wiki pages (https://discovery.endeavouros.com/pacman/easy-downgrade-by-date/2021/06/) and (https://wiki.archlinux.org/title/Arch_Linux_Archive#How_to_restore_all_packages_to_a_specific_date).

The documentation shows this method doesn’t effect AUR packages, which are of particular interest to me because of my graphics driver. How would I downgrade those?

Of course, right at the 11th hour, it failed. Reinstalling my OS looks more tempting after this roadblock. I received the following error, after pacman checked for file conflicts:

error: failed to commit transaction (conflicting files)
ntfs-3g: /usr/include/ntfs-3g/acls.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/attrib.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/attrlist.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/bitmap.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/bootsect.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/cache.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/collate.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/compat.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/compress.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/debug.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/device.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/device_io.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/dir.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/ea.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/efs.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/endians.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/index.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/inode.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/ioctl.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/layout.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/lcnalloc.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/logfile.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/logging.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/mft.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/misc.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/mst.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/ntfstime.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/object_id.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/param.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/plugin.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/realpath.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/reparse.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/runlist.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/security.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/support.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/types.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/unistr.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/volume.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/include/ntfs-3g/xattrs.h exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/lib/libntfs-3g.so exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/lib/libntfs-3g.so.89 exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/lib/libntfs-3g.so.89.0.0 exists in filesystem (owned by libntfs-3g)
ntfs-3g: /usr/lib/pkgconfig/libntfs-3g.pc exists in filesystem (owned by libntfs-3g)
Errors occurred, no packages were upgraded.

15

In a stunning turn of events, reinstalling EndeavourOS did not fix the problem. I wonder if it has to do with the GPU. Since I’m running on an NVIDIA 1080, I installed without a desktop interface and then downloaded my drivers and KDE. However, doing so this time did not change my bizarre boot issues.

I would like to check the UUID of my graphics card, but nvidia-smi returns the following error:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I tried updating my drivers on the Live USB, which did not fix it. I tried rebooting my Live USB install after updating my drivers, and this also did not fix it (the update didn’t persist the reboot). Any recommendations on how to get my GPU UUID?

My next attempt will be to reinstall EndeavourOS and to deselect GPU drivers from the packages selector, even though there is no mention of NVIDIA within the selector. EDIT: I did more research about the GPU drivers within the packages selector and decided not to install without them. I think it would have been catastrophic since they were related to X11 function.

16

Hopefully you are using the latest ISO. You just use the default menu item when you boot the ISO. Once you install eos and reboot you can then install the Nvidia drivers from the AUR for the GTX 1080. It’s that simple. I have the GTX 1060 and no issues also dual booting with Windows using grub as the bootloader.

17

Excellent suggestion, I updated my USB install. Unfortunately, same issue.

18

I tried a few other things as well, including building the packages for the nvidia-580xx packages while booted from the ISO and then installing with pacman --sysroot. I am still getting the same errors.

Maybe this is a sign that I need to drop the NVIDIA card and switch to AMD.

19

I upgraded to a new AMD card. In hindsight, I could have tried booting with integrated graphics (if I have it on my PC, I don’t even remember.) Thanks everyone for the support, you are why I chose to stay with Endeavour!