Stuck on "A start job is running for /dev/disk/by-uuid/" on booting

Hi.
I just upgraded my system using yay. The system recommended my to reboot since “core” packages were upgraded (linux kernel) and so I did.

But on booting all I see is; a start job is running for /dev/disk-by-uuid/3b2fa9a7-9b9b-444c-aa78-5787cf0e868c. I wait for a few minutes but nothing happens.

I’m running EndeavourOS Gallileo Neo (new install) along with dracut and refind as boot manager.

Here’s what I’ve done till now;

  • arch-chroot from a Live USB
  • pacman -Syu: No updates to install
  • sudo dracut-rebuild: Results in no errors that I can see. Can share the logs if needed.
  • Checked that UUID of the disk is the same in /etc/fstab and what shows in fdisk -l
  • fsck results in no errors either:
❯ sudo fsck.ext4 -f /dev/nvme1n1p2
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
endeavouros: 557375/121454592 files (0.7% non-contiguous), 40493136/485814628 blocks

I’m quite confused and can’t seem to figure out what the issue is.
I’ve gone through similar thread on the forum but nothing seems to work.

Result of fdisk -l
The error occurs when booting from nvme1 disks endeavourOS.
Note that I have another EOS installation on the second disk that I’m writing from.

❯ sudo fdisk -l
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WD Blue SN580 2TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3D153D11-4706-4444-9F5B-A5037EF172C0

Device              Start        End    Sectors  Size Type
/dev/nvme1n1p1       4096    2052095    2048000 1000M EFI System
/dev/nvme1n1p2    2052096 3888569126 3886517031  1.8T Linux filesystem
/dev/nvme1n1p3 3888569127 3907024064   18454938  8.8G Linux swap


Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Micron_2450_MTFDKBA1T0TFK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9CAF4CBF-217B-4550-BF9E-45ACF829C282

Device              Start        End   Sectors   Size Type
/dev/nvme0n1p1       2048     534527    532480   260M EFI System
/dev/nvme0n1p2     534528     567295     32768    16M Microsoft reserved
/dev/nvme0n1p3     567296  974845951 974278656 464.6G Microsoft basic data
/dev/nvme0n1p4  974845952  976494591   1648640   805M Windows recovery environment
/dev/nvme0n1p5  976496640 1228154879 251658240   120G Linux filesystem
/dev/nvme0n1p6 1228154880 1902096383 673941504 321.4G Linux filesystem
/dev/nvme0n1p7 1902096384 1952428031  50331648    24G Linux swap
/dev/nvme0n1p8 1953861632 1999998975  46137344    22G Microsoft basic data
/dev/nvme0n1p9 1999998976 2000408575    409600   200M Windows recovery environment


Disk /dev/sda: 15.23 GiB, 16357785600 bytes, 31948800 sectors
Disk model: Flash Disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4e4305b5

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *         64 5069119 5069056  2.4G  0 Empty
/dev/sda2       5069120 5390655  321536  157M ef EFI (FAT-12/16/32)

Here’s what I use to arch-chroot from the Live USB

sudo mount /dev/nvme1n1p2 /mnt
sudo mount -t proc /proc /mnt/proc
sudo mount -t sysfs /sys /mnt/sys
sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /run /mnt/run
sudo mount /dev/nvme0n1p1 /mnt/boot/efi
sudo arch-chroot /mnt

Post this

lsblk -o name,uuid,size,fstype,mountpoints

Here you go.
(ran from my other disk, hence the mountpoints / and /home)

❯ lsblk -o name,uuid,size,fstype,mountpoints
NAME        UUID                                   SIZE FSTYPE MOUNTPOINTS
nvme1n1                                            1.8T
├─nvme1n1p1 7016-0637                             1000M vfat
├─nvme1n1p2 3b2fa9a7-9b9b-444c-aa78-5787cf0e868c   1.8T ext4
└─nvme1n1p3 def17d23-50d4-4105-9114-bb4d0a6390dc   8.8G swap
nvme0n1                                          953.9G
├─nvme0n1p1 A40A-06A6                              260M vfat   /boot/efi
├─nvme0n1p2                                         16M
├─nvme0n1p3 FC2E0BD02E0B833E                     464.6G ntfs
├─nvme0n1p4 B82C24842C24402C                       805M ntfs
├─nvme0n1p5 6446609f-cd84-43ee-8d45-82b57536a2e7   120G ext4   /
├─nvme0n1p6 43c65df5-36ef-4a38-a3cd-2c72bc74f5c4 321.4G ext4   /home
├─nvme0n1p7 632b5e36-5771-45cf-84d3-509352b9e9db    24G swap   [SWAP]
├─nvme0n1p8 52E4A904E4A8EB83                        22G ntfs
└─nvme0n1p9 A0A9-1B2D                              200M vfat

As a FYI, using arch-chroot you could skip:

sudo mount -t proc /proc /mnt/proc
sudo mount -t sysfs /sys /mnt/sys
sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /run /mnt/run

arch-chrrot will set it all up for you.

:eye:
man arch-chroot
https://wiki.archlinux.org/title/Chroot#Using_arch-chroot

mount the partition nvme1n1p2 and show me the contents of fstab file

cat /etc/fstab

Thanks for the tip on arch-chroot!

Here’s the contents of fstab
As I said in the first post, the UUID is the same from lsblk so I’m quite confused as to what’s wrong here.

❯ cat /mnt/etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=7016-0637                            /efi           vfat    fmask=0137,dmask=0027 0 2
UUID=3b2fa9a7-9b9b-444c-aa78-5787cf0e868c /              ext4    noatime    0 1
UUID=def17d23-50d4-4105-9114-bb4d0a6390dc swap           swap    defaults   0 0
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0

Take a look here https://bbs.archlinux.org/viewtopic.php?id=288368

They solved with sudo pacman -R exfatprogs

¯\_(ツ)_/¯

Let me try to chroot and uninstall this then.

Okay this did not fix the issue. Seems like it worked randomly for the user.
Any ideas what I could try next?

Any help would be appreciated.

Wanted to avoid doing a re-install but I guess I’ll have to do that now.