[ Solved ] Boots Into Emergency Shell After Update (Encrypted root is not mounted)

Hystrix · March 6, 2023, 3:35pm

I have been running EnOS smoothly for a long time with encrypted root with btrfs and /boot on the same encrypted partition .

After a recent update - which includes linux kernel - when booting after grub menu it shows the encrypted root partition cannot be decrypted or mounted and drops to an emergency shell .

Basically the root partition is decrypted during initial boot by grub and the second time it’s not decrypted . I tried regenerating initramfs from live usb , since that’s the most found answer when searching . Also my mkinitcpio.conf haven’t changed after the update , it still has the keyfile to use to decrypt the root partition .

Funny thing is I restored a timeshift backup and tried updating again to see if update had any issue . Update happened without incident and with same results

I can’t find any similar behaviour in my searches . Hope to see some useful info from you guys

Pacman logs

[2023-02-28T22:07:49+0530] [PACMAN] Running 'pacman -S -y -u --config /etc/pacman.conf --'
[2023-02-28T22:07:49+0530] [PACMAN] synchronizing package lists
[2023-02-28T22:08:02+0530] [PACMAN] starting full system upgrade
[2023-02-28T22:09:10+0530] [ALPM] running '60-mkinitcpio-remove.hook'...
[2023-02-28T22:09:10+0530] [ALPM] transaction started
[2023-02-28T22:09:10+0530] [ALPM] upgraded eos-translations (23-11 -> 23-12)
[2023-02-28T22:09:11+0530] [ALPM] upgraded iso-codes (4.12.0-1 -> 4.13.0-1)
[2023-02-28T22:09:11+0530] [ALPM] upgraded eos-bash-shared (23-3 -> 23-4)
[2023-02-28T22:09:12+0530] [ALPM] upgraded firewalld (1.3.0-1 -> 1.3.0-2)
[2023-02-28T22:09:12+0530] [ALPM] upgraded libass (0.17.0-1 -> 0.17.1-1)
[2023-02-28T22:09:12+0530] [ALPM] upgraded libwacom (2.6.0-1 -> 2.6.0-2)
[2023-02-28T22:09:18+0530] [ALPM] upgraded linux (6.1.12.arch1-1 -> 6.2.1.arch1-1)
[2023-02-28T22:09:23+0530] [ALPM] upgraded linux-headers (6.1.12.arch1-1 -> 6.2.1.arch1-1)
[2023-02-28T22:09:38+0530] [ALPM] upgraded make (4.4-1 -> 4.4.1-1)
[2023-02-28T22:09:41+0530] [ALPM] transaction completed
[2023-02-28T22:09:42+0530] [ALPM] running '30-systemd-daemon-reload.hook'...
[2023-02-28T22:09:43+0530] [ALPM] running '30-systemd-hwdb.hook'...
[2023-02-28T22:09:45+0530] [ALPM] running '30-systemd-udev-reload.hook'...
[2023-02-28T22:09:45+0530] [ALPM] running '30-systemd-update.hook'...
[2023-02-28T22:09:45+0530] [ALPM] running '60-depmod.hook'...
[2023-02-28T22:09:48+0530] [ALPM] running '90-mkinitcpio-install.hook'...
[2023-02-28T22:09:48+0530] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2023-02-28T22:09:48+0530] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2023-02-28T22:09:48+0530] [ALPM-SCRIPTLET] ==> Starting build: 6.2.1-arch1-1
[2023-02-28T22:09:48+0530] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2023-02-28T22:09:48+0530] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2023-02-28T22:09:49+0530] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2023-02-28T22:09:49+0530] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2023-02-28T22:09:49+0530] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2023-02-28T22:09:50+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2023-02-28T22:09:51+0530] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2023-02-28T22:09:51+0530] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
[2023-02-28T22:09:52+0530] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
[2023-02-28T22:09:52+0530] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-02-28T22:09:53+0530] [ALPM-SCRIPTLET]   -> Running build hook: [encrypt]
[2023-02-28T22:09:54+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2023-02-28T22:09:54+0530] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2023-02-28T22:09:55+0530] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-02-28T22:09:56+0530] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
[2023-02-28T22:09:58+0530] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-02-28T22:09:58+0530] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2023-02-28T22:09:58+0530] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2023-02-28T22:09:58+0530] [ALPM-SCRIPTLET] ==> Starting build: 6.2.1-arch1-1
[2023-02-28T22:09:58+0530] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2023-02-28T22:09:59+0530] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2023-02-28T22:09:59+0530] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2023-02-28T22:09:59+0530] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2023-02-28T22:10:00+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2023-02-28T22:10:00+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2023-02-28T22:10:01+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2023-02-28T22:10:01+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2023-02-28T22:10:01+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2023-02-28T22:10:01+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2023-02-28T22:10:02+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2023-02-28T22:10:03+0530] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2023-02-28T22:10:04+0530] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
[2023-02-28T22:10:05+0530] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
[2023-02-28T22:10:05+0530] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-02-28T22:10:05+0530] [ALPM-SCRIPTLET]   -> Running build hook: [encrypt]
[2023-02-28T22:10:05+0530] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qat_4xxx
[2023-02-28T22:10:06+0530] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2023-02-28T22:10:08+0530] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-02-28T22:10:08+0530] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2023-02-28T22:10:10+0530] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-02-28T22:10:10+0530] [ALPM] running 'dbus-reload.hook'...
[2023-02-28T22:10:10+0530] [ALPM] running 'eos-reboot-required.hook'...
[2023-02-28T22:10:10+0530] [ALPM-SCRIPTLET] ==> INFO: Reboot is recommended due to the upgrade of core system package(s).
[2023-02-28T22:10:10+0530] [ALPM] running 'glib-compile-schemas.hook'...
[2023-02-28T22:10:11+0530] [ALPM] running 'gtk-update-icon-cache.hook'...
[2023-02-28T22:10:11+0530] [ALPM] running 'rebuild-detector.hook'...
[2023-02-28T22:10:16+0530] [ALPM] running 'texinfo-install.hook'...
[2023-02-28T22:10:16+0530] [ALPM] running 'update-desktop-database.hook'...

joekamprad · March 6, 2023, 3:52pm

logic would call for using different kernel.
I do not see a general issue for this setup reported…
And as you said you can open the containers… ISO has older kernel …

joekamprad · March 6, 2023, 4:13pm

this indicates it is also not an issue with failed run for rebuilding kernel images…

if i understand right you done this inside chroot and not on the livesession…

In addition to kernel images i would go also to reinstall grub (grub-install) and regenerate grub.cfg…
hmm just was curious and installed BTRFS encrypted grub and changing to mkinitcpio and got the same error… that it can not find uuid encryption was not the issue but it was not able to find the UUID device

… to me it looks like mkinitcpio.conf is overwritten and missing hooks?

PeterRies · March 6, 2023, 5:37pm

same here. updating my laptops frequently. just updated both laptops (systemd has been updated here)

one laptop boots up fine the second one drops to an emergency shell when running hook encrypt. Waiting 10 seconds. then throws a device /dev/mapper/luks/… not found and drops to an emergency shell.

weird that it works on the other laptop. lenovo t14 AMD and a lenovo x270. both fully encrypted, 99% identical installation.

(edited some typos - should learn 10 finger typing or type slower )

joekamprad · March 6, 2023, 5:48pm

i can not really reproduce the same issue because of using dracut and switching to mkinitcpio…

I will go now creating a vm with old iso to have the same starting point…
On my first test i was able to boot after fixing mkinitcpio.conf…

pgu · March 6, 2023, 5:51pm

Seems i am experiencing the same.

However, booting into the option with “fallback initramfs” works!

joekamprad · March 6, 2023, 5:52pm

hmm interesting!
and welcome at the purple race

Counting 3 with the same issue…

houston we have a problem ?

PeterRies · March 6, 2023, 5:57pm

Hi, I managed to boot with my regular kernel in grub. Usually I use lts kernel.

I recreated the initram kernel img (or however it’s called) with

sudo mkinitcpio -k 6.1.15-1-lts -g /boot/initramfs-linux-lts.img

After this I could boot again as always. Seems the img has been corrupted somehow…

Could somebody else please confirm or write a how to for users without second kernel?

Boot from Live USB, chroot, blabla, do this do that…

As mentioned: I updated via ‘yay’. The only basic package that has been updated was systemd. One machine ok, other with problem above.

joekamprad · March 6, 2023, 6:03pm

i hear from others over the last days that they had issue with mkinitcpio was not able to rebuild image…
Can be that simple…

I just run an install BTRFS grub encryped latest update no issue to boot.

  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [filesystems]

PeterRies · March 6, 2023, 6:08pm

may be I should have watched out for an error there…

is there an option to pause on errors / confirm errors when using yay for an upgrade - I never scroll back through the whole output to look for that.

(Note for the next time: I’ll check the output, I’ll check the output…)

joekamprad · March 6, 2023, 6:09pm

so make sure to run: sudo mkinitcpio -P if you are using mkinitcpio and not dracut… what is default up from cassini here…

joekamprad · March 6, 2023, 6:10pm

would be cool to have something like i see for Virtualmachine where it omits to reboot/poweroff till you save it…
could be grub again also??

pgu · March 6, 2023, 6:14pm

Thanks a lot for the warm welcome!

Did just regenerate via mkinitcpio -P and am up and running again.

But, initramfs files have been generated just before it seems:

Before:

.rw------- root root  27 MB Mon Mar  6 18:41:41 2023  initramfs-linux-fallback.img
.rw------- root root  10 MB Mon Mar  6 18:40:41 2023  initramfs-linux.img

After:

.rw------- root root  27 MB Mon Mar  6 19:10:21 2023  initramfs-linux-fallback.img
.rw------- root root  10 MB Mon Mar  6 19:09:23 2023  initramfs-linux.img

PeterRies · March 6, 2023, 6:18pm

just for completeness: I took a look in /var/log/pacman.log

Image generation successfull for both kernels, but one did not work after reboot. Weird, but solution is quite easy. Lesson learned: always have a second kernel at hand

good to know that fallback worked, too.

joekamprad · March 6, 2023, 6:25pm

yes i see the same just also installed LTS kernel and it had no issue building images too… only this error from grub … but as i was using old ISO and not reinstalled grub… i bet it could be something from before…

Could be some issue that is resolved already with latest packages…

joekamprad · March 6, 2023, 6:26pm

Hystrix · March 6, 2023, 8:07pm

HOOKS="base udev autodetect modconf block keyboard keymap consolefont encrypt filesystems"

these are the hooks in mkinitcpio.conf

FILES="/crypto_keyfile.bin"

is also present . When I ran mkinitcpio -P encrypt hook did execute .

Fallback image and linux-lts does not work for me either . To be clear , on both linux and linux-lts kernel image generation is successful , but booting this image creates problem .

joekamprad · March 7, 2023, 12:34am

i see this on telegram just now too… but i am far from beeing an expert… but is always on BTRFS filesystem…

Will need someone with more knowledge about BTRFS to check in…

csteinforth · March 7, 2023, 6:32am

Could it be this one?

https://bbs.archlinux.org/viewtopic.php?id=284011

PeterRies · March 7, 2023, 6:49am

This could be an explanation why mkinitcpio worked for me. I’m on ext4.