Systemd-bootloader won't remember latest kernel, part 2

Yes, 7.0.2 got removed and did not show up in the boot entries, so I’d say that’s progress!

@dalto Sorry for the late ping, but I finally figured out what the whole problem was between this post and my original post for having ghost kernel files still on my system after kernel updates which has been an ongoing problem. I had installed the package kernel-modules-hook and from what I can understand, this was conflicting with kernel-install-for-dracut.

If I misunderstood that please let me know. I’ve since uninstalled the kernel-modules-hook, so hopefully within a week a new kernel will drop and I can fully test it, but I think I’ve finally solved this issue. I was tempted to make a very tiny PSA saying not to use the kernel-modules-hook package if you have kernel-install-for-dracut installed also, but maybe that was already mentioned somewhere because I don’t think anyone else has run into this issue except me.

During my updates today that had both kernels update, the pre-transaction removal showed:

:: kernel-install removing kernel 6.18.36-1-lts
:: kernel-install removing kernel 7.0.13-arch1-1

So both kernels were indeed removed, but then I noticed the post-transaction showed:

Running kernel-install for 7.0.13-arch1-2   ✓ new kernel
Running kernel-install for 6.18.37-1-lts    ✓ new lts kernel
Running kernel-install for 6.18.36-1-lts    ← this is the odd one

But then I think kernel-install ran for it again in the post-transaction step and essentially brought the 6.18.36-1-lts kernel back and kept it as my default so when I rebooted after these kernel updates instead of getting the latest 6.18.37-1 kernel, it kept and booted into 6.18.36-1-lts.

Checking sudo bootctl list shows a more complete picture after the update showing multiple kernels:

scott@endeavourOS:~$ sudo bootctl list
         type: Boot Loader Specification Type #1 (.conf)
        title: EndeavourOS (6.18.36-1-lts) (default) (selected)
           id: a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts.conf
       source: /efi//loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts.conf (on the EFI System Partition)
     sort-key: endeavouros-6.18.36-1-lts
      version: 6.18.36-1-lts
   machine-id: a718c5c3b60c42aa9332ecb00ebf37f6
        linux: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.36-1-lts/linux
       initrd: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.36-1-lts/initrd
      options: nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=31a2ff46-f831-4fbc-99ff-01546a396837 nvidia_drm.modeset=1 nvidia_drm.fbdev=1 systemd.machine_id=a718c5c3b60c42aa9332ecb00ebf37f6
         type: Boot Loader Specification Type #1 (.conf)
        title: EndeavourOS (6.18.36-1-lts-fallback)
           id: a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts-fallback.conf
       source: /efi//loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts-fallback.conf (on the EFI System Partition)
     sort-key: endeavouros-6.18.36-1-lts-fallback
      version: 6.18.36-1-lts-fallback
   machine-id: a718c5c3b60c42aa9332ecb00ebf37f6
        linux: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.36-1-lts/linux
       initrd: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.36-1-lts/initrd-fallback
      options: nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=31a2ff46-f831-4fbc-99ff-01546a396837 nvidia_drm.modeset=1 nvidia_drm.fbdev=1 systemd.machine_id=a718c5c3b60c42aa9332ecb00ebf37f6
         type: Boot Loader Specification Type #1 (.conf)
        title: EndeavourOS (6.18.37-1-lts)
           id: a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts.conf
       source: /efi//loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts.conf (on the EFI System Partition)
     sort-key: endeavouros-6.18.37-1-lts
      version: 6.18.37-1-lts
   machine-id: a718c5c3b60c42aa9332ecb00ebf37f6
        linux: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.37-1-lts/linux
       initrd: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.37-1-lts/initrd
      options: nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=31a2ff46-f831-4fbc-99ff-01546a396837 nvidia_drm.modeset=1 nvidia_drm.fbdev=1 systemd.machine_id=a718c5c3b60c42aa9332ecb00ebf37f6
         type: Boot Loader Specification Type #1 (.conf)
        title: EndeavourOS (6.18.37-1-lts-fallback)
           id: a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts-fallback.conf
       source: /efi//loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts-fallback.conf (on the EFI System Partition)
     sort-key: endeavouros-6.18.37-1-lts-fallback
      version: 6.18.37-1-lts-fallback
   machine-id: a718c5c3b60c42aa9332ecb00ebf37f6
        linux: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.37-1-lts/linux
       initrd: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/6.18.37-1-lts/initrd-fallback
      options: nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=31a2ff46-f831-4fbc-99ff-01546a396837 nvidia_drm.modeset=1 nvidia_drm.fbdev=1 systemd.machine_id=a718c5c3b60c42aa9332ecb00ebf37f6
         type: Boot Loader Specification Type #1 (.conf)
        title: EndeavourOS (7.0.13-arch1-2)
           id: a718c5c3b60c42aa9332ecb00ebf37f6-7.0.13-arch1-2.conf
       source: /efi//loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-7.0.13-arch1-2.conf (on the EFI System Partition)
     sort-key: endeavouros-7.0.13-arch1-2
      version: 7.0.13-arch1-2
   machine-id: a718c5c3b60c42aa9332ecb00ebf37f6
        linux: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/7.0.13-arch1-2/linux
       initrd: /efi//a718c5c3b60c42aa9332ecb00ebf37f6/7.0.13-arch1-2/initrd
      options: nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=31a2ff46-f831-4fbc-99ff-01546a396837 nvidia_drm.modeset=1 nvidia_drm.fbdev=1 systemd.machine_id=a718c5c3b60c42aa9332ecb00ebf37f6
scott@endeavourOS:~$ 

scott@endeavourOS:~$ sudo bootctl list | grep -E "title|version"
        title: EndeavourOS (6.18.36-1-lts) (default) (selected)
      version: 6.18.36-1-lts
        title: EndeavourOS (6.18.36-1-lts-fallback)
      version: 6.18.36-1-lts-fallback
        title: EndeavourOS (6.18.37-1-lts)
      version: 6.18.37-1-lts
        title: EndeavourOS (6.18.37-1-lts-fallback)
      version: 6.18.37-1-lts-fallback
        title: EndeavourOS (7.0.13-arch1-2)
      version: 7.0.13-arch1-2
        title: EndeavourOS (7.0.13-arch1-2-fallback)
      version: 7.0.13-arch1-2-fallback
        title: Reboot Into Firmware Interface
scott@endeavourOS:~$ 

scott@endeavourOS:~$ sudo ls /efi/loader/entries/
a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts.conf          a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts.conf        a718c5c3b60c42aa9332ecb00ebf37f6-7.0.13-arch1-2.conf
a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts-fallback.conf  a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts-fallback.conf  a718c5c3b60c42aa9332ecb00ebf37f6-7.0.13-arch1-2-fallback.conf

I can see there were still some left overs to remove:

scott@endeavourOS:~$ ls /usr/lib/modules/
6.18.36-1-lts  6.18.37-1-lts  7.0.13-arch1-2

So I removed a bunch of things:

sudo rm /efi/loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts.conf
sudo rm /efi/loader/entries/a718c5c3b60c42aa9332ecb00ebf37f6-6.18.36-1-lts-fallback.conf
sudo rm -rf /efi/a718c5c3b60c42aa9332ecb00ebf37f6/6.18.36-1-lts/
sudo rm -rf /usr/lib/modules/6.18.36-1-lts
sudo pacman -Rns kernel-modules-hook
sudo bootctl set-default a718c5c3b60c42aa9332ecb00ebf37f6-6.18.37-1-lts.conf

tl;dr: I removed kernel-modules-hook which was conflicting with kernel-install-for-dracut and causing old kernels to get re-added to EFI after every new kernel update.

Yes, there actually was an issue for that on gitlab.

The issue is that kernel-install-for-dracut installs all the installed kernels. However, kernel-modules-hook makes a copy of /usr/lib/modules including the kernel itself. So when is checks the installed kernels, it finds the kernel that the hook copied.

I think to fix that you would need to make minor changes to kernel-install-hook. Either make the post hook run after the kernel-install hook or stop kernel-install-hook from copying the actual kernel files and only copy the modules.

Ah I didn’t know about the GitLab issue. I appreciate your suggestions. I’m trying to keep my setup as minimal for my needs as I can, and trying to keep as much of it as I can automated i.e. sytemctl timers for cleanup, but the kernel-install-hook I think I’m just better off not using that package.

I reboot after kernel updates and I don’t backup as frequently as I used to (just once a week now) so having the modules load after a kernel update just so I can maybe plug in my external SSD to backup files I forgot before a reboot hasn’t really been all that necessary for my needs like I had thought. It almost feels unnecessary for my needs now. I’m just glad I think I finally figured out the ghost kernel issue.