I was able to solve the issue so I’m just posting this if it happens to anyone else, I’m seeing lot of other posts already, sadly I forgot to take logs while I was in the live usb so I cannot give specific details but they were similar to the ones in the linked discussions anyway.
Please be careful as this was an emergency fix and you may break your system even more, so don’t follow these steps blindly especially without being ready to reformat everything.
After running yay to update some packages (nothing big actually) the system freezed and I had to reboot it with the power button. I’m on a Lenovo laptop with an AMD cpu and NVIDIA GPU running kernel 6.6.23-1-lts with windows in dual boot
the boot menu works fine and windows works correctly
endeavour will not start with the normal or fallback kernel, giving various errors
I downloaded the latest ISO and booted it live, entered the main installation with arch-chroot and noticed a lot of weird behaviour, some libraries inside /usr/lib and /usr/lib32 got corrupted, making it impossible to run pacman, nano, inxi and almost everything else.
Long story short:
I updated the live system with pacman -Syu to match the libraries version with the installed linux
installed on the live system the packages which were broken on the main installation but missing on the live system
copied the valid libraries from \usr\lib and \usr\lib32 on the live system to the main one, using cp -p to preserve permissions etc, overwriting the broken ones. I did not need to copy them all, but just enough to make pacman work. Start from liblzma
at this point I was able to run pacman inside the chrooted system and reinstall all the packages with pacman -S $(pacman -Qnq)
the system now boots fine and everything is working again
Weird behaviours on the chrooted system:
ls did not work, it always gave empty results, but if I used tab to trigger the automatic completion it did show the correct results
creating and copying files did not work, as if the changes were not applied to the disk
Workarounds:
copying/editing/creating or deleting files must be done from the live usb on and not directly from the chrooted system. Be careful to use the correct paths starting with /mnt to avoid breaking the live usb!
pacman-static can be downloaded and run as a standalone pacman. As with the user in the linked discussion there were various permissions errors, just copy it from the live usb to /mnt/usr/bin (or where you mounted it) and give it a chmod +x pacman-static for good measure. At this point the chrooted system was able to see and run it but it was still too broken to work. This can probably be done with other binaries too
broken signatures: I temporarily set in /etc/pacman.confSigLevel = Never to try and skip them but it did not work, I just realized that the one I set it is the default value but it gets overwritten below, so maybe it would have worked if I set it everywhere. Anyway nothing worked until the correct libraries were copied. In the meanwhile I had tried to recreate them deleting /mnt/etc/pacman.d/gnupg from the live usb, and then using from the chrooted system pacman-key --init which worked and pacman-key --populate which failed (arch wiki link). After copying the libraries pacman still failed because of the signatures but I was able to follow the key reset procedure again and it worked, allowing pacman to update
The system got broken again in the same way so I’m adding some more logs:
chrooted system
[root@EndeavourOS /]# yay
yay: error while loading shared libraries: /usr/lib/libassuan.so.0: file too short
[root@EndeavourOS /]# ls -la /usr/lib/libassuan.so*
lrwxrwxrwx 1 root root 18 Apr 1 12:15 /usr/lib/libassuan.so -> libassuan.so.0.8.7
lrwxrwxrwx 1 root root 18 Apr 1 12:15 /usr/lib/libassuan.so.0 -> libassuan.so.0.8.7
-rwxr-xr-x 1 root root 0 Apr 1 12:15 /usr/lib/libassuan.so.0.8.7
# we can use this to check for missing files but this was not the issue since the files are available but empty
[root@EndeavourOS /]# pacman -Qk 2>/dev/null | grep -v ' 0 missing files' | cut -d: -f1
# not sure if this is correct
[root@EndeavourOS /]# find /usr/lib -name files -size 0
live system
[liveuser@eos-2024.01.25 ~]$ ls -la /usr/lib/libassuan.so*
lrwxrwxrwx 1 root root 18 Jun 20 2023 /usr/lib/libassuan.so -> libassuan.so.0.8.6
lrwxrwxrwx 1 root root 18 Jun 20 2023 /usr/lib/libassuan.so.0 -> libassuan.so.0.8.6
-rwxr-xr-x 1 root root 84072 Jun 20 2023 /usr/lib/libassuan.so.0.8.6
# different version so lets check what the related packages are:
[liveuser@eos-2024.01.25 ~]$ pacman -Qi libassuan
Name : libassuan
Version : 2.5.6-1
Description : IPC library used by some GnuPG related software
Architecture : x86_64
URL : https://www.gnupg.org/related_software/libassuan/
Licenses : GPL3
Groups : None
Provides : libassuan.so=0-64
Depends On : glibc libgpg-error sh
Optional Deps : None
Required By : gnupg pinentry
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 220.92 KiB
Packager : David Runge <dvzrv@archlinux.org>
Build Date : Tue 20 Jun 2023 12:03:40 AM CEST
Install Date : Thu 25 Jan 2024 07:25:45 PM CET
Install Reason : Installed as a dependency for another package
Install Script : No
Validated By : Signature
# update the packages list
[liveuser@eos-2024.01.25 ~]$ sudo pacman -Syy
:: Synchronizing package databases...
# install the correct package version
[liveuser@eos-2024.01.25 ~]$ sudo pacman -S gnupg
resolving dependencies...
looking for conflicting packages...
Package (1) Old Version New Version Net Change Download Size
core/gnupg 2.4.3-2 2.4.5-1 0.17 MiB 2.68 MiB
# if installing packages throws a 404 error update them
[liveuser@eos-2024.01.25 ~]$ eos-rankmirrors
==> eos-rankmirrors: info: fetching https://gitlab.com/endeavouros-filemirror/PKGBUILDS/-/raw/master/endeavouros-mirrorlist/endeavouros-mirrorlist ...
==> eos-rankmirrors: info: ranking EndeavourOS mirrors, please wait ...
# copy the files
[liveuser@eos-2024.01.25 ~]$ sudo cp -p /usr/lib/libassuan.so* /mnt/usr/lib
back on the chrooted system
[root@EndeavourOS /]# ls -la /usr/lib/libassuan.so*
lrwxrwxrwx 1 root root 18 Apr 1 12:15 /usr/lib/libassuan.so -> libassuan.so.0.8.7
lrwxrwxrwx 1 root root 18 Apr 1 12:15 /usr/lib/libassuan.so.0 -> libassuan.so.0.8.7
-rwxr-xr-x 1 root root 84072 Jun 20 2023 /usr/lib/libassuan.so.0.8.6
-rwxr-xr-x 1 root root 84072 Jun 20 2023 /usr/lib/libassuan.so.0.8.7
[root@EndeavourOS /]# pacman -Qi libassuan
Name : libassuan
Version : 2.5.7-2
Description : None
Architecture : None
URL : None
Licenses : None
Groups : None
Provides : None
Depends On : None
Optional Deps : None
Required By : gnupg pinentry
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 0.00 B
Packager : None
Build Date : Thu 01 Jan 1970 01:00:00 AM CET
Install Date : Thu 01 Jan 1970 01:00:00 AM CET
Install Reason : Explicitly installed
Install Script : No
Validated By : Unknown
# broken db lock
[root@EndeavourOS /]# sudo pacman -Sy endeavouros-keyring
:: Synchronizing package databases...
error: failed to synchronize all databases (unable to lock database)
[root@EndeavourOS /]# rm /var/lib/pacman/db.lck
# reinstall all the packages again
# if we get "exists in filesystem" errors we need to clean the pacman cache
[root@EndeavourOS /]# pacman -S $(pacman -Qnq)
...
node-gyp: /usr/lib/node_modules/node-gyp/node_modules/ip-address/dist/address-error.js exists in filesystem
node-gyp: /usr/lib/node_modules/node-gyp/node_modules/ip-address/dist/address-error.js.map exists in filesystem
# cache cleaning with this and the run the reinstall again, everything should work
[root@EndeavourOS /]# pacman -Scc
Sadly updating the live image is now a problem with the switch to kde6 + new kernels and it broke the mount so I’m doing this without updating
I also noticed the kernel name switched to arch
Anyway if it breaks again I’d like to fix it permanently instead of reinstalling
Exact same configuration. I am on a Yoga Pro 7 14ARP8. I have been experiencing some freezes and kernel panics in the last few months. Not sure since when precisely. I had issues resuming from suspend in integrated graphics mode. They’re gone now that I use only hybrid mode, but still getting crashes from time to time. Especially on shutdown. And twice while updating.
Any idea why, or if the issue/bug has been identified?
After kernel panic during the update, I also managed to solve in a similar way. Since chrooted pacman was broken also in my case, instead of pacman-static I figured that it’s possible to launch pacman from the live system but with the --root /mnt flag so that it affects packages on the mounted (broken) system.
Damn I missed your post it would have spared me a couple of hours of search.
Anyway I checked the manjaro forum and it seems they have the same issues (a bit less, if I’m not wrong manjaro is slower to update stuff so maybe that’s the reason).
If I wanted to try anything I would switch to an older kernel, either 6.4 or 6.1, and then wait for 6.7 (or maybe 6.8) to be stable, since from the posts on the manjaro forum it seems 6.5 and 6.6 are problematic.
In the meanwhile I’ll just avoid updating the system until at least kernel 6.7 stable is available or I’ll move from lts to directly use it.
For now I tried creating the file, I tried downgrading nvidia-dkms to 545 but it whines about nvidia-utils and another one not being on the right version (of course). Don’t have time to switch to another driver or investigate how to downgrade correctly (it seems I just need to uninstall them and then install the 545 version). I will report after rebooting and running yay since I can see a kernel update, finger crossed
$ sudo downgrade nvidia-dkms
loading packages...
warning: downgrading package nvidia-dkms (550.67-1 => 545.29.06-4)
resolving dependencies...
warning: cannot resolve "nvidia-utils=545.29.06", a dependency of "nvidia-dkms"
:: The following package cannot be upgraded due to unresolvable dependencies:
nvidia-dkms
:: Do you want to skip the above package for this upgrade? [y/N]
error: failed to prepare transaction (could not satisfy dependencies)
:: unable to satisfy dependency 'nvidia-utils=545.29.06' required by nvidia-dkms
without looking at the eos-nvidia-fix file, i would say keep the file you created earlier and stay on the 545 drivers. also did you need to downgrade your kernel when you installed the 545 drivers?
edit: i havent updated in a few days but i do have the file present on my system and it just rebuilds kernels after nvidia driver installs. not a fix for this issue
I just used the file, did not downgrade the nvidia drivers or kernel, I should still have the 550 for nvidia and 6.6 for the kernel. I had 2 updates, both working with no issues,
i havent updated in a few days but i do have the file present on my system and it just rebuilds kernels after nvidia driver installs.
this fixes the problem of the system breaking down on updates, is this still happening to you? Nvidia drivers are “kernel modules”, which need to be embedded in the kernel when a new version is out.
A kernel module (or loadable kernel mode) is an object file that contains code that can extend the kernel functionality at runtime (it is loaded as needed); When a kernel module is no longer needed, it can be unloaded. Most of the device drivers are used in the form of kernel modules.
the do-not-udevadm-trigger-on-update option just stops udev from reloading devices after a driver update
When system updates or changes occur, such as installing or updating device drivers, udevadm might be triggered to reload device information or perform other actions related to device management.
The “do-not-udevadm-trigger-on-update” directive could serve to prevent udevadm from being triggered during specific update operations. This might be necessary to avoid potential conflicts or unintended consequences, especially when updating critical system components like drivers or kernel modules.
