I updated my system on March 1st (from memory most likely via ‘yay’), there were no errors and things went smooth. After my next reboot I was greeted with a lot of “Failed to start”-spam: https://i.imgur.com/i9aZQul.png … Trying to enter via the lts kernel or with other initramfs all lead to the same result.
I didn’t have the time to look further into it and put off as much until today, booted into a live environment & used arch-chroot to get into the system.
Output of lsblk -o name,type,fstype,uuid,size (sdb2 is my EndeavourOS, sda1 popOS, sda3 Win10)
Summary
[liveuser@eos-2021.12.17 ~]$ lsblk -o name,type,fstype,uuid,size
NAME TYPE FSTYPE UUID SIZE
loop0 loop squashfs 1.7G
sda disk 931.5G
├─sda1 part ext4 399cabc3-c5a4-4b58-965f-745351207608 619.9G
├─sda2 part ntfs 920E739D0E7378D5 50M
├─sda3 part ntfs F006B44606B41018 307.6G
└─sda4 part swap 7b15e5af-8ae3-48b7-a5ad-b2529b2e139d 4G
sdb disk 931.5G
├─sdb1 part 8M
└─sdb2 part btrfs 32166223-8f4f-4f69-9149-fe979df93e58 931.5G
sdc disk 1.8T
├─sdc1 part ntfs 6276CC3176CC082D 100M
└─sdc2 part ntfs 4C8AB5D98AB5BFAE 1.8T
sdd disk 2.7T
├─sdd1 part 128M
└─sdd2 part ntfs 983EA77B3EA750D4 2.7T
sde disk iso9660 2021-12-17-11-32-24-00 3.6G
├─sde1 part iso9660 2021-12-17-11-32-24-00 1.8G
└─sde2 part vfat 6656-B5BD 102M
sdf disk 3.6T
├─sdf1 part 128M
└─sdf2 part ntfs D4A4DC46A4DC2CAC 3.6T
-uname a output surprised me (is this the likely culprit?):
Linux EndeavourOS 5.15.8-arch1-1 #1 SMP PREEMPT Tue, 14 Dec 2021 12:28:02 +0000 x86_64 GNU/Linux
(compared to “5.16.11-arch1-2” that I got via tty initially)
Trying to do anything with pacman in the chroot throws me a simple “Bus error” with no further comments.
Since BUS errors are scary here is the output of dmesg, I didn’t see anything super obvious in there but I wouldn’t know what exactly to look out for besides some kind of “ERROR” either: https://pastebin.com/dh9c2hcD
Help please, two weeks of primarily using windows were enough already. ;;
PS: If I do have to make changes via grub2 I have to admit I primarily used grub-customizer so far besides the initial setup and it’s throwing “no fstab found” errors when trying to access sdb2 (which makes sense since the fstab is in the subvolume @) so I’d need a bit of guidance if it goes towards that direction.
Thank you guys in advance & let me know if you need any more info. <3
I don’t think I did have network access (since I couldn’t start the network manager service either), but I also wouldn’t know how to doublecheck since I only tried a few pacman commands there that all failed. Should I reboot into it & what should I try there?
Some additional info: from the chroot pacman -Syu linux-lts actually throws: Bus error (core dumped)
With coredumpctl gdb pacman I can find the output
Summary
[root@EndeavourOS /]# coredumpctl gdb pacman
PID: 7051 (pacman)
UID: 0 (root)
GID: 0 (root)
Signal: 7 (BUS)
Timestamp: Sat 2022-03-19 13:30:22 CET (46s ago)
Command Line: pacman -Syu
Executable: /mnt/usr/bin/pacman
Control Group: /user.slice/user-1000.slice/session-1.scope
Unit: session-1.scope
Slice: user-1000.slice
Session: 1
Owner UID: 1000 (evo)
Boot ID: d6ce9e101f13410a97a6baccac6f8fb1
Machine ID: 6bbea445a130495a9891c21ccd28de12
Hostname: EndeavourOS
Storage: none
Message: Process 7051 (pacman) of user 0 dumped core.
Coredump entry has no core attached (neither internally in the journal nor externally on disk).```
…afterwards (gdb) bt from here throws me a "bash: syntax error near unexpected token `bt’
" though. Not sure if that’s useful info but thought I’d add it since it’s a tiny step more.
I think if it were me i would boot on the live ISO and arch-chroot into the installed system and then try updating. You should have network off the live ISO.
I think if it were me i would boot on the live ISO and arch-chroot into the installed system and then try updating. You should have network off the live ISO.
Apologies if you missed it in the OP, but I did try as much already:
I didn’t have the time to look further into it and put off as much until today, booted into a live environment & used arch-chroot to get into the system.
[…]
Trying to do anything with pacman in the chroot throws me a simple “Bus error” with no further comments.
I added a snippet to an earlier comment, but here’s the full output from arch-chroot via live ISO:
Summary
[liveuser@eos-2021.12.17 ~]$ sudo mount /dev/sdb2 /mnt -o subvol=@
[liveuser@eos-2021.12.17 ~]$ sudo arch-chroot /mnt
[root@EndeavourOS /]# uname -a
Linux EndeavourOS 5.15.8-arch1-1 #1 SMP PREEMPT Tue, 14 Dec 2021 12:28:02 +0000 x86_64 GNU/Linux
[root@EndeavourOS /]# pacman -Syu
Bus error (core dumped)
[root@EndeavourOS /]# pacman -Syyu
Bus error (core dumped)
[root@EndeavourOS /]# coredumpctl gdb pacman
PID: 5437 (pacman)
UID: 0 (root)
GID: 0 (root)
Signal: 7 (BUS)
Timestamp: Sat 2022-03-19 20:23:09 CET (36s ago)
Command Line: pacman -Syyu
Executable: /mnt/usr/bin/pacman
Control Group: /user.slice/user-1000.slice/session-1.scope
Unit: session-1.scope
Slice: user-1000.slice
Session: 1
Owner UID: 1000 (evo)
Boot ID: d15c429f3851491e8ae7f539f63d4881
Machine ID: 4be261af67284575b77db1360133b4df
Hostname: EndeavourOS
Storage: /var/lib/systemd/coredump/core.pacman.0.d15c429f3851491e8ae7f539f63d4881.5437.1647717789000000.zst (missing)
Message: Process 5437 (pacman) of user 0 dumped core.
Found module /usr/lib/libcurl.so.4.7.0 with build-id: b94db1d264ad1946b72b6c4d4cef2df149bde810
Found module /usr/lib/libcrypto.so.1.1 with build-id: 4c926b672d97886b123e03a008387aecf0786de4
Found module /usr/lib/libc.so.6 with build-id: 85766e9d8458b16e9c7ce6e07c712c02b8471dbc
Found module /usr/lib/libarchive.so.13.6.0 with build-id: 985731a3b1f779e47cfa73d88ad8f36e561d6ec6
Found module /usr/lib/libalpm.so.13.0.1 with build-id: 1407b7abcb95846a537d3909fb2549d118a5aa48
Found module /usr/lib/ld-linux-x86-64.so.2 with build-id: c09c6f50f6bcec73c64a0b4be77eadb8f7202410
Found module linux-vdso.so.1 with build-id: 4346c3c0ced3b2e43138af8f90f9dea2a2c99c1f
Found module pacman with build-id: 1cc848f0803908da8d4b35421402635ffb84dce8
Stack trace of thread 21:
#0 0x00007f62eaeeb3fa n/a (/usr/lib/ld-linux-x86-64.so.2 + 0x283fa)
#1 0x00007f62eaecc7ac n/a (/usr/lib/ld-linux-x86-64.so.2 + 0x97ac)
File "/var/lib/systemd/coredump/core.pacman.0.d15c429f3851491e8ae7f539f63d4881.5437.1647717789000000.zst" is not readable: No such file or directory
[root@EndeavourOS /]# (gdb) bt
bash: syntax error near unexpected token `bt'
[root@EndeavourOS /]# ```
That is where I’m out of ideas however, the only things I personally notice are the consistent Bus errors (Signal 7) and the different outputs to uname -a, I am not sure if that’s actually related to the error however.
e: Fixed the formatting, now it should actually be readable, apologies!
It’s fine. When in chroot, the kernel is not running, so the command reads the host’s kernel.
It looks like a failed update. Try to update the broken system, using USB installer (EnOS or Arch) with pacman --sysroot <dir> without chroot. Read man pacman.
Looking into coredumps right away, sounds like you are a Linux expert. I have only tried it once for fun. It’s for grown-ups . And you are using it wrong, I think (I am no expert, only a mind reader…).
Oh I have no idea what those say, I just thought I might as well include them when the error message told me it got dumped. =P
Alright, that very much seems like the right direction, thank you!
I followed the pacman wiki (Pacman crashes during an upgrade) and ran into a bunch of “error: failed retrieving file” from mirrors. The servers were able to be pinged, I doublechecked the mirrorlist and ended up finding this thread over on the arch forums that seemed to describe this precise issue & a working solution:
…and via that I was able to update the system. I am now able to access pacman commands just fine via the live medium & arch-chroot. No more Bus errors!
However: After trying to boot the system I ran into a “initramfs-linux.img not found”-error, tried again with fallbacks and I was greeted with the exact same issue that made me open this thread to begin with: https://i.imgur.com/vEgwaP5.jpg
pacman -Syu via arch-chroot from the live medium gave me this when a small update was pushed:
[liveuser@eos-2021.12.17 ~]$ sudo mount /dev/sdb2 /mnt -o subvol=@
[liveuser@eos-2021.12.17 ~]$ sudo arch-chroot /mnt
[root@EndeavourOS /]# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
multilib is up to date
:: Starting full system upgrade...
there is nothing to do
[root@EndeavourOS /]# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra 1717.3 KiB 6.14 MiB/s 00:00 [--------------------------------] 100%
community is up to date
multilib is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
Package (1) Old Version New Version Net Change Download Size
extra/python-tomli 2.0.0-1 2.0.1-1 0.01 MiB 0.02 MiB
Total Download Size: 0.02 MiB
Total Installed Size: 0.08 MiB
Net Upgrade Size: 0.01 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
python-tomli-2.0.1-1-any 22.6 KiB 752 KiB/s 00:00 [--------------------------------] 100%
(1/1) checking keys in keyring [--------------------------------] 100%
(1/1) checking package integrity [--------------------------------] 100%
(1/1) loading package files [--------------------------------] 100%
(1/1) checking for file conflicts [--------------------------------] 100%
(1/1) checking available disk space [--------------------------------] 100%
:: Processing package changes...
(1/1) upgrading python-tomli [--------------------------------] 100%
ldconfig: /opt/cuda/lib64/libcufftw.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcufftw.so.10 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcufftw.so.10.7.1.112 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusolver.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusolver.so.11 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusolver.so.11.3.3.112 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusparse.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusparse.so.11 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libcusparse.so.11.7.2.112 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppicc.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppicc.so.11 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppicc.so.11.6.2.112 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppist.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppist.so.11 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnppist.so.11.6.2.112 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnvToolsExt.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnvToolsExt.so.1 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /opt/cuda/lib64/libnvToolsExt.so.1.0.0 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: file /usr/lib32/libasan.so is truncated
ldconfig: file /usr/lib32/libasan.so.6 is truncated
ldconfig: file /usr/lib32/libasan.so.6.0.0 is truncated
ldconfig: File /usr/lib32/libgcc_s.so.1 is empty, not checked.
ldconfig: File /usr/lib32/libgdruntime.so is empty, not checked.
ldconfig: File /usr/lib32/libgdruntime.so.2 is empty, not checked.
ldconfig: File /usr/lib32/libgdruntime.so.2.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libgfortran.so is empty, not checked.
ldconfig: File /usr/lib32/libgfortran.so.5 is empty, not checked.
ldconfig: File /usr/lib32/libgfortran.so.5.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libgomp.so is empty, not checked.
ldconfig: File /usr/lib32/libgomp.so.1 is empty, not checked.
ldconfig: File /usr/lib32/libgomp.so.1.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libitm.so is empty, not checked.
ldconfig: File /usr/lib32/libitm.so.1 is empty, not checked.
ldconfig: File /usr/lib32/libitm.so.1.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libobjc.so is empty, not checked.
ldconfig: File /usr/lib32/libobjc.so.4 is empty, not checked.
ldconfig: File /usr/lib32/libobjc.so.4.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libquadmath.so is empty, not checked.
ldconfig: File /usr/lib32/libquadmath.so.0 is empty, not checked.
ldconfig: File /usr/lib32/libquadmath.so.0.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libstdc++.so is empty, not checked.
ldconfig: File /usr/lib32/libstdc++.so.6 is empty, not checked.
ldconfig: File /usr/lib32/libstdc++.so.6.0.29 is empty, not checked.
ldconfig: File /usr/lib32/libubsan.so is empty, not checked.
ldconfig: File /usr/lib32/libubsan.so.1 is empty, not checked.
ldconfig: File /usr/lib32/libubsan.so.1.0.0 is empty, not checked.
ldconfig: File /usr/lib32/libgio-2.0.so is empty, not checked.
ldconfig: File /usr/lib32/libgio-2.0.so.0 is empty, not checked.
ldconfig: File /usr/lib32/libgio-2.0.so.0.7000.4 is empty, not checked.
ldconfig: File /usr/lib32/libglib-2.0.so is empty, not checked.
ldconfig: File /usr/lib32/libglib-2.0.so.0 is empty, not checked.
ldconfig: File /usr/lib32/libglib-2.0.so.0.7000.4 is empty, not checked.
ldconfig: File /usr/lib32/libgobject-2.0.so is empty, not checked.
ldconfig: File /usr/lib32/libgobject-2.0.so.0 is empty, not checked.
ldconfig: File /usr/lib32/libgobject-2.0.so.0.7000.4 is empty, not checked.
ldconfig: file /usr/lib/libasan.so is truncated
ldconfig: file /usr/lib/libasan.so.6 is truncated
ldconfig: file /usr/lib/libasan.so.6.0.0 is truncated
ldconfig: file /usr/lib/libgdruntime.so is truncated
ldconfig: file /usr/lib/libgdruntime.so.2 is truncated
ldconfig: file /usr/lib/libgdruntime.so.2.0.0 is truncated
ldconfig: file /usr/lib/libgfortran.so is truncated
ldconfig: file /usr/lib/libgfortran.so.5 is truncated
ldconfig: file /usr/lib/libgfortran.so.5.0.0 is truncated
ldconfig: File /usr/lib/libgomp.so is empty, not checked.
ldconfig: File /usr/lib/libgomp.so.1 is empty, not checked.
ldconfig: File /usr/lib/libgomp.so.1.0.0 is empty, not checked.
ldconfig: File /usr/lib/liblsan.so is empty, not checked.
ldconfig: File /usr/lib/liblsan.so.0 is empty, not checked.
ldconfig: File /usr/lib/liblsan.so.0.0.0 is empty, not checked.
ldconfig: file /usr/lib/libstdc++.so is truncated
ldconfig: file /usr/lib/libstdc++.so.6 is truncated
ldconfig: file /usr/lib/libstdc++.so.6.0.29 is truncated
ldconfig: file /usr/lib/libtsan.so is truncated
ldconfig: file /usr/lib/libtsan.so.0 is truncated
ldconfig: file /usr/lib/libtsan.so.0.0.0 is truncated
ldconfig: file /usr/lib/libubsan.so is truncated
ldconfig: file /usr/lib/libubsan.so.1 is truncated
ldconfig: file /usr/lib/libubsan.so.1.0.0 is truncated
ldconfig: File /usr/lib/libbabl-0.1.so is empty, not checked.
ldconfig: File /usr/lib/libbabl-0.1.so.0 is empty, not checked.
ldconfig: File /usr/lib/libbabl-0.1.so.0.189.1 is empty, not checked.
ldconfig: File /usr/lib/libgettextlib-0.21.so is empty, not checked.
ldconfig: File /usr/lib/libgettextlib.so is empty, not checked.
ldconfig: file /usr/lib/libgettextpo.so is truncated
ldconfig: file /usr/lib/libgettextpo.so.0 is truncated
ldconfig: file /usr/lib/libgettextpo.so.0.5.7 is truncated
ldconfig: File /usr/lib/libgettextsrc-0.21.so is empty, not checked.
ldconfig: File /usr/lib/libgettextsrc.so is empty, not checked.
ldconfig: file /usr/lib/libgio-2.0.so is truncated
ldconfig: file /usr/lib/libgio-2.0.so.0 is truncated
ldconfig: file /usr/lib/libgio-2.0.so.0.7000.4 is truncated
ldconfig: file /usr/lib/libglib-2.0.so is truncated
ldconfig: file /usr/lib/libglib-2.0.so.0 is truncated
ldconfig: file /usr/lib/libglib-2.0.so.0.7000.4 is truncated
ldconfig: file /usr/lib/libgobject-2.0.so is truncated
ldconfig: file /usr/lib/libgobject-2.0.so.0 is truncated
ldconfig: file /usr/lib/libgobject-2.0.so.0.7000.4 is truncated
ldconfig: file /usr/lib/libfwupdplugin.so is truncated
ldconfig: file /usr/lib/libfwupdplugin.so.5 is truncated
ldconfig: file /usr/lib/libfwupdplugin.so.5.0.0 is truncated
ldconfig: File /usr/lib/libcc1.so is empty, not checked.
ldconfig: File /usr/lib/libcc1.so.0 is empty, not checked.
ldconfig: File /usr/lib/libcc1.so.0.0.0 is empty, not checked.
ldconfig: /usr/lib/libLTO.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /usr/lib/libLTO.so.13 is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /usr/lib/libRemarks.so is not an ELF file - it has the wrong magic bytes at the start.
ldconfig: /usr/lib/libRemarks.so.13 is not an ELF file - it has the wrong magic bytes at the start.
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Cleaning pacman cache...
==> finished: 263 packages removed (disk space saved: 2.89 GiB)
[root@EndeavourOS /]#
I would assume that the ldconfig issues are (harmless/unrelated) results of the (apparently very much failed) update and would be fixed by reinstalling the respective packages. Please correct me if I’m wrong there.
The only other idea I got from my google-fu was checking the output of LC_ALL=C pacman -Qk | grep -v ', 0 altered files' which resulted in exactly one entry with missing files that sounds a little bit terrifying to mess with: grub: 1104 total files, 4 missing files
No idea if that is the cause of the issue on boot but maybe that’s something we can find out somehow?
Until now I’ve thought grub should be alright since I can e.g. boot into my Win10 install as usual, but the couple missing files make me reconsider that idea. Of note is that the missing initramfs-linux.img was not an issue initially and that should count as one of those missing files I think? That must have gotten killed at some point during the troubleshooting (how? why? ;;).
Can grub really be the cause of all the “failed to start” spam? I guess it’s the best lead though.
Grub does not break on its own. It just reveals resulted problems of the real breakage.
How did you set your filesystem up with btrfs subvolumes? Did you manually set it up or from an installer. Give more details. It is important.
btrfs is mostly used to help recover broken systems, using snapshots. Have you set any relevant restore utility and how? You could have used a snapshot to restore your previous state.
This could have been a 3rd party caused problem, from one of your multi-boot systems messing with BIOS/UEFI/boot. You should provide system details. From chroot, this sudo inxi -Faz, or equivalent.
I set it up manually a bit more than a year ago. Me from back then was smart and separated / and /home into subvolumes of @ and @home, nothing more complicated than that. Fstab looks like this for reference:
You have no idea how much I’m facepalming right now. Me from two weeks ago tried to boot into a snapshot and got a error: you need to load the kernel first, after which I realized I don’t have the time to look into it and forgot about it.
Me from two days ago was a major idiot and I manually deleted some snapshots to make space for updating the system via chroot. ;_; … The most recent snapshot that is still around is from the beginning of February. Goddamnit.
How long do you have this setup without errors?
I am no btrfs expert, but it looks awful to me for some reason. Especially mounting permanently a partition in fstab on /mnt is unreliable IMHO. But having mounted both @ and /(subvolid=5) subvolumes looks scary to me.
I would check btrfs layout/listing both from outside and inside chroot. I am out of here. Call @dalto
Thank you, that was but a minor hiccup on the way sadly.
I was like “I know that name somehow…”
Last year July I had a boot issue after resizing partitions and that was who helped me back then and that fstab setup got a “Looks fine” at least from them. It did run without errors apart from when the Windows recovery tool / resizing the partition took a dump on grub back then. Initial setup was November 2020.
Thank you for the help though so far, being able to update/access pacman from the chroot was already a step forward. <3
I did get into a TTY initially when the issue first popped up 2-3 weeks ago, these pictures are what I can offer from back then: https://imgur.com/a/QRBKbYz - I have not tried getting into one since.
These were the things that stood out to me (just typing them out because google): (code=dumped, signal=BUS)
NetworkManager.service: Start request repeated too quickly.
NetworkManager.service: Failed with result 'core-dump'.
Failed to start Network Manager.
Is there something useful in there and if not what should I check for?