Kernel panic during update broke my system. Thanks God and Linux Community for the arch-chroot

General system Newbie
1

It’s Arch Linux experience time, something that didn’t happend with me for last 4 years since I’ve switched to linux.
Laptop freezed on update, no responses, Caps Lock light blinking(Kernel panic), manualy turned it off - got an error:

/boot/vmlinuz-linux not found

Thankfully, I’ve already had an EOS installed on one of mine external SSDs, so I connected it to my laptop via usb.
And proceeded to chroot into my main system -

  1. Mounted the drive
  2. sudo arch-chroot path-to-EOS-partition
  3. Had problems with pacman database and keyring sudo rm /var/lib/pacman/db.lck
  4. Reinstalled the keying sudo pacman -S archlinux-keyring
  5. Once again tried to reinstall linux package but had many errors that lib.something.so exists but empty, so it’s heavy artillery time - sudo pacman -Syu --overwrite \* linux
  6. Disconnected the external drive, rebooting laptop - getting many weird errors on boot about Dbus and that failed to start User Login Manager.
  7. Booting back to my external SSD with EOS, chrooting.
  8. Even heavier artillery time, reinstalling all packages - sudo pacman -S $(pacman -Qqn) --overwrite '*'
  9. Rebooting again
  10. At last, it’s worked!
11 Likes
2

same thing happened to me a few weeks ago. however i had been planning on re-installing endeavour (i cluttered up my previous install and it was not in good shape sadly). so i chrooted in from my old endeavour live usb and after a bunch of trial and error i found that i had to NOT mount my boot partition and only the home partition(i was mounting both before) then regenerate grub config to get it to boot. backed up everything that couldn’t be downloaded to an external drive and re-installed, this time with kde(was on xfce before).

PS: i am also glad endeavour uses systemd-boot by default now. my first endeavour install was right after the grub update issue in summer 2022.

2 Likes
3

Same problem occured today, I guess it’s a sign that’s better to finally do a fresh system reinstall.

4

Did you check for hardware issues, RAM, disk?

5

Yes, I have a theory that perhaps kernel panic was caused by running out of physical RAM and zram with a physical swep were not enough.

6

What was getting updated?

Recently a user reported on frozen system while updating. electron25 from AUR seemed to be the culprit.

Others also have had issues with said package.

Perhaps you could monitor the system with htop (or some such) to see the RAM-zram-swap consumption during an update.

1 Like
7

meanwhile on windows…
search backup…
search installation disk
prepare a pot of coffee
cancel all meetings for the day
reinstall
reconfigure
still can’t find backup
…

1 Like
8

Tonns of stuff, 50 packages I think including Nvidia dkms. And yeah, electron too. ut I think I have it only from main repo, not aur.

9

It took me 5 minutes I think, 4 of which was downloading 4.5gb of all packages again for overwrite.

1 Like
10

Huh, I’ve discovered that I’ve actually had electron25, seems it’s was a dependency for some of the already uninstalled AUR packages.

+1 reason to hate electron

11

Well, that’s happend again right now even though electron package was removed. So I gues that’s not the problem at least this time.
Guess it’s time to migrate important stuff to external drive and to make a full fresh reinstall of the EOS.

12

So, now it’s happend after a fresh reinstall during a transaction hooks.
That’s getting annoying, I’ll try to switch to the LTS-kernel and see if this will change the situation.
I’m curious why it’s started happening so often, perhaps recent switch of the KDE to Wayland may cause so much troubles.

13

Maybe the disk is failing?
The first thing I’d do is backup of personal data, which is a good idea even if the disk isn’t failing… :wink:

14

I already have 2 backups on 2 different external disks, thanks.
SMART test of m2 nvme says it’s passed, don’t see any corrupted blocks.

1 Like
15

Huh, Hardware diagnostics in uefi says that smart check passed, but Short DST: Warning.

UPD
Seems that HP uefi says that it’s for all nvmes of my laptop model, including brand news.
As other tests show that ssd is perfectly fine.

16

Eh, now suddenly after fresh install I’ve got a system stuck on boot at Reached target Graphical Interface…
Several years of nigh flawless experience with Linux, last major problem was a mass breakage of GRUB and now I feel like I’m having a marathon of all what’s could be possibly broken.

17

Don’t know what tests you ran, but note that it is possible that (alternatively) RAM is not OK.
So thorough hardware check could be needed. If the warranty is still valid, you might want to have them check it too.

Alternatively, you could test another distro to see if it has any similar issues. That might validate a hardware problem, or not.

18

I did cheked RAM, tests have shown it’s fine too.

19

There could be many different reasons, both hardware and software.

Easy thing is to test with e.g. different kernel. Or reinstall some essential packages like kernel and graphics driver and related utilities.

BTW, is the power source powerful enough for your hardware, especially GPU?

20

That’s a valid concern. X11 seems to be working more reliably still.