Booting into a black screen after a failed update

[eldelacajita]

NOTE: I managed to fix the issue, so this is more like a success story than the cry for help I thought it would be. But I had already written all of this, and I thought I could post it anyway as a cautionary tale and for future reference (for future me, or for others).

Where I am

When starting the system, after the Grub screen, all monitors go totally blank/black and I can’t do anything else, not even switch to command line.
I can edit the grub boot line and log in without the graphic environment, and I can start the system and even update it with no errors.
I’ve ruled out hardware errors because with liveUSBs and on Windows everything works.

How I got here

You know those comedy films where, trying to fix an awkward situation, the protagonist ends up digging themselves deeper and deeper into a complete mess? That’s how these past days felt.

I can’t summarize the whole process, but it started with executing one month worth of updates on very low disk space, and it involved booting into vmlinuz-linux not found errors, going back to previous BTRFS snapshots, trying to update from there to no avail (yea, still not enough disk space), reading a lot of forum posts and guides, modifying disk partitions to gain some space (yeah, past the warning about moving a system partition), sweating and fixing NTFS partition errors, several chroot sessions that included updating the system and Grub, and finally getting stuck where I am now: at a Blank Screen of Death.

I have read lots of posts here in the forum and in other places, but most don’t seem to fit my case.

Things I’ve tried

• ctrl-alt-f2 doesn’t seem to work, so I restarted in text mode from grub.
  • For future forgetful me, that means pressing eat the Grub screen, adding a 3 at the end of the line that starts with linux, and quitting with Ctrl+X.
• After logging in to the command line, doing a full system update.
  • It works with no errors, but it doesn’t fix anything.
• Finding out which display manager I’m using (GDM) and seeing if it’s working

  • systemctl status gdm.service

  • Loaded: loaded (/usr/lib/systemd/system/gdm.service; enabled; preset: disabled)
    Active: inactive (dead)
    

  • So yeah, GDM is dead. Now I have to find out why.

• Restarting GDM
  • systemctl restart gdm.service
  • Now I get a blank screen with a blinking cursor. An improvement, I guess?
• Reading the journalctl log to see if I can learn something from it.
  • Admitting I don’t know what to look for.
  • Just search for “error” and see.

Relevant (I think) info

• I’m on a full AMD system, no Nvidia at all.
• Output of inxi -z -CG

	  apl
	  CPU:
	    Info: 6-core model: AMD Ryzen 5 PRO 4400G with Radeon Graphics bits: 64 type: MT MCP cache:
	      L2: 3 MiB
	    Speed (MHz): avg: 1425 min/max: 1400/4000 cores: 1: 1400 2: 1400 3: 1400 4: 1400 5: 1400
	      6: 1400 7: 1400 8: 1400 9: 1400 10: 1700 11: 1400 12: 1400
	  Graphics:
	    Device-1: AMD Navi 14 [Radeon RX 5500/5500M / Pro 5500M] driver: amdgpu v: kernel
	    Device-2: AMD Renoir [Radeon Vega Series / Radeon Mobile Series] driver: amdgpu v: kernel
	    Device-3: Logitech C920 PRO HD Webcam driver: snd-usb-audio,uvcvideo type: USB
	    Display: unspecified server: X.org v: 1.21.1.13 with: Xwayland driver: X: loaded: N/A
	      failed: amdgpu,modesetting,radeon gpu: amdgpu tty: 240x75 resolution: 1: 1920x1200 2: 1920x1200
	      3: 1920x1200
	    API: EGL Message: No EGL data available.
	    API: OpenGL Message: GL data unavailable in console for root.

• Output of last boot to blank screen via journalctl -b -1
  • https://pastebin.com/Ghh3TCcc

Questions

• What should I look for in the journalctl log?
• What can I try to reinstall, asuming this is caused by a broken update? Is there a way of reinstalling every package, or something like that?
• Why on earth does the system not warn you when there’s not enough disk space for the updates? Duh.

[eldelacajita]

Now, to the success part, so I can mark this as solved:

• I searched for the word “error” in the journalctl log.
• First error I found: Process org.freedesktop.systemd1 exited with status 1
  - This gave me a clue to find some more related posts. This one seemed particularly relatable, and it seemed to point to graphic driver issues. It was solved by reinstalling. I didn’t want to reinstall
• Check which version of mesa I have:
  • pacman -Qi mesa
    • 24.1.6-1
    • There didn’t seem to be an issue with this particular version.
• Reinstalling mesa
  • pacman -Sdd mesa
  • Oh. OH. OOOOOOH. That fixed it!

Turns out, one of the things damaged during the failed update was the graphics driver.

[eldelacajita]

Things I learned during this crisis

• DON’T UPDATE IF THERE’S NOT MUCH DISK SPACE LEFT. I remember reading this in some other post in the forum, but I can’t seem to find it. It was not this one (which is great too).
  • How much is not much? No idea. I just know that if you haven’t updated in a month, the update will download more than 1,5 GB in packages, and it will need even more space to install them.
• Don’t mindlessly rely on BTRFS snapshots (created with TimeShift). You can easily use them to go to a previous system state, but it may not solve your issue or you may end up with some new issues to solve.
• Work-Linux balance: If your main system fails and you have a backup laptop, don’t try to work while fixing the main system. Just focus on work until you can save some time to sit and focus on fixing your system.
• Some basic terminal skills I had to look up:
  • Getting to the terminal if the GUI is not loading:
    • Press eat the Grub screen.
    • Add a 3 at the end of the line that starts with linux.
    • Quit and boot with Ctrl+X
  • Working with journalctl to check for errors.
    • This guide was very useful to me.
  • Writing the journalctl output to a file in the current folder
    • journalctl (options) > filename.txt
  • Mounting a usb drive so you can copy the log and study it in another computer. I used this answer.
    • Plug the USB drive in. Seriously, I skipped this step once.
    • Use lsblk and fdisk -l to identify your USB drive.
      • The first command gives you a simple outline of available devices, the second command gives you more info (like disk size) which can be useful for identifying a drive. Use any or both of them in any order.
    • mkdir (mount poing, e. g. /mnt/usbdrive)
      • This step is almost never writen in the guides, but if I don’t do it, I always get a missing folder error when mounting. I’m probably doing something wrong here.
    • sudo mount (external drive or partition e. g. /dev/sdc1) (mount point, e. g. /mnt/usbdrive)
• If you grow bad feelings against Linux for all this you didn’t want in your life, remember that on Windows your only option would be wiping and reinstalling, or hiring a tech support that would wipe and reinstall for you. So this is just you taking advantage of the possibility Linux gives you to fix your system. If you’re not up for it, you know… wipe and reinstall.