[root] Update your system the smart way

EndeavourOS users are clever


Otherwise you wouldn’t have chosen a distribution so close to Arch.

If you have chosen EndeavourOS because you have heard so many good things of the community and Arch in general - these tips on updating the system may prove invaluable knowledge on your journey.

A couple of topics [1] [2] [3} have spurred me to rewrite a short article [6] on what to look out for when updating an Arch based system thus avoiding - hopefully - issues on restart.

Using an update notifier


Either when you use your system or shortly after login - a message appears - telling you there is updates available. Almost per instinct you initiate the update - provide credentials and off you go.

Suddenly your screen goes blank - you wait - nothing happens. You press CtrlAltF4 - nothing happens - you wait - nothing happens. You hold the power button until the system powers off and press it again to power on your system - and now you are having a black screen maybe a message …

What now? Could the situation have been avoided?

While - of course there is no guarantee - but the following headlines may help you understand what goes on behind the scenes.

  1. Know your system
  2. What happens when you update
  3. What happens or why when an update stalls
  4. Updating the smart way
  5. Conclusion

Know your system


There is a few packages - which when updated most likely will require a restart. A short list of important system packages which probably requires restart

  • ucode
  • cryptsetup
  • linux
  • nvidia
  • mesa
  • systemd
  • wayland
  • xf86-video
  • xorg

Another part of knowing your system is which foreign packages you have installed. Foreign packages is defined as everything not in the official repository so keeping a list of packages installed by an AUR helper or using makepkg -i will be of help when you update your system.

Before you update - you can query pacman for the list - pipe it to a text file for later referral

pacman -Qm > ~/aur-pkglist.txt && xdg-open aur-pkglist.txt

What happens on update?


Most of us has been using Windows and we know from Windows that system files may be locked an the system must be restarted to the file can be replaced with the updated versions.

Due to the modularity of Linux only needed portions of an application is loaded into memory and the rest is loaded on as-needed-basis - and this includes kernel, kernel modules and drivers.

When you build a package using an AUR helper the package is build using the - at build time - available system libraries. When you update your system - you are potentially replacing those dependencies with new libraries - which in turn may influence your system stability - especially when it comes to a kernel module build for a specific kernel.

When you reboot your system after an update the system will usually use the updated kernel upon reboot - and the device relying on a custom kernel module will fail because the module has not been rebuild.

Only after a successful update and a successful restart you can begin to update your custom modules.

What happens when an update stalls?


The following is a simplified description - of what may happen.

When you run huge updates within the graphical environment - whether you use a terminal or a GUI package manager - the system may end in a state where it needs to load a specific library at a specific memory address pointing at a specific routine in the library - but due to updates - the library is no longer existing - it has been updated to a never version - and your update stalls.

Updating the smart way


Updating the smart way includes knowing which packages may pose a problem if the update is run with the GUI. I showed you a list of packages which by a lot of users experience often requires special attention.

If you are using the eos-update-notifier - it will tell you if a reboot is required after running the updates.

If you are using other means of updating you can use a small utility script to evaluate the package list.

Utility script
#!/usr/bin/env bash
reboot="(ucode|cryptsetup|linux|nvidia|mesa|systemd|wayland|xf86-video|xorg)"
if [[ $(which yay) =~ (yay) ]]; then
    updates=$(checkupdates; yay -Qua)
else
    updates=$(checkupdates)
fi
echo "$updates"
if [[ $updates =~ $reboot ]]; then
    echo "Updating possibly requires system restart ..."
fi

[4] [5]

When you have evaluated the list and found this update to have the potential to stall - the smart way is to update using the console.

Ensure mirror is up-to-date

Before switching to console and executing the update command - ensure your primary mirror is up-to-date and remember to save the new mirror list

$ reflector-simple

Updating

Logout of your GUI and switch to TTY- CtrlAltF4 and run the update.

$ sudo pacman -Syu

If you changed mirrorlist use a double y to force sync of the package database - often necessary when primary mirror is changed by reflector simple

$ sudo pacman -Syyu

When you reboot after a successful update - first thing to do is rebuilding your foreign packages, drivers and kernel modules. This can be done from a terminal in the GUI session - but if you want to ensure you get no problems - TTY is recommended CtrlAltF4.

Depending on your DE you switch back to GUI using CtrlAltF7 or CtrlAltF1

Rebuilding all your AUR packages using a single command e.g.

$ yay --aur

Conclusion


Following a few basic rules will help you keep your system healthy and hopefully you will avoid problems from a stalled update.

8 Likes

In general, this is all solid advice.

I do have some differences in how I update my system.

  1. I think update notifiers are pointless on an distro that uses Arch repos. Unless you updated in the last few hours, it’s fairly safe to assume that there is an update available. I do use this script to safely check what updates are available, without synchronising the local package database.

  2. I never update from the TTY. I did that on Manjaro, and it somewhat makes sense there, since the updates are huge, and updating on Manjaro is a bit more stressful. But on Arch (and EndeavourOS) the updates are typically very small, if you update regularly. It seems to me that updating from the TTY is a myth and I have decided to test it by never doing it, and seeing how long until something breaks. So what I do to update is I simply run yay from the terminal emulator (three letters, plus a few more for my password). Nothing else. So far so good. :slight_smile:

  3. How often do I update? I update whenever I think to do it and I’m not busy with anything else and have a moment to do it. In practice, that’s typically every other day or so.

  4. I read this forum, so I have an idea when big updates, like the major kernel release are due. I keep that in mind when updating. I may delay an update to see whether other people are experiencing any issues.

  5. I think a good idea is to enable the magic SysRq button, in case you get the dreaded black screen after a botched update, so you can easily and safely reboot.

  6. I use Timeshift, btw. I haven’t desperately needed it (yet), Arch is a remarkably stable OS and if you’re not stupid, it doesn’t just break on its on. However, Timeshift gives me confidence when I tinker with my computers (which is the best way to learn, in my opinion).

  7. I don’t user reflector-simple I just use reflector. Ironically, I find the latter to be simpler to use (after RTFM).

6 Likes

3 Arch based systems I’ve installed and not one of them ever just froze during a update. Might have taken awhile to finish a update, but freeze during no. The only time I update the mirrors and databases is when I’m told the update can’t be done, that normally means that one of the two is out of date. If updating the mirrors and databases doesn’t do the job a normal reboot usually solves the problem. As for how I update I usually check Pamac to see what updates are out their after being told there are updates, then I decide if I want to launch Konsole and run topgrade or just wait.

Yes - the magnitude of updates on Manjaro - does seem to increase the possibility of stalling an update.

But in the event your window manager, xorg or display drivers (especially nvidia) is updated from GUI your are taking a risk. I am not saying it will happen - I am saying it may happen - and users which lacks experience with EOS - any Arch based system - will benefit from applying the necessary precautions.

One of the topics which spurred this post - was in fact a user booting to black screen. I am not sure if this post would have made a difference - but in any case - knowledge is power.

This is also my favorite - I was a little reluctant with this - but it has saved me a couple of times.

Knowledge is power

1 Like

:enos_flag:

6 Likes

Rocksolid as ever (✯ᴗ✯) nothing to add or drag (◍•ᴗ•◍)

Simply switching to a TTY doesn’t help very much because X and the DE/WM continue to run (traditionally on “F7”).

If you really want to be on a more secure side, you change runlevel:
systemctl isolate multi-user.target
which is systemd-ish for the traditional init 3.

3 Likes

If you don’t log out you are correct - but even if you don’t the TTY let’s you do things uninterrupted - I think I mentioned logout too.

Switching runlevels (should) automatically logs you out.
You’re basically closing the graphical session and starting a new non-graphical session.

Today, yes, I think I would agree. Many years back though, it was sort of a recommendation.

2 Likes

I told you - EndeavourOS users are smart - so are Arch users.

How do you suggest a user attracted by all the positivity surrounding EndeavourOS should handle situations where this huge update on a maintenance neglected system involves nvidia driver update - systemd update - xorg update - all within the same set?

To my experience, doing so from within X is the recipe for disaster.

I know - I the one always preparing for the worst - but doing so has saved my a… from time to time.

1 Like

Fair enough, but

you should not neglect maintenance on a rolling release distro. Maybe that’s something’s worth adding to your list?

3 Likes

Please don’t update with pacman -Syyu everytime unless you update mirrors/ have some good reason to do it. There’s no reason to download EVERYTHING everytime, and all you’re doing is causing more charges for whomever hosts the mirror. pacman -Syu is normally sufficient. If you only update once a month, it’s one thing, but if you’re updating daily or more, this is excessive.

Also, I generally do update from tty (I tend to update once a monthish though, so I’m definitely on the heavier download side), and I end up using F2, not F4. Different DE’s are different. You may just want to note it somewhere.

4 Likes

As for the TTYs, there are few knobs which you can use to change the behaviour.

For example, on KDE with SDDM, you can open the sddm config file /etc/sddm.conf and change
MinimumVT=n
in [X11] section.
If you set it to e.g. 5, then your KDE will run on “F5”.

Not sure about other DEs and login managers, but they might have similar options somewhere.

1 Like

That is correct - it is also mentioned that the double yy is usually only needed when you change your primary mirror.

Rephrased to clarify when what

If I was absent long enough for this to occur (!) - I suspect I would ShiftTime back long enough to limit the number of updates - and step back to the present in 2 week increments… :grin: It makes an interesting alternative to doing a complete re-install to avoid it!

1 Like

They do - but it is out of scope for the intended audience

lxdm which happens to be my choice

[server]
## arg used to start xserver, not fully function
arg=/usr/bin/X -background vt7

But I know lightdm has something similar

Same result :slight_smile: merely a habit

The TTY where X is running generally falls into two possibilities TTY1 or TTY7 - the latter - if I recall correct has historical reasons where TTY7 was the first non-occupied by system VT - or something like that :slight_smile:

1 Like