Optimising Stability

Apart from spacing your updates out and using Timeshift, what tips do you guys have for making sure the system is as stable as possible? I’ve officially quit dual-booting and have phased out Windows entirely, and am now 100% Endeavour. It’s awesome, and my hope is to have this one installation last me years… So anything helps. Thanks guys

Stability can mean a lot of things, see here for my thoughts on that.

A few things I would recommend:

  • Do updates as frequently as possible, the more you wait, the more risk of introducing issues.
  • Use the LTS kernel unless you have very new hardware that requires a newer kernel.
  • Don’t install lots of extra packages you don’t use.
  • When adding or removing packages, read the list of packages that is about to removed/installed to ensure it looks sensible.
  • If something goes wrong, don’t panic and start taking random actions. Approach the situation calmly and resolve the issue. If you need help, ask.

Arch-based distros rarely if ever need to be reinstalled. You can run the same installation as long as the hardware lasts.

10 Likes

I think Dalto is right on the money, though I don’t run an LTS kernel, I do keep one installed as a fallback plan.

4 Likes

Read the output of any package changes, some will require manually checking and merging configuration files (.pacnew) which can be done from EOS Welcome or a diff utility of your choice like Meld.

1 Like

I’ve just switched to a LTS kernel. My machine was showing some strange stability problems, and my first thoughts were that I had hardware issues. Them I remembered I had the LTS installed and swapped to it. Now everything is stable again and there is no performance hit in any way. Not surprising really as the current LTS kernel is quite new anyway being 5.10. :grin:

4 Likes

On an OS like Arch/EndeavourOS, it is the user error that is the biggest source of system instability.

In every case of my Arch or EndeavourOS systems breaking, it was my fault, 100% of time. I was careless and stupid and did something that broke my system (this is not the case with 'buntu, though). I have yet to experience a major OS breakage on Arch which wasn’t my fault, in the sense that I had no way of seeing it coming and couldn’t have prevented it by being less stupid.

So, the most general advice I have for system stability on Arch/EndeavourOS is: don’t be careless and stupid.

I know this is not very specific, so here are a few specific points (not an exhaustive list):

  1. Back up all important data. Verify that backups work, keep duplicate backups, keep backups on external media. Not doing so is the pinnacle of stupid.
  2. Don’t just copy-paste terminal commands from the internet not knowing what they do (that’s really stupid, almost as stupid as not having a backup).
  3. Whenever you type the word sudo into the terminal, let this be a cue for you to think twice about what you’re about to do. The three questions to ask yourself are: “What am I trying to do?”, “What am I actually doing?”, and “Why am I using sudo to do this?”.
  4. Never use sudo with GUI applications. Never use sudo with software-specific package managers like pip, gem, npm… (on most distros you can do this, but on Arch this will cause you trouble). Never use sudo with wine (this is incredibly stupid!). Never do sudo bash or sudo zsh. Avoid using sudo su (except on a live image). Do not login as your root user in an interactive shell (unless your system is utterly broken and you know exactly what you’re doing).
  5. In general, never use sudo when you can do the same task without sudo.
  6. When you are in doubt, ask on the forum. Even asking a stupid question is less stupid than not asking it and doing a stupid thing. Read the forum topics about other people having problems and what the solutions are. Smart people learn from the mistakes of others.
  7. Protect your cp, mv and rm commands with the -i flag in your .bashrc:
    alias cp="cp -i"
    alias mv="mv -i"
    alias rm="rm -i"
    
    This will require you to confirm with y every time you’re about overwrite or delete a file. When you have a lot of files to delete, you can override this with the -f flag manually (which also forces you to think about what you’re about to do). Also consider turning off redirection clobbering:
    set -o noclobber
    
    In my specific case, this has prevented about 90% of all my terminal accidents due to carelessness.
  8. Before doing something you don’t have a lot of experience with, think about it and plan it. Know what you want to do and how to do it in advance. Take the time to read the manpages and/or ArchWiki, whenever you’re about to enter uncharted territory.
  9. On the flip side, whenever you’re doing routine system maintenance and things you’re very comfortable doing and have a lot of experience with, resist the urge to be on autopilot. Do your best to pay attention to what’s going on. When updating your system, do not walk away from the keyboard. Never ignore error messages and warnings.
  10. Don’t use a GUI package manager (that’s stupid).
  11. Do not use the power and reset switches on your computer case to reboot your computer, even if it is completely frozen. Learn about the magic SysRq key. This will prevent filesystem corruptions due to hard reset.
5 Likes

Great advices by @dalto & @Kresimir

Although i strongly disagree with that one, at that point it’s almost theoretical, if let’s say you have a musical studio with all your software installed and ready to rock - you can do updates perfectly safe once a month or once a 6 months, it doesn’t really matter as long as you make it right (reading news, merging .pacnew files).

1 Like

Regarding the frequency of updates, it’s a balancing thing. If you do updates very frequently, the updates are smaller, there are fewer things to worry about, fewer .pacnews, etc… Fewer packages get updated, so there is a smaller chance of something going wrong.

On the other hand, when there is something broken in an update, you’ll be the first to get it. Sometimes it may be better to wait with updates, until other people update first and if something is broken, they fix it (of course, this will not work if everyone delays updating).

I think the second scenario is quite rare: updates, at least on Arch, tend to get tested fairly well before being pushed to the stable branch. Maybe with major kernel releases, it’s worth waiting a day or two, or use the LTS kernel, but @dalto already covered that.

1 Like

Great advice guys! :+1:

This is also a good point. Please note that program eos-pacdiff integrates programs pacdiff and <your-diff-program> to be easy to use. You can configure which diff program to use, but usually meld is the default.

Welcome runs eos-pacdiff with button Pacdiff & <your-diff>.

But as rightfully noted above, eos-pacdiff must be used carefully. Some files (e.g. /etc/passwd) are not meant to be modified with pacdiff, but most are.

1 Like

Before using any utility to handle .pacnew files, it’s a good idea to do it manually for awhile, using a text editor. That way you know what’s going on, and using an automated utility is then safer.

Same thing with building packages from the AUR. It’s a good idea to do it with makepkg at least a few times before using an AUR helper like yay.

2 Likes

But it’s not automated, in case of meld at least…It’s manual :upside_down_face:

1 Like

This is good advice, but I have to admit, I have never done it except when I was playing with those tools to learn how they worked. Not on any of my Arch-based systems. In the rare cases something has broken(which it has), I then go look to see if there is a pacnew file hanging around which needs merging.

To be clear, I am not recommending others do that. It will cause breakages.

Doing updates every 6 months on an Arch-based distro is a risky proposition unless you have a good amount of expertise. I maintain over a hundred Linux VMs for testing purposes at this point. Many are Arch-based distros. It isn’t uncommon for me to go 6 months between updates. On a basic system, with no extra software installed, it isn’t trivial but it also isn’t that hard as long as there isn’t some broken replacement chain(which happens).

However, on a system I have used more which has more packages and a mix of AUR and repo packages, it can be a nightmare if you don’t know what you are doing and how to deal with difficult package management scenarios. There are times when software has to be removed to get things working or you have to be forcible and ignore certain dependencies. That isn’t something most people want to have to deal with even if they have the expertise to do so. Perhaps an even larger problem is, if you do it wrong, you can easily break your system entirely.

On the other hand, I am not saying you need to check for updates every 5 minutes and install them as soon as they arrive. My advice is try to update once per week or so.

Lastly, if you have a critical stability need such as a workstation used by others in a business setting, I recommend not using a rolling distro for that use case. The ability to selectively install updates is a big win for that use case.

While I agree with everything you have said, there is a pretty big caveat here. You have to have enough experience to know which packages you should wait on. Because packages come in a steady endless stream there is no way to hold updates on a generalized basis.

2 Likes

I haven’t encountered that many problems really (at least not ones i couldn’t resolve so far), especially to the point of breaking system entirely.

Not sure what exactly do you mean by broken replacement chain, is it just when you have to remove something beforehand to do successful update?

After using Manjaro Stable it’s not big problem to me :rofl:

You also are an experienced user. Things which you may consider part of updating the system a less experienced user might consider a totally frustrating situation.

Consider a replacement chain where package A gets replaced by package B which then gets replaced by package C. Now package B is removed from the repos and the chain is broken. This doesn’t happen if you update with any degree of frequency but once you get into really long-time windows like 6 months it can/does happen depending on which packages you have installed.

1 Like

image

Yeah i get what you’re saying…But still stuff like .pacnew is necessity on Arch, meaning if you don’t do it - it will backfire. :upside_down_face:

Oh and i get replacement chain problem now, that’s why i keep whole Arch repo downloaded and backed up on my drive, juuuuuuuust in case :shushing_face:

In case of nuclear winter…or no internet…or… :alien:

2 Likes

To be fair, 95% or of .pacnew files are for the mirrorlist, and well… using reflector or equivalent renders that one pointless.

My personal advise is:

  1. Update every two weeks minimum.
  2. Pop in here and see if there’s a forum post about problems with an update before doing it (this is the main reason why NOT to do an update daily)
  3. Installing the LTS kernel as a backup might be a good idea, so you have two options to get into the system if needed.
1 Like

Sometimes those .pacnew files bring important changes. That’s why it is a good idea to check them at times with pacdiff (I prefer eos-pacdiff which does the same in an easy way).

2 Likes

Just a thought here about ‘safely’ updating an Arch system left to its own devices for 6 months. Try updating using the Archive (set a date and use -Syyuu) and step through them 1 month (perhaps) at a time. Much less likely to break any chains that way!

I am successfully (so far) using this trick another way for updating to a week previous only on a mirror server I run on EnOS. The week delay will usually sort out any gotchas from the leading edge… :grin:

BTW - check the EndeavourOS wiki for an easy way to use the archives :grin: It’s under the pacman category…

https://discovery.endeavouros.com/pacman/easy-downgrade-by-date/2021/06/

I don’t run an lts kernel. I just keep rolling! This is Arch BTW. :laughing:

1 Like