Can't update system (broken ssl?)

I updated my system this morning and after reboot; my Welcome message tells me I have no internet connection (I do and most things work), my Steam client has stopped connecting, and I can no longer do a pacman system update.

Here is the output of ‘pacman -Syu’:

sudo pacman -Syu
[sudo] password for beyondlife: 
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
 endeavouros.db failed to download
error: failed retrieving file 'endeavouros.db' from ca.gate.endeavouros.com : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirrors.tuna.tsinghua.edu.cn : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.alpix.eu : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from de.freedif.org : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.moson.org : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from endeavour.remi.lu : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.jingk.ai : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.freedif.org : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.funami.tech : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from ftp.acc.umu.se : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.linux.pizza : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirror.archlinux.tw : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from fastmirror.pp.ua : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed retrieving file 'endeavouros.db' from mirrors.42tm.tech : error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
error: failed to synchronize all databases (download library error)

This led me to check the certificate path which is a symlink to “/etc/ca-certificates/extracted/tls-ca-bundle.pem”. This file appears to be empty.

I tried running ‘update-ca-trust’ and it appears to work correctly. All of the pem files get overwritten but they’re still all 0 byte files.

I tried reinstalling all of the ‘ca-certificates*’ packages using ‘sudo pacman -S $(pacman -Ssq ca-certificates)’ but it doesn’t seem to have made any difference.

Here is a list of the packages that upgraded this morning:

[2022-07-24T09:30:47-0500] [PACKAGEKIT] upgraded gcc-libs (12.1.0-2 -> 12.1.0-3)
[2022-07-24T09:30:47-0500] [PACKAGEKIT] upgraded bind (9.18.4-1 -> 9.18.5-1)
[2022-07-24T09:30:47-0500] [PACKAGEKIT] upgraded ca-certificates-mozilla (3.80-1 -> 3.81-1)
[2022-07-24T09:30:47-0500] [PACKAGEKIT] upgraded fzf (0.30.0-3 -> 0.31.0-1)
[2022-07-24T09:30:48-0500] [PACKAGEKIT] upgraded gcc (12.1.0-2 -> 12.1.0-3)
[2022-07-24T09:30:48-0500] [PACKAGEKIT] upgraded lame (3.100-3 -> 3.100-4)
[2022-07-24T09:30:48-0500] [PACKAGEKIT] upgraded lib32-gcc-libs (12.1.0-2 -> 12.1.0-3)
[2022-07-24T09:30:48-0500] [PACKAGEKIT] upgraded nss (3.80-1 -> 3.81-1)
[2022-07-24T09:30:48-0500] [PACKAGEKIT] upgraded lib32-nss (3.80-1 -> 3.81-1)
[2022-07-24T09:30:50-0500] [PACKAGEKIT] upgraded linux-headers (5.18.13.arch1-1 -> 5.18.14.arch1-1)
[2022-07-24T09:30:50-0500] [PACKAGEKIT] upgraded linux (5.18.13.arch1-1 -> 5.18.14.arch1-1)
[2022-07-24T09:30:50-0500] [PACKAGEKIT] upgraded logrotate (3.19.0-1 -> 3.20.1-1)
[2022-07-24T09:30:50-0500] [PACKAGEKIT] upgraded pacman-mirrorlist (20220605-1 -> 20220724-1)

I’m not sure where else to go with this. Any help would be greatly appreciated.

It shouldn’t be empty.
Does it help to reinstall package ca-certificates-utils?

EDIT: I guess you already tried that…

EDIT2: you could try updating Arch keyring first:
sudo pacman -Sy archlinux-keyring
and then update system.

Note that if endeavouros.db is the only problematic repo then you can temporarily comment it out in /etc/pacman.conf.

This seems like the real issue. Maybe DNS has problems, can you ping a site?

DNS seems to be problem free as far as I can tell.

$ ping www.google.com
PING www.google.com (142.250.190.100) 56(84) bytes of data.
64 bytes from ord37s35-in-f4.1e100.net (142.250.190.100): icmp_seq=1 ttl=119 time=10.3 ms
64 bytes from ord37s35-in-f4.1e100.net (142.250.190.100): icmp_seq=2 ttl=119 time=10.6 ms
64 bytes from ord37s35-in-f4.1e100.net (142.250.190.100): icmp_seq=3 ttl=119 time=11.4 ms
64 bytes from ord37s35-in-f4.1e100.net (142.250.190.100): icmp_seq=4 ttl=119 time=11.1 ms

Most of my applications are still working normally; Discord, Thunderbird, Chrome, Firefox, etc.

Are there any other issues than updating the system (especially the endeavouros.db file)? If so, can you identify them? The more details, the better. :wink:

E.g. a journal of the situation when the problem occurs.

A little short on good details I’m afraid but Steam will not connect at all and just gives me a generic “Connection Error”. There is a toast notification welcome message that pops up briefly after the firewall activation prompt that says I’m not connected to the internet, aside from those 2 things and the endeavouros.db file not downloading, I haven’t noticed anything else just yet.

OK. I’d try temporarily commenting out the [endeavouros] repo in /etc/pacman.conf and then update system. That could either work or reveal other issues.

Commenting out the three EndeavourOS lines in the pacman.conf file did allow a system update to complete. Packages that updated were pretty inconsequential though; gsm, protobuf, & imagemagick.

I also noticed your previous comment about the archlinux-keyring. With those lines commented, I was also able to reinstall that package but no change in behaviour.

Enabling those lines again and attempting a another system update, the same errors occur.

Edit: Thanks for all your help thus far. I’ve been dealing with this all day and I’m really at my wits’ end. Also, I just realized it’s 3am here and I need to get some sleep. If you come up with anything else, I’ll test it later this morning.

1 Like

Please show the contents of your /etc/pacman.d/endeavouros-mirrorlist, to see if there are any problems.

Another thing to try, if possible, is

sudo pacman -Syyu

(note the double y).

And please show your “foreign” packages: pacman -Qm

######################################################
####                                              ####
###        EndeavourOS Repository Mirrorlist       ###
####                                              ####
######################################################
#### Entry in file /etc/pacman.conf:
###     [endeavouros]
###     SigLevel = PackageRequired
###     Include = /etc/pacman.d/endeavouros-mirrorlist
######################################################
### Tip: Use the 'eos-rankmirrors' program to rank
###      these mirrors or re-order them manually.
######################################################

## Canada
Server = https://ca.gate.endeavouros.com/endeavouros/repo/$repo/$arch

## China
Server = https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/$repo/$arch

## Germany
Server = https://mirror.alpix.eu/endeavouros/repo/$repo/$arch
Server = https://de.freedif.org/EndeavourOS/repo/$repo/$arch
Server = https://mirror.moson.org/endeavouros/repo/$repo/$arch

## France
Server = https://endeavour.remi.lu/repo/$repo/$arch

## Singapore
Server = https://mirror.jingk.ai/endeavouros/repo/$repo/$arch
Server = https://mirror.freedif.org/EndeavourOS/repo/$repo/$arch

## South Korea
Server = https://mirror.funami.tech/endeavouros/repo/$repo/$arch

## Sweden
Server = https://ftp.acc.umu.se/mirror/endeavouros/repo/$repo/$arch
Server = https://mirror.linux.pizza/endeavouros/repo/$repo/$arch

## Taiwan
Server = https://mirror.archlinux.tw/EndeavourOS/repo/$repo/$arch

## Ukraine
Server = https://fastmirror.pp.ua/endeavouros/repo/$repo/$arch

## Vietnam
Server = https://mirrors.42tm.tech/endeavouros/repo/$repo/$arch

That reminds me of another issue. I tried to ‘pacman -S endeavouros-mirrorlist’ to make sure everything was up to date there and that led to this:

warning: endeavouros-mirrorlist-4.6-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)                         Old Version  New Version  Net Change

endeavouros/endeavouros-mirrorlist  4.6-1        4.6-1          0.00 MiB

Total Installed Size:  0.00 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                 [----------------------------------] 100%
(1/1) checking package integrity                               [----------------------------------] 100%
(1/1) loading package files                                    [----------------------------------] 100%
(1/1) checking for file conflicts                              [----------------------------------] 100%
:: Processing package changes...
(1/1) reinstalling endeavouros-mirrorlist                      [----------------------------------] 100%
:: Running post-transaction hooks...
(1/3) Refreshing PackageKit...
(2/3) Hook to rank EndeavourOS mirrors after installing or upgrading the related mirrorlist package
eos-rankmirrors: error: internet connection not available!
(3/3) Checking which packages need to be rebuilt

‘sudo pacman -Syyu’ leads to the same errors as ‘sudo pacman -Syu’ but I get to see some progress bars for the other categories first.

Foreign Packages (Pretty innocuous stuff really):

pacman -Qm
atlauncher-bin 3.4.19.0-1
google-chrome 103.0.5060.114-1
latte-dock-git r5957.2e252b54-1
mangohud 0.6.7.1-4
mangohud-common 0.6.7.1-4
openrgb-bin 0.7-3
powershell-bin 7.2.5-1
thunderbird-beta-bin 102.0b8-1
vscodium-bin 1.69.1-1

Ok, I’m really going to bed this time. Thanks for all the help.

1 Like

Now that you now you have a connection, you could temporarily uninstall eos-rankmirrors and try again.

Another thing: write a bash file /usr/local/bin/eos-connection-checker and make it simply exit with 0:

#!/bin/bash
exit 0

This makes it not to check internet connection.

Have you run out of disk space?

Definitely not out of diskspace. Have about 700GB to spare on the primary disk.

I temporarily removed eos-rankmirrors and eos-mirrorlist reinstalled successfully. After creating the requested batch file and reinstalling eos-rankmirrors, I tried reinstalling eos-mirrorlist and I am seeing a different error message because it skips the online check.

pacman -S endeavouros-mirrorlist
warning: endeavouros-mirrorlist-4.6-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)                         Old Version  New Version  Net Change

endeavouros/endeavouros-mirrorlist  4.6-1        4.6-1          0.00 MiB

Total Installed Size:  0.00 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                 [----------------------------------] 100%
(1/1) checking package integrity                               [----------------------------------] 100%
(1/1) loading package files                                    [----------------------------------] 100%
(1/1) checking for file conflicts                              [----------------------------------] 100%
:: Processing package changes...
(1/1) reinstalling endeavouros-mirrorlist                      [----------------------------------] 100%
:: Running post-transaction hooks...
(1/3) Refreshing PackageKit...
(2/3) Hook to rank EndeavourOS mirrors after installing or upgrading the related mirrorlist package
====> Fail: no response: mirror https://mirror.alpix.eu/endeavouros/repo/$repo/$arch, url https://mirror.alpix.eu/endeavouros/repo/state,
==> Creating /root/.config/curl-exit-code-to-string/curl-code-to-string-converter
====> Error code 77: 'Problem reading the SSL CA cert (path? access rights?).'
(3/3) Checking which packages need to be rebuilt

I was able to fix this temporarily by creating an EndeavourOS VM and copying the files from “/etc/ca-certificates/extracted/” to the same location on my local machine. However, if I run ‘update-ca-trust’, it wipes all of those files back to zeros. Unless anyone has other ideas, I guess I’ll just keep those files handy and wait for another future package update to maybe fix what has gone wrong.

1 Like

Great idea with the VM! :+1:
Now it gives the opportunity to check the real reason why the certificates get wiped.

What comes to mind now:

  • pacman log
  • journal about the the time during system update
  • check which packages (might be a conflict with some AUR package) own those certificate files, like: pacman -Qo <filename>

What’s the output of trust

1 Like

What about in the root partition? Just to verify, what’s the output of df -h ?

1 Like

Sounds like a different problem. I suggest you create another thread about this, because likely the fix will be different.

1 Like

Just ‘trust’ by itself gives no information since the command requires flags. Have a more specific command?

trust
usage: trust command <args>...

Common trust commands are:
  list             List trust or certificates
  extract          Extract certificates and trust
  extract-compat   Extract trust compatibility bundles
  anchor           Add, remove, change trust anchors
  dump             Dump trust objects in internal format

See 'trust <command> --help' for more information