Slow online install ranking EndeavourOS mirrors

abcfy2 · January 16, 2022, 4:35pm

Same issue in China.

Update Mirrors only rate archlinux repo, but not for EOS.

Here is the output in livecd (after update mirrors):

$ cat /etc/pacman.d/endeavouros-mirrorlist 
######################################################
####                                              ####
###        EndeavourOS Repository Mirrorlist       ###
####                                              ####
######################################################
#### Entry in file /etc/pacman.conf:
###     [endeavouros]
###     SigLevel = PackageRequired
###     Include = /etc/pacman.d/endeavouros-mirrorlist
######################################################
### Tip: Use the 'eos-rankimirrors' program to rank
###      these mirrors or re-order them manually.
######################################################

## Germany
Server = https://mirror.alpix.eu/endeavouros/repo/$repo/$arch
Server = https://de.freedif.org/EndeavourOS/repo/$repo/$arch
Server = https://mirror.moson.org/endeavouros/repo/$repo/$arch

## Sweden
Server = https://ftp.acc.umu.se/mirror/endeavouros/repo/$repo/$arch
Server = https://mirror.linux.pizza/endeavouros/repo/$repo/$arch

## Canada
Server = https://ca.gate.endeavouros.com/endeavouros/repo/$repo/$arch

## China
Server = https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/$repo/$arch

## Singapore
Server = https://mirror.jingk.ai/endeavouros/repo/$repo/$arch
Server = http://mirror.jingk.ai/endeavouros/repo/$repo/$arch

## France
Server = https://endeavour.remi.lu/repo/$repo/$arch

$ cat /etc/pacman.d/mirrorlist
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --verbose -c CN --protocol https --sort rate --latest 10
# When:       2022-01-16 15:38:04 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-01-16 15:37:52 UTC
# Last Check: 2022-01-16 15:24:08 UTC

Server = https://mirrors.bfsu.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.sjtug.sjtu.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.aliyun.com/archlinux/$repo/os/$arch
Server = https://mirrors.hit.edu.cn/archlinux/$repo/os/$arch

And livecd chroot:

$ cat /tmp/calamares-root-vxdcm_q9/etc/pacman.d/endeavouros-mirrorlist 
######################################################
####                                              ####
###        EndeavourOS Repository Mirrorlist       ###
####                                              ####
######################################################
#### Entry in file /etc/pacman.conf:
###     [endeavouros]
###     SigLevel = PackageRequired
###     Include = /etc/pacman.d/endeavouros-mirrorlist
######################################################
### Tip: Use the 'eos-rankimirrors' program to rank
###      these mirrors or re-order them manually.
######################################################

## Germany
Server = https://mirror.alpix.eu/endeavouros/repo/$repo/$arch
Server = https://de.freedif.org/EndeavourOS/repo/$repo/$arch
Server = https://mirror.moson.org/endeavouros/repo/$repo/$arch

## Sweden
Server = https://ftp.acc.umu.se/mirror/endeavouros/repo/$repo/$arch
Server = https://mirror.linux.pizza/endeavouros/repo/$repo/$arch

## Canada
Server = https://ca.gate.endeavouros.com/endeavouros/repo/$repo/$arch

## China
Server = https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/$repo/$arch

## Singapore
Server = https://mirror.jingk.ai/endeavouros/repo/$repo/$arch
Server = http://mirror.jingk.ai/endeavouros/repo/$repo/$arch

## France
Server = https://endeavour.remi.lu/repo/$repo/$arch

## South Korea
Server = https://mirror.funami.tech/endeavouros/repo/$repo/$arch

$ cat /tmp/calamares-root-vxdcm_q9/etc/pacman.d/mirrorlist
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --verbose -c CN --protocol https --sort rate --latest 10
# When:       2022-01-16 15:38:04 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-01-16 15:37:52 UTC
# Last Check: 2022-01-16 15:24:08 UTC

Server = https://mirrors.bfsu.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.sjtug.sjtu.edu.cn/archlinux/$repo/os/$arch
Server = https://mirrors.aliyun.com/archlinux/$repo/os/$arch
Server = https://mirrors.hit.edu.cn/archlinux/$repo/os/$arch

And I find when installer download EOS packages very very slow.

I think update mirrors should also rate EOS repo.

By the way, why eos-rankimirrors not exists ?

endeavouros-mirrorlist shows:

######################################################
### Tip: Use the 'eos-rankimirrors' program to rank
###      these mirrors or re-order them manually.
######################################################

But why:

$ eos-rankimirrors
bash: eos-rankimirrors: command not found

Thanks.

ricklinux · January 16, 2022, 8:46pm

Command is not spelled correctly!

Edit: eos-rankmirrors

manuel · January 16, 2022, 10:12pm

Thanks for pointing out this typo!
I’ll fix it as soon as possible.

Note that you can also manually re-order the mirrorlist, simply move the best mirror for your location as first.

abcfy2 · January 16, 2022, 11:54pm

Thanks, but how to do this with online installer ?

joekamprad · January 17, 2022, 2:29am

The EndeavourOS repo holds only some small packages and as I can see your arch-mirrorlist holds only 5 mirrors all from china as far as I can see.

from all-rounded 200 packages may 15 will be fetched from EndeavourOS repository:

I would try to use “ranking mirrors” before starting install process from the welcome app, and add some nearby countries plus USA and Germany to the rank so that you make sure to have some faster mirrors in the list for archrepositories.

We will indeed add ranking for our Mirrorlist in the future, as the number is rising.
To sort the list for online installs you would need to change it inside the chroot … possible but not easy to do.

abcfy2 · January 17, 2022, 2:34am

Yes. If online installer will also rate and ranking EOS repo should also be great.

I checked installer log, and I find there is a job like:

    ..  Job 12 "initialize pacman ... ranking mirrors ... copy pacman mirrorlist and keyring to target ..." +wt 1 tot.wt 4

But this job only ranking arch repo, not ranking EOS repo.

I think if this job could ranking EOS repo would be great.

joekamprad · January 17, 2022, 2:42am

/calamares/modules/shellprocess_initialize_pacman.conf

https://raw.githubusercontent.com/endeavouros-team/EndeavourOS-calamares/main/calamares/modules/shellprocess_initialize_pacman.conf

is where installer is ranking…
you could add there to copy the /etc/pacman.d/endeavouros-mirrorlist to target:

sudo xed /etc/calamares/modules/shellprocess_initialize_pacman.conf

- "cp /etc/pacman.d/endeavouros-mirrorlist @@ROOT@@/etc/pacman.d/"

after resorting it or ranking on host (livesession)

abcfy2 · January 17, 2022, 2:48am

Great. And I find this job will execute script bash /etc/calamares/scripts/update-mirrorlist.

Maybe we can make this script also ranking endeavouros-mirrorlist, is it possible ?

joekamprad · January 17, 2022, 2:51am

we have a script to rank EndeavourOS mirrors called eos-rankmirrors you can call it with:
sudo eos-rankmirrors this will rank the mirrorlist on live-session but without copy it over to the install target it will not be used on install process.

abcfy2 · January 17, 2022, 2:56am

So we just add this job to the install process will solve this issue, right ?

joekamprad · January 17, 2022, 3:00am

Would be nice to see the endeavouros-mirrorlist from your side after ranking:
cat /etc/pacman.d/endeavouros-mirrorlist | eos-sendlog from live session after using sudo eos-rankmirrors

joekamprad · January 17, 2022, 3:02am

If the issue is to rank EndeavourOS mirrors yes adding the copyover to target and ranking EndeavourOS mirrors before starting install would be the same as what is done with Archlinux mirrrors.

abcfy2 · January 17, 2022, 3:05am

Yes eos-rankmirrors works very well: https://clbin.com/b1vwA

And TUNA mirror will be the first in my endeavouros-mirrorlist, and it’s very fast in my environment:

$ wget https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/endeavouros/x86_64/paper-icon-theme-1.5.0.832.aa3e8af-1-any.pkg.tar.zst
--2022-01-17 11:05:13--  https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/endeavouros/x86_64/paper-icon-theme-1.5.0.832.aa3e8af-1-any.pkg.tar.zst
SSL_INIT
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 2402:f000:1:400::2, 101.6.15.130
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|2402:f000:1:400::2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52341928 (50M) [application/octet-stream]
Saving to: ‘paper-icon-theme-1.5.0.832.aa3e8af-1-any.pkg.tar.zst’

paper-icon-theme-1.5.0.832.aa3e8af-1-any.pkg.tar.z 100%[===============================================================================================================>]  49.92M  32.0MB/s    in 1.6s    

2022-01-17 11:05:15 (32.0 MB/s) - ‘paper-icon-theme-1.5.0.832.aa3e8af-1-any.pkg.tar.zst’ saved [52341928/52341928]

joekamprad · January 17, 2022, 3:09am

fetch times

# mirror                                                             update-level  fetch-time
# ~~~~~~                                                             ~~~~~~~~~~~~  ~~~~~~~~~~
# https://mirrors.tuna.tsinghua.edu.cn/endeavouros/repo/$repo/$arch  1197          0.418316
# https://de.freedif.org/EndeavourOS/repo/$repo/$arch                1197          0.643520
# https://mirror.moson.org/endeavouros/repo/$repo/$arch              1197          0.662301
# http://mirror.jingk.ai/endeavouros/repo/$repo/$arch                1197          0.724536
# https://mirror.funami.tech/endeavouros/repo/$repo/$arch            1197          1.093975
# https://mirror.linux.pizza/endeavouros/repo/$repo/$arch            1197          1.197736
# https://endeavour.remi.lu/repo/$repo/$arch                         1197          1.325790
# https://ftp.acc.umu.se/mirror/endeavouros/repo/$repo/$arch         1197          1.756877

theser are mine from Germany
fetchtimes

joekamprad · January 17, 2022, 3:13am

B.T.W welcome at the forum and thanks for providing info and report!

We will integrate the EndeavourOS ranking with the next ISO, may we can add this also by a HotFix.

I will need to get some sleep now… we have already 4 in the morning here

abcfy2 · January 17, 2022, 3:13am

Thank you for your hard working.

system · January 19, 2022, 3:13am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.