Safesync Resurrected - mirrorlist admin for experts - feedback is appreciated

petsam · June 2, 2022, 6:41pm

Safesync wants to be yet another Archlinux Repositories Administrator, for more advanced usage, especially useful for experts. Edit (2022-07-09T21:00:00Z): I came to believe it is also good for not experts, but this has to be confirmed. A programmer never knows how others interact with their code…

Feedback, is appreciated.

Why Ressurected? - History (updated)

Once upon a time, safesync was born, to overcome a distribution’s own utility, with which I had several disagreements on the functionality and usability. It was a personal bash self educating project, which served its purpose (at least for one user ). After being renamed to m1ms, it was left to silently disappear, while it helped me walk away from that other distribution

In the end, after more experience, I realized that I was wrong on a critical part of how pacman updates and mirrors work. Nevertheless, I now know a lot more about this important subject, and I had a nice opportunity to learn more on bash programming, which was my primary goal. So, I won in the end

What can safesync do?

Creates a pacman mirrorlist from known providers’ repositories
Prompts for custom countries selection (with a menu, when rofi is installed)
Save the new mirrorlist file at the default system path (/etc/pacman.d/), or in the invoking user’s local respective folder ($HOME/.local/share/safesync/mirrorlists/), or in a custom path.
Rate the servers of a given mirrorlist file and sort them by speed.
Adds a file heading with info about the creation and modification time, as well as the command and parameters that were used to create/modify the list.

What can safesync do that is different or better than rankmirrors, or reflector?

It creates a new mirrorlist and sorts the servers, from known providers’ repositories, using the upstream published url, or a .pacnew file. Currently, there is information for Archlinux, Archlinux ARM, EndeavourOS, Chaotic-AUR, Arcolinux, while it is easily extendable for other repositories.
It is compatible with Arch Linux ARM.
It also checks any servers Included in pacman.conf
NEW: Toggle individual servers enabled/disabled, selecting with rofi
Better speed ranking, in cases where the mirrorlist serves more than one repository (i.e. extra or community for Archlinux mirrorlist), as it can download a larger file, depending on the given repository, using <$repo>.files of the selected $repo.
It does better speed ranking, as it subtracts the connection time from the total download time, leading to a more accurate actual download speed, especially useful on secure protocols (HTTPS).
It can check remote repo timestamp and disable servers that are synced before the local repository DB timestamp.
It saves activity logs and debug info in temporary log files, that can be reviewed, for better understanding of servers behavior, or for bug tracking investigation.
It can print normal, extra, or no activity information during runtime usage.
It uses normal user privileges, elevating them only if needed, to save at a system path.
It can be used in scripted mode, where no check for user privileges is accomplished.
It does not modify or save system mirrorlists, if pacman lock exists, saving in a user folder instead.
It keeps backups from previous files, when they are modified.

Known issues - TODOs

Create configuration file for custom defaults and settings.
Use rsync for check and download, to make it rsync compatible.
Check pacman DB lock, when saving at system path
Remove rofi dependency
Use a country selection method with bash/cli function
Add parameters or configuration file options on --verbose, --prefer-secure-protocol, --max-timeout
Add pacman sync functionality, that could replace the standard update command (pacman -Syu)
Set a lock for system save actions, until exit.
Set trap(s) for errors to clear temp folder and lock.

Install the latest version 0.2.6 using the included PKGBUILD.

git clone https://gitlab.com/petsam/safesync.git
cd safesync
makepkg -si

sradjoker · June 2, 2022, 7:19pm

Link shows page not found.

Edit:
Is it possible to add archarm repos?

petsam · June 2, 2022, 7:23pm

I don’t know why. It shows up when I click. Could it be private to me only?..
https://gitlab.com/petsam/safesync

I only need a url to a raw mirrorlist file and the repo name (as in pacman.conf)

sradjoker · June 2, 2022, 7:28pm

The way archlinuxarm mirrors work is a bit different.
The default mirror is a global load balancing mirror and the other mirrors are normal mirrors like in archlinux. I currently manually uncomment the mirrors that are geographically located closest to me.

## Geo-IP based mirror selection and load balancing
#Server = http://mirror.archlinuxarm.org/$arch/$repo

### Mirrors by country

### Australia (not Austria!)
## Sydney
# Server = http://au.mirror.archlinuxarm.org/$arch/$repo

### Brazil
## Sao Paulo
# Server = http://br.mirror.archlinuxarm.org/$arch/$repo

### Denmark
## Aalborg
# Server = http://dk.mirror.archlinuxarm.org/$arch/$repo

### Germany
## Aachen
# Server = http://de3.mirror.archlinuxarm.org/$arch/$repo
## Berlin
# Server = http://de.mirror.archlinuxarm.org/$arch/$repo
## Coburg
# Server = http://de4.mirror.archlinuxarm.org/$arch/$repo
## Falkenstein
# Server = http://eu.mirror.archlinuxarm.org/$arch/$repo
# Server = http://de5.mirror.archlinuxarm.org/$arch/$repo

### Greece
## Athens
# Server = http://gr.mirror.archlinuxarm.org/$arch/$repo

### Hungary
## Budapest
# Server = http://hu.mirror.archlinuxarm.org/$arch/$repo

### Japan
## Tokyo
# Server = http://jp.mirror.archlinuxarm.org/$arch/$repo

### Singapore
# Server = http://sg.mirror.archlinuxarm.org/$arch/$repo

### South Africa
## Johannesburg
# Server = http://za.mirror.archlinuxarm.org/$arch/$repo

### Taiwan
## Hsinchu
# Server = http://tw2.mirror.archlinuxarm.org/$arch/$repo
## New Taipei City
# Server = http://tw.mirror.archlinuxarm.org/$arch/$repo

### United Kingdom
## London
# Server = http://uk.mirror.archlinuxarm.org/$arch/$repo

### United States
## California
#Server = http://ca.us.mirror.archlinuxarm.org/$arch/$repo
## Florida
# Server = http://fl.us.mirror.archlinuxarm.org/$arch/$repo
## Illinois
#Server = http://il.us.mirror.archlinuxarm.org/$arch/$repo
## New Jersey
#Server = http://nj.us.mirror.archlinuxarm.org/$arch/$repo

Anything to help me choose the best mirrors for me would be great. Looking forward to your project!!

keybreak · June 2, 2022, 7:28pm

Yep, you need to set project to public

Settings → General → Visibility, project features, permissions

petsam · June 2, 2022, 7:30pm

How the … do I do that?
I hate smart tools!!!

petsam · June 2, 2022, 7:35pm

That’s OK. A URL to download this within the program, is one of the advanced features. Is there a public link for this file?

ricklinux · June 4, 2022, 3:38pm

Link works for me.

petsam · June 15, 2022, 10:46pm

After 11 days of no negative feedback, I am proud to pronounce safesync as a bug-free program, in par to pacman!!!

Just joking, of course. We all know pacman has some small bugs.

Thankfully, we have one but excellent tester! Thanks a lot @sradjoker . I hope Arch-ARM make use of this (soon…).

sradjoker · June 22, 2022, 2:08pm

Do you have instructions to use it for EndeavourOS repos?

petsam · June 22, 2022, 2:31pm

Everything is in the help message.
In short,

Use --init to create a complete mirrorlist with a lot or all the servers:

safesync --all-countries -10 -t system -i endeavouros

If you have rofi installed, you can select countries:

safesync -10 -t system -i endeavouros

That’s all!

Only if you have download problems and suspect a server is down, you can run --safe to re-evaluate the current mirrorlist, without any speed test:

safesync -10 -t system -s endeavouros

Only if the top server looks like it has degraded its speed, you can run --rate to rate the current mirrorlist’s servers:

safesync -10 -t system -r endeavouros

Any time safesync modifies a system mirrorlist file (/etc/pacman.d/), it creates a backup of the pre-existing file, at the same directory.
Any time safesync modifies a local mirrorlist file (~/.local/share/safesync/mirrorlists), it creates a backup of the pre-existing file, at the same directory.

petsam · June 30, 2022, 9:28am

A new version (0.2.4) is up.

Add a bash function country selection
Add a TODO list in README

Feedback is welcome and much appreciated.

petsam · July 10, 2022, 7:21pm

New version 0.2.6 is out!
Several improvements and additions.
Install with:

git clone https://gitlab.com/petsam/safesync.git
cd safesync
makepkg -si

Updated help message, for a quick idea:

Creates, sorts by speed and changes mirror servers safely, for any defined repository
in archlinux/pacman based Linux systems.

  Usage:
  safesync [-h|--help] [-V|--version]
  safesync [OPTIONS] ACTION [LIST]

  The ACTION should always be after all OPTIONS, followed by optional LIST REPOs and/or FILEs
  Read below for explanation of proper usage

  -h|--help : Show this help message
  -V|--version : Print program name and version


  ACTION
  -i |--init : (Re)Initialize the REPO|FILE mirrorlist(s)
        Default: REPO=extra
  -r |--rate : Rate and sort (by fastest) repository servers
        Default: REPO=extra
  -s |--safe : Ensure the active servers are "safe" to sync (synced at a later date than local DB)
        Default: REPO=archlinux (or FILE=/etc/pacman.d/mirrorlist)
  -g |--toggle : Toggle selected servers in a mirrorlist from enabled to disabled and vice versa
        Default: REPO=archlinux (or FILE=/etc/pacman.d/mirrorlist). It requires rofi

  LIST
  A LIST of repositories and/or mirrorlist files to use with ACTION, in this form: [REPO|FILE [REPO|FILE] ...]
  You can mix REPOs and FILEs, if you wish to.
  If LIST is ommited, a default is used. All repositories that use the same mirrorlist,
  are checked and tested, except when using --toggle.

  REPO repository must be defined in pacman.conf. "all" acts on all defined repositories.
  For mirrorlists that serve more than one REPO, you can use any of the known repo names.
  The reason to use a repo instead of a mirrorlist file is the size of the file that will be downloaded.
  For example, with Archlinux "extra" REPO, a larger file is downloaded during speed ranking, producing
  more accurate real world results, than using "core".
  For convenience, we include aliases for known providers, which then use a default REPO per repository.
  "arch" and "archlinux" is equivalent to REPO=extra.
  "archarm" and "archlinuxarm" is equivalent to REPO=extra.
  "arco" and "arcolinux" is equivalent to REPO=arcolinux_repo.
  "chaotic" is equivalent to REPO=chaotic-aur.

  FILE can be absolute, or relative. When a FILE is given, the file name (basename) is used as REPO name.
  For example, when the FILE is a valid REPO name in the current directory, like with

safesync -i extra

  will first check for an existing file "$PWD/extra".
  If the file is found, it will use the entry as a mirrorlist file with REPO="extra" .
  If the file is not found, it will then use the entry as a REPO. So, when the FILE is absolute,
  or in a relative folder, it cannot be a valid REPO name and thus, it will be skipped.

  OPTIONS
  -p |--pacnew : Use pacnew files with --init, instead of fetching upstream list
  -c |--conf FILE : Use custom configuration file FILE, instead of the default
        (Default: FILE=/etc/pacman.conf)
  -NUM |--enable [NUM] : Enable top/first NUM servers in REPO mirrorlist
        (Default: NUM=3)
  -t |--target [TARGET] : Save the output list to TARGET
        (Default: prompt for user input. See bellow for more details)
  --no-conf-servers : Do not check for included Servers directly in pacman.conf
        WARNING: There are usually no Servers in pacman.conf, but it is UNSAFE to use it, in case there are
  --no-safe-test : Do not check servers for sync status, with --init and --rate
  --all-countries : With --init, use all countries in the mirrorlist (do not prompt for selection)
        WARNING: Avoid using all countries with Archlinux repos. There are too many!!
  -x| --scripted : Run without prompting for user input, using parameters, or the defaults
  -q |--quiet : Do not show standard info messages
  -v |--verbose|--debug : Show debug info messages. Used for troubleshooting and bug reports

  TARGET is a PATH or one of the reserved words "system", "local" or "print".
  If "system", file is copied to the pacman.conf defined mirrorlist path.
  If "local", file is copied to safesync local data path ($HOME/.local/share/safesync/mirrorlists).
  If "print", file is printed to terminal output.
  If TARGET is empty, it defaults to $PWD (current working directory/path).
  If --target is not used, it prompts for target, unless --scripted is used which defaults to "print".

  To list all currently configured repositories, run: pacman-conf -l
  If rofi is installed, it will be used for countries selection with --init, and --toggle, or else
  a bash based selection is used.

Thanks to everyone which gave feedback!

petsam · June 7, 2023, 11:51pm

New version 0.2.9 is out, with updated Arch and ArchlinuxARM repos and a PKGBUILD file.

Although there are many mirrorlist creators utilities, systems with mirrorlist problems still continue to appear.

Safesync continues to be successful for 100% of its users!!