Can piclone be used in a RPi installation of Endeavouros?

As this thread seems to have ground to a standstill, I took the step of researching the option of using dd to make a copy, despite coming across a warning that it was unwise to try to copy a running OS. The only precaution I took to try to avoid problems was to disable the internet connection so that incoming messages could not interfere with the process. The result was as follows:

[petern@localhost ~]$ sudo dd if=/dev/mmcblk0 of=/dev/sdb bs=4M status=progress
63864569856 bytes (64 GB, 59 GiB) copied, 2714 s, 23.5 MB/s
15226+1 records in
15226+1 records out
63864569856 bytes (64 GB, 59 GiB) copied, 2735.29 s, 23.3 MB/s
[petern@localhost ~]$

The microSD card (/dev/sdb in the above) booted successfully, and a small number of tests all worked as expected, so the warning about copying a running system doesn’t seem to have been relevant. However, I would be very grateful if anyone with greater skill/experience than I have in systems management could point out any real risks in using this procedure to make a full backup.

Creating a snapshot of a running system is not straight forward.

What you want is technically labelled as a ‘cold backup’.

I think the warnings are related to possible file system inconsistencies where the file system changes as a consequence of e.g. (but not limited to)

  • database transactions
  • log transactions
  • dynamic application configuration
  • shared folder changes
  • mail systems
  • database servers
  • package transactions

No matter the quality of the SD-card it has a limited lifetime, which is proportional to the number of writes. The more you alter the content of an SD-card the shorter lifespan it has.

If you use a reasonable quality and only use the device for e.g. a DNS server - it may run for years without any issues - because the data is only read - rarely changed - example I used to run a bind service of an rpi2, then I moved the card to an rpi3 and the card was in use for 8 years.

But if you run a file server and/or a mail server or next cloud instance - then you will be increasing the number of changes and writes - and logically the lifespan will decrease.

The lifespan of an SD-card is not an exact science - it depends on the type and quality of the memory cells used.

If you want to run a system from an SD-card with a fair amount of data security, then you should ensure that data is stored on a removable USB attached disk or use a casing with built-in disk RPI5 has an nvme interface. Then symlink your datadirs to the nvme.

The aforementioned shrink-backup can run on any system and it can send the backup to any location on your system, including mounted devices or network locations.

Thanks for the run-down of possible problems with a simple dd command

Since the Pi is running as a straightforward single user “desktop” computer, I don’t think there is much risk of these kinds of operation, but I have left the System Monitor running for some time, and the only ‘write’ operations I observe are actually done by Firefox, although on brief occasions a Firefox write seems to trigger a write by plasmashell. I can’t really turn of Firefox while composing this reply, but I would guess that running the dd command when Firefox is not running would be pretty safe.

However, I am keen to try shrink-backup. I have downloaded the shrink-backup-main.zip file, and see that shrink-backup is a bash script. As I am not a systems person, I would welcome your advice on where shrink-backup should be stored, with what permissions, and from which folder it should be invoked.

Yes, the script is pure bash and as such it can be used anywhere you place it (suffice any binaries called by the script is available).

Clone the repo or unpack the zip

git clone https://github.com/UnconnectedBedna/shrink-backup

Change into the folder

cd shrink-backup

The script permissions is default executable, I prefer to run scripts using bash <script-name>.

If you want it in your path I suggest you create the folder ~/.local/bin and add it to your path. Example of adding it to your path as part when loading the system by use of a configuration file named /etc/profile.d/home-local-bin.sh

 $ cat /etc/profile.d/home-local-bin.sh
case ":${PATH}:" in
  *:"$HOME/.local/bin":*) ;;
  *) export PATH="$HOME/.local/bin:$PATH" ;;
esac

The --help snippet is quite informative

[nix@eos-sd shrink-backup]$ bash ./shrink-backup -h
Script for creating an .img file and subsequently keeing it updated (-U), autoexpansion is enabled by default
Directory where .img file is created is automatically excluded in backup
######################################################################################################################
Usage: sudo shrink-backup [options] imagefile.img [extra space (MiB)]
  -U             Update existing img file (rsync to existing img)
                   Optional [extra space] extends img root partition
  -a             Autocalculate root size partition, [extra space] is ignored.
                   When used in combination with -U:
                   Expand if partition is >=256MiB smaller than autocalculated recommended minimum
                   Shrink if partition is >=512MiB bigger than autocalculated recommended minimum
  -t             Use exclude.txt in same folder as script to set excluded directories
                   One directory per line: "/dir" or "/dir/*" to only exclude contents
                   Wildcards work, f.ex "/dir*" will eclude all directories starting with "/dir"
  -y             Disable prompts in script (please use this option with care!)
  -e             Disable autoexpansion on root filesystem when image is booted
  -l             Write debug messages to logfile shrink-backup.log located in same directory as script
  -z             Make script zoom at light-speed, only question prompts might slow it down
                   Can be combined with -y for UNSAFE ultra-mega-superduper-speed
  -q --quiet     Do not print rsync copy process
  --no-color     Run script without color formatted text
  --fix          Try to fix the img file if -a fails with a "broken pipe" error
                   Will activate rsync options --delete-before & --fsync
  --rsync        Define custom rsync line manually. Will print rsync line for user to edit
  --loop [img]   Loop img file and exit, works in combination with -l & -z
                   If optional [extra space] is defined, the img file will be extended with the amount before looping
                   NOTE that only the file gets truncated, no partitions
                   Useful if you for example want to manually manage the partitions
  --chroot [img] Use systemd-nspawn. Loop img file, mount to temp directory, enter chroot environment and drop to shell
                   This will let you make changes in a chroot environment directly on the img file
                   For example update with package manager or rebuild initramfs
                   The script will keep running in the background
                   Type 'exit' when done. Script will unmount, remove temp directory/loop and exit
  --f2fs         Convert root filesystem on img from ext4 to f2fs
                   Only works on new img file, not in combination with -U
                   Will make backups of fstab & cmdline.txt to: fstab.shrink-backup.bak & cmdline.txt.shrink-backup.bak
                   Then change ext4 to f2fs in both files and add discard to options on root partition in fstab
  --version      Print version and exit
  -h --help      Show this help snippet
######################################################################################################################
Examples:
sudo shrink-backup -a /path/to/backup.img (create img, automatically set size)
sudo shrink-backup -e -y /path/to/backup.img 1024 (create img, ignore prompts, do NOT autoexpand, add 1024MiB extra space)
sudo shrink-backup -Utl /path/to/backup.img (update img backup, use exclude.txt and write log to shrink-backup.log)
sudo shrink-backup -U /path/to/backup.img 1024 (update img backup, expand img size/root partition with 1024MiB)
sudo shrink-backup -Ua /path/to/backup.img (update img backup, automatically resizes img file if needed)
sudo shrink-backup -Ua --fix /path/to/backup.img 1024 (update img backup, automatically resizes img file if needed, fix img free space)
sudo shrink-backup -l --loop /path/to/backup.img 1024 (write to log file, expand IMG FILE (not partition) by 1024MiB, loop then exit script)

The backup is an .img file - which can subsequently store on any medium.

From personal experience

  • always use the -t option (see exclude.txt next to the script)
  • store the backup on an attached USB device
  • never backup directly to a mounted network share
    • when final storage is on the network
      • create the image locally
      • move to network location
  • use timestamped image names

Sample script - to be adapted to your system…

#!/usr/bin/env bash

# define variables - prevents human errors
# adapt the variables to your use case
DEV_PART="/dev/sda2"
DEV_MOUNT="/mnt/my-backup"
IMG_NAME="$(hostnamectl hostname)-$(date '+%j.%H%m').img"
# it is also possible to update an existing image
# in that case set the IMG_NAME to the correct filename
# also change the cmdline arguments for shrink-backup 
# - making sure you update instead of replace

# ensure the mount point is empty
if [[ mount =~ $DEV_PART ]]; then
    echo "$DEV_PART mounted... will forcibly unmount now"
    echo "sudo umount --force $DEV_PART"
    sudo umount --force $DEV_PART
else
    echo "$DEV_PART can be mounted..."
fi

# create the folder - ignore if the folder exist by using the `-p` option
echo "sudo mkdir -p $DEV_MOUNT"
sudo mkdir -p $DEV_MOUNT

# mount the partiion 
echo "sudo mount $DEV_PART $DEV_MOUNT"
sudo mount $DEV_PART $DEV_MOUNT

# run shrink-backup 
# - using the exclude file
# - auto calculating the image size
# - suppress the rsync output
echo "sudo bash shrink-backup -t -q -a $DEV_MOUNT/$IMG_NAME"
sudo bash shrink-backup -t -q -a $DEV_MOUNT/$IMG_NAME

# force cache to disk
echo "sync"
sync

# unmount
echo "sudo umount $DEV_PART"
sudo umount $DEV_PART

Hello, I’m the maintainer of shrink-backup. I saw incoming links for endeavouros forum and it made me curious what was being said about it.

Yes, sadly he passed away a while ago, but the project is still maintained by me (twin brother). He left a LOT of notes, documentation, and all of his keys and logins so I have full control and access to all his accounts. It has taken me a while to go through all of the notes.
Fun fact: the idiot actually used an image of ME on his github, so I didn’t have to change anything when taking over his account… :melting_face:
But for you Fredde, you are actually mentioned pretty fondly in it. Might be nice to know. :heart:
Example quote: “linux-aahus is a nice bearded dude from Denmark that has been very helpful on the Manjaro forum”

Moving on from that.
shrink-backup IS maintained, updates are just not as rapid as previously. Mostly if bugs are discovered or if requests for specific distros are made to make sure autoexpansion works as it should.
gpt partition table is for example something in the works to fix, there is a branch that works for gpt that can be used, but I haven’t felt any rush to push it for a new release since I think it is a very small portion of users that actually use gpt for single board computers.
Clone the repo and `git checkout gpt` if you actually use a gpt partition table and it should work.

I also saw someone ask where to place the script.
As @Root mentioned, it works from anywhere, but if you want to aCsHuAlLy be “correct”, it should go into /usr/local/sbin/shrink-backup, then there is no need to edit $PATH since that directory is already in your $PATH.
There is also an alternative installation method that can be used that places everything like it’s a package installed on the system, although I am not a huge fan of running curl into bash, but users requested it so it was created: curl https://raw.githubusercontent.com/UnconnectedBedna/shrink-backup/install/installer.sh | sudo bash
More info can be found here: https://github.com/UnconnectedBedna/shrink-backup/wiki/Installing#curl---shrink-backup-install-script

I would also like to add that running shrink-backup over network should not be an issue since rsync is used in the copy process, so crc checks are made automatically. (I run all my backups to an NFS mount)

So, the github repo is monitored and both issues and questions posted in the discussion section of the github repo will be addressed.

There is also a helper script that can be created (can be found in one of the questions about compression on the discussion section of the github repo), it’s “on-the-list-of-things-to-be-added-to-the-wiki” soon™. xD
It includes compression for those who want that “on the go” and mounts/unmounts backup location. You need to edit the script to fit your setup though (primarily mount location/method, APP & IMG_FILE variables). Name it for example backup.sh, make it executable and run it with one of the options available in the case statement. (or edit/add your own options)

#!/usr/bin/env bash

# This whole function can be removed/commented out if you do not mount the backup location using this script
# This function ensures the backup locations is unmounted if you press n (in script confirmation) or ctrl+c during the script
# Make sure path is correct to where your mount is located
# The space after the path is intentional, ie NOT: '/mnt/backup', but rather: '/mnt/backup ' is REQUIRED
function cleanup() {
  if grep -qs '/mnt/backup ' /proc/mounts; then
    echo '## Unmounting backup mount'
    umount /mnt/backup
  fi
}
trap cleanup EXIT SIGTERM SIGINT

# Check if script is running as root
if [ "$EUID" -ne 0 ]; then
  echo '!! THIS SCRIPT MUST BE RUNNING AS ROOT! (WITH SUDO)'
  exit 1
fi

# If you want the script to mount the backup location, NFS (192.168.1.100 at /mnt/backup) in this example.
# Change to the method you mount or comment out/remove if not wanted
echo '## Mounting backup location'
mount -t nfs 192.168.1.100:/foo/bar /mnt/backup
sleep 1

# Check if the backup location is mounted, comment out/remove if not wanted
# Make sure path is correct to where your mount is located
# The space after the path is intentional, ie NOT: '/mnt/backup', but rather: '/mnt/backup ' is REQUIRED
if ! grep -qs '/mnt/backup ' /proc/mounts; then
  echo 'ERROR! Backup location not mounted!'
  exit 1
fi

APP='/path/to/shrink-backup'
IMG_FILE='/mnt/backup/backup.img'

# Options
case "$1" in
  -a) "$APP" -Utlaz "$IMG_FILE";;
  -N) if [ -n "$2" ]; then
        "$APP" -tlz "$IMG_FILE" "$2"
      else
        "$APP" -atlz "$IMG_FILE"
      fi
      chown 1000:1000 "$IMG_FILE"
      ;;
  --loop) "$APP" -lz --loop "$IMG_FILE";;
  --fix) "$APP" -Utl --fix "$IMG_FILE" "$2";;
  --chroot) "$APP" -lz --chroot "$IMG_FILE";;
  *) "$APP" -Utlz "$IMG_FILE" "$1";;
esac

sleep 1

# Add compression here, change to compression method of your choice by editing the `tar` line
echo '## Compressing backup img'
tar -czvf /mnt/backup/backup.tar.gz "$IMG_FILE"
sleep 1

# Removing img file after compression is done
# If you update the img file, maybe you don't want this
# Or maybe you want to add a decompression of the tar.gz file at the start of this script
rm /mnt/backup/backup.img

exit 0

Make sure the script is executable with chmod u+x /path/to/backup.sh
And then run it with the option you prefer:

    -N = New backup, (-atlz options)
        (optional to set desired extra space here
        for example: ./backup.sh -N 1024)
    -a = Update backup and autoresize if needed (-Utlaz options)
--loop = loop img file
--fix  = Update backup and fix "broken pipe" errors (-Utl --fix options)

Example: sudo /path/to/backup.sh -N will create a new backup at /mnt/backup/backup.img and then compress it into /mnt/backup/backup.tar.gz and finally remove the img file.
Running without any options, ie: sudo /path/to/backup.sh will result in an image update with zoom speed activated (* = -Utlz options)

For detailed information about the script, read the README.md or visit the repo on github.

I sincerely hope you find shrink-backup to your liking.

I am sorry to hear of your loss. :broken_heart:
Welcome to the EndeavourOS forum. I hope you enjoy your time here.

Thank you for your input on shrink-backup

Pudge