Moving to Endeavour & dealing with current partition setup

General system Installation
#22

Wow, that’s a lot of drives! Honestly can’t give you advice there, sorry, I am still learning myself.

I have narrowed down why I am not seeing any btrfs snapshots in grub. When I run grub-mkconfig, I see this at the end of the output:

### BEGIN /etc/grub.d/41_snapshots-btrfs ###
###### - Grub-btrfs: Snapshot detection started - ######
# Info: Separate boot partition not detected 
# No snapshots found.
# If you think an error has occurred , please file a bug report at " https://github.com/Antynea/grub-btrfs "
# Nothing to do. Abort.
###### - Grub-btrfs: Snapshot detection ended   - ######
### END /etc/grub.d/41_snapshots-btrfs ###
done

@2000, perhaps you might have some insight into this? I can confirm there are snapshots present in /run/timeshift/backup/

#23

This is what i get but i think @2000 will have to advise you on that as he is the expert here.

[ricklinux@eos-xfce ~]$ sudo grub-mkconfig -o /boot/grub/grub.cfg
[sudo] password for ricklinux: 
Generating grub configuration file ...
Found theme: /boot/grub/themes/EndeavourOS/theme.txt
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
###### - Grub-btrfs: Snapshot detection started - ######
# Info: Separate boot partition not detected 
# Found snapshot: 2020-08-22 10:16:28 | timeshift-btrfs/snapshots/2020-08-22_10-16-28/@
# Found snapshot: 2020-08-22 09:19:46 | timeshift-btrfs/snapshots/2020-08-22_09-19-46/@
# Found snapshot: 2020-08-21 20:22:44 | timeshift-btrfs/snapshots/2020-08-21_20-22-44/@
# Found snapshot: 2020-08-21 20:10:20 | timeshift-btrfs/snapshots/2020-08-21_20-10-20/@
# Found snapshot: 2020-08-21 14:22:27 | timeshift-btrfs/snapshots/2020-08-21_14-22-27/@
# Found snapshot: 2020-08-21 13:37:02 | timeshift-btrfs/snapshots/2020-08-21_13-37-02/@
# Found 6 snapshot(s)
###### - Grub-btrfs: Snapshot detection ended   - ######
done
[ricklinux@eos-xfce ~]$
1 Like
#24

Hey, @ricklinux just out of curiosity, would you mind checking what the output of

cat /etc/timeshift.json

Is on your system?

#25 
[ricklinux@eos-xfce ~]$ cat /etc/timeshift.json
{
  "backup_device_uuid" : "0901c653-3f70-4f30-9ab9-f13818e07187",
  "parent_device_uuid" : "efc1098d-46e9-4d34-b4bf-a1b77b3a2361",
  "do_first_run" : "false",
  "btrfs_mode" : "true",
  "include_btrfs_home_for_backup" : "false",
  "include_btrfs_home_for_restore" : "false",
  "stop_cron_emails" : "true",
  "schedule_monthly" : "false",
  "schedule_weekly" : "false",
  "schedule_daily" : "true",
  "schedule_hourly" : "false",
  "schedule_boot" : "true",
  "count_monthly" : "2",
  "count_weekly" : "3",
  "count_daily" : "5",
  "count_hourly" : "6",
  "count_boot" : "3",
  "snapshot_size" : "0",
  "snapshot_count" : "0",
  "exclude" : [
  ],
  "exclude-apps" : [
  ]
}
[ricklinux@eos-xfce ~]$
1 Like
#26

Did you miss step #6

#27

Huh, ok, so I see some differences there. Did you configure timeshift separately afterwards, maybe for another type of backup?

This is my output:


{
  "backup_device_uuid" : "313dc380-0d06-44ec-8652-062bd60bd5c6",
  "parent_device_uuid" : "",
  "do_first_run" : "false",
  "btrfs_mode" : "false",
  "include_btrfs_home_for_backup" : "false",
  "include_btrfs_home_for_restore" : "false",
  "stop_cron_emails" : "true",
  "btrfs_use_qgroup" : "true",
  "schedule_monthly" : "false",
  "schedule_weekly" : "false",
  "schedule_daily" : "false",
  "schedule_hourly" : "false",
  "schedule_boot" : "false",
  "count_monthly" : "2",
  "count_weekly" : "3",
  "count_daily" : "5",
  "count_hourly" : "6",
  "count_boot" : "5",
  "snapshot_size" : "5711986393",
  "snapshot_count" : "226379",
  "date_format" : "%Y-%m-%d %H:%M:%S",
  "exclude" : [
    "/home/magtuired/**"
  ],
  "exclude-apps" : [
  ]

To my knowledge, this what is constructed in step #6 of the tutorial, though mine would have been the unencrypted one posted by 2000 in another thread.

It could be that somehow in modifying #6 for the unencrypted setup I messed it up.

#28

I have the encrypted setup because every time i have tried the unencrypted i just didn’t get it. It seemed to give me errors.

Edit: Probably because i was doing something wrong.

#29

Is it working right? I want to do the same install unencrypted but i haven’t yet.

#30

Nope, still haven’t figured out what’s going wrong with the creation of Grub entries for snapshots. I’m still digging around, but I think I might need insight from somebody more experience.

I also noticed for some reason I still have the following entry in Grub:

/boot/efi/EFI/Manjaro//boot/efi/EFI/Manjaro

I also have the two sperate folders under /boot/efi/EFI, namely:

boot and EndeavourOS

Not sure if that’s routine or residue from a sloppy install. Hope I can figure this out, would be a pain to have to start over again. :sweat_smile:

#31

Well @2000 is the user who created the wiki so hopefully he can help you figure it out since you were going from another setup too. I don’t know much about it really. I understand the concept but not the intricacies of how it works and how it’s set up exactly. The manjaro is probably left over nvram as EFI keeps a list.

Edit: I have had separate /boot partitions before but it’s not required. I don’t bother anymore much.

1 Like
#32

Ok, so I found that my entry for /boot/grub/grub-btrfs.cfg is blank, hmm…

#33

Which is the parent device?

#34

Haha, sorry, I copied and pasted that accidentally. I was running diff on our versions of timeshift.json and had highlighted that. My partent_device_uuid is blank in timeshift.json too.

#35

[EDIT] See post below!

@magtuired @ricklinux
Hi guys,

no idea what went wrong. :pleading_face:
There have been several requests for an unencrypted guide so I quickly adapted the commands. In theory this should work but to be honest I haven’t actually tested it in real life. I normally only do encrypted. :wink:
I’ll fire up a VM and try to figure it out later today.

In the meantime …

  1. Did you try running Settings from inside Timeshift? Just choose the correct Location.
    This overwrites anything that might have been set incorrectcly during installation.
    Make a snapshot, run sudo grub-mkconfig -o /boot/grub/grub.cfg, reboot and check for grub entries.

  2. You wrote you made several snapshots. Did you actually try to restore one of them? If this works correctly then the problem may be grub-btrfs and not timeshift. On the other hand, restoring a defect snapshot may mess up the system. Your call …

#36

OK, just tried out the unencrypted install and it works as expected. :relieved:

I assume you didn’t choose to run #07 from the guide.

By default btrfs-grub only seems to update grub (by hook) after a system update. It doesn’t run if you invoked the creation of a snapshot manually or if Timeshift creates them by schedule.
That’s why I wrote a systemd service (#07) that deletes old snapshots and runs grub-mkconfig on shutdown.
It’ll slow down shutdown by a few seconds but I can live with that.

So, you could try creating that service and reboot (a couple of times) and your snapshots should show up in the boot menu eventually.

[Edit] Regarding my former post: Timeshift was set up correctly; no need to do this manually from inside Timeshift.

[Edit 2] I’ve been meaning to change that shutdown service to a path unit that monitors the snapshot folder for changes but haven’t gotten to it.

#37

Sorry to take so much of your time! I don’t want to be a help vampire or anything.

Hmm, I believe I did - this is what I have in /etc/systemd/system/limit-timeshift-snapsots-update-grub.service:

Description=Keep only latest 15 timeshift snapshots and update grub.cfg before shutdown
Before=poweroff.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecStop=/usr/bin/sh -c "cd /run/timeshift/backup/timeshift-btrfs/snapshots/ && rm -rf `ls -1r | awk 'NR>15'` && /usr/bin/grub-mkconfig -o /boot/grub/grub.cfg"

[Install]
WantedBy=multi-user.target

I also get this output running systemctl status for that service:

● limit-timeshift-snapshots-update-grub.service - Keep only latest 15 timeshift snapshots and update grub.cfg before shutdown
     Loaded: loaded (/etc/systemd/system/limit-timeshift-snapshots-update-grub.service; enabled; vendor preset: disabled)
     Active: active (exited) since Sun 2020-08-23 13:41:11 IST; 7min ago
    Process: 791 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 791 (code=exited, status=0/SUCCESS)

Aug 23 13:41:11 Rocinante systemd[1]: Starting Keep only latest 15 timeshift snapshots and update grub.cfg before shutdown...
Aug 23 13:41:11 Rocinante systemd[1]: Finished Keep only latest 15 timeshift snapshots and update grub.cfg before shutdown.

EDIT: Should I be able to see snapshots running this:

btrfs subvolume list /
ID 256 gen 4341 top level 5 path @
ID 257 gen 4341 top level 5 path @home
ID 260 gen 22 top level 256 path var/lib/portables
ID 261 gen 23 top level 256 path var/lib/machines
ID 268 gen 4318 top level 5 path @var-cache-pacman-pkg
ID 269 gen 4326 top level 5 path @swap

The place that I see snapshots is /run/timeshift/backup/timeshift
It really makes me think something went wrong with the generation of timeshift.json, since mine and @ricklinux’s setup are different.

#38

Yes, snapshots should show too.

They are supposed to be in /run/timeshift/backup/timeshift-btrfs/snapshots/
That’s probably why grub-btrfs can’t find any snapshots.

I don’t know what went wrong on your end; I just tested the modified commands in a VM installation and everything was set up OK.

Maybe a reinstall (remove and install) of Timeshift will set things up as they should be.

1 Like
#39

Thanks, just did that. I am starting with step #6 again.

I am following the steps to modify the commands for the unencrypted version as you suggest in the other thread. Removing and changing the lines as instructed, is this what I should be left with?

sudo cp /etc/default/timeshift.json /etc/timeshift.json &&
fs_uuid=`sudo blkid -o device | grep sda2` &&
fs_uuid=`sudo blkid -o value -s UUID $fs_uuid` &&
sudo sed -i "s/backup_device_uuid\" : \"/backup_device_uuid\" : \"$fs_uuid/" /etc/timeshift.json &&
sudo sed -i 's/run" : "true/run" : "false/' /etc/timeshift.json &&
sudo sed -i 's/btrfs_mode" : "false/btrfs_mode" : "true/' /etc/timeshift.json &&
sudo sed -i 's/  "include_btrfs_home" : "false",/  "include_btrfs_home_for_backup" : "false",\n  "include_btrfs_home_for_restore" : "false",/' /etc/timeshift.json &&
sudo sed -i 's/schedule_daily" : "false/schedule_daily" : "true/' /etc/timeshift.json &&
sudo sed -i 's/schedule_boot" : "false/schedule_boot" : "true/' /etc/timeshift.json &&
sudo sed -i 's/count_boot" : "5/count_boot" : "3/' /etc/timeshift.json
sudo timeshift --create --comments "1st"
#40

I actually meant just reinstalling Timeshift (as in removing Timeshift including config files and then installing again) on the “broken” system. :smiley:

But if you’re redoing the whole thing anyway it won’t hurt to double check the instructions; thanks.

Looks right, yes.

For reference, the whole modified instructions for an unencrypted system should be as following:

#03
Don’t choose encrypt .

#05

sudo mkdir /mnt/btrfs
btrfsonluks=/dev/sda2
sudo mount -o subvolid=5 $btrfsonluks /mnt/btrfs
sudo sed -i 's/compress=zstd 0 1/compress=zstd 0 0/' /mnt/btrfs/@/etc/fstab
sudo sed -i 's/compress=zstd 0 2/compress=zstd 0 0/' /mnt/btrfs/@/etc/fstab
sudo btrfs subvolume create /mnt/btrfs/@var-cache-pacman-pkg
fstabstr=`grep "/home" /mnt/btrfs/@/etc/fstab`
fstabstr=${fstabstr//\/home/\/var\/cache\/pacman\/pkg}
fstabstr=${fstabstr//@home/@var-cache-pacman-pkg}
sudo sh -c "echo $fstabstr >> /mnt/btrfs/@/etc/fstab"
sudo btrfs subvolume create /mnt/btrfs/@swap
sudo mkdir /mnt/btrfs/@/swap
sudo sh -c "printf '\n$btrfsonluks /swap          btrfs   subvol=@swap,defaults,compress=no 0 0\n' >> /mnt/btrfs/@/etc/fstab"
sudo truncate -s 0 /mnt/btrfs/@swap/swapfile
sudo chattr +C /mnt/btrfs/@swap/swapfile
sudo btrfs property set /mnt/btrfs/@swap/swapfile compression none
sudo fallocate -l $swapsize /mnt/btrfs/@swap/swapfile
sudo chmod 600 /mnt/btrfs/@swap/swapfile
sudo mkswap /mnt/btrfs/@swap/swapfile
sudo swapon /mnt/btrfs/@swap/swapfile
sudo sh -c "echo '/swap/swapfile none swap defaults 0 0' >> /mnt/btrfs/@/etc/fstab"
wget https://raw.githubusercontent.com/osandov/osandov-linux/master/scripts/btrfs_map_physical.c
gcc -O2 -o btrfs_map_physical btrfs_map_physical.c
offset=`sudo ./btrfs_map_physical /mnt/btrfs/@swap/swapfile`
offset_arr=(`echo ${offset}`)
offset_pagesize=(`getconf PAGESIZE`)
offset=$(( offset_arr[25] / offset_pagesize ))
sudo sed -i "s/loglevel=3/loglevel=3 resume_offset=$offset/" /mnt/btrfs/@/etc/default/grub
sudo sed -i "s#loglevel=3#resume=$btrfsonluks loglevel=3#" /mnt/btrfs/@/etc/default/grub
sudo sed -i 's/keymap encrypt filesystems/keymap encrypt filesystems resume/' /mnt/btrfs/@/etc/mkinitcpio.conf
sudo mkdir /mnt/chroot
sudo mount -o compress=zstd,subvol=@ $btrfsonluks /mnt/chroot
efidevice=`sudo blkid -o device -l -t TYPE=vfat`
sudo mount $efidevice /mnt/chroot/boot/efi
sudo arch-chroot /mnt/chroot
sudo pacman --noconfirm -S cronie
sudo systemctl enable cronie.service
sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo mkinitcpio -p linux

#06

sudo pacman --noconfirm -Syy &&
yay --noconfirm -S timeshift timeshift-autosnap grub-btrfs

sudo cp /etc/default/timeshift.json /etc/timeshift.json &&
fs_uuid=`sudo blkid -o device | grep sda2` &&
fs_uuid=`sudo blkid -o value -s UUID $fs_uuid` &&
sudo sed -i "s/backup_device_uuid\" : \"/backup_device_uuid\" : \"$fs_uuid/" /etc/timeshift.json &&
sudo sed -i 's/run" : "true/run" : "false/' /etc/timeshift.json &&
sudo sed -i 's/btrfs_mode" : "false/btrfs_mode" : "true/' /etc/timeshift.json &&
sudo sed -i 's/  "include_btrfs_home" : "false",/  "include_btrfs_home_for_backup" : "false",\n  "include_btrfs_home_for_restore" : "false",/' /etc/timeshift.json &&
sudo sed -i 's/schedule_daily" : "false/schedule_daily" : "true/' /etc/timeshift.json &&
sudo sed -i 's/schedule_boot" : "false/schedule_boot" : "true/' /etc/timeshift.json &&
sudo sed -i 's/count_boot" : "5/count_boot" : "3/' /etc/timeshift.json
sudo timeshift --create --comments "1st"
1 Like
#41

Oh no, sorry I did actually only reinstall Timeshift, but I redid step #6 to set up the timeshift.json file properly. Good news - it seems to have worked! Btrfs snapshots are now visible in Grub. I can select the ‘Arch linux snapshots’ option from the Grub menu, then select a snapshot, and then if I choose one of the initramfs options in the next menu it boots into the snapshot (the top option did nothing though).

I can also see them here:

ID 256 gen 4525 top level 5 path @
ID 257 gen 4525 top level 5 path @home
ID 260 gen 22 top level 256 path var/lib/portables
ID 261 gen 23 top level 256 path var/lib/machines
ID 268 gen 4453 top level 5 path @var-cache-pacman-pkg
ID 269 gen 4326 top level 5 path @swap
ID 295 gen 4451 top level 5 path timeshift-btrfs/snapshots/2020-08-23_14-57-41/@
ID 296 gen 4462 top level 5 path timeshift-btrfs/snapshots/2020-08-23_15-00-01/@
ID 298 gen 4483 top level 5 path timeshift-btrfs/snapshots/2020-08-23_15-09-13/@
ID 299 gen 4512 top level 5 path timeshift-btrfs/snapshots/2020-08-23_15-25-07/@

I did catch some errors updating though:

 
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Creating new backup...(BTRFS)
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Saving to device: /dev/sda2, mounted at path: /run/timeshift/backup
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Created directory: /run/timeshift/backup/timeshift-btrfs/snapshots/2020-08-23_14-57-41
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Created subvolume snapshot: /run/timeshift/backup/timeshift-btrfs/snapshots/2020-08-23_14-57-41/@
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Created control file: /run/timeshift/backup/timeshift-btrfs/snapshots/2020-08-23_14-57-41/info.json
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] BTRFS Snapshot saved successfully (0s)
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Tagged snapshot '2020-08-23_14-57-41': ondemand
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] ------------------------------------------------------------------------------
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] E: ERROR: can't list qgroups: quotas not enabled
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] 
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] E: btrfs returned an error: 256
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] E: Failed to query subvolume quota
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Enabled subvolume quota support
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Added cron task: /etc/cron.d/timeshift-hourly
[2020-08-23T14:57:41+0100] [ALPM-SCRIPTLET] Added cron task: /etc/cron.d/timeshift-boot
[2020-08-23T14:57:42+0100] [ALPM-SCRIPTLET] Generating grub configuration file ...
[2020-08-23T14:57:42+0100] [ALPM-SCRIPTLET] Found theme: /boot/grub/themes/EndeavourOS/theme.txt
[2020-08-23T14:57:43+0100] [ALPM-SCRIPTLET] Found linux image: /boot/vmlinuz-linux
[2020-08-23T14:57:43+0100] [ALPM-SCRIPTLET] Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
[2020-08-23T14:57:43+0100] [ALPM-SCRIPTLET] Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] ###### - Grub-btrfs: Snapshot detection started - ######
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] # Info: Separate boot partition not detected 
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] # Found snapshot: 2020-08-23 14:57:41 | timeshift-btrfs/snapshots/2020-08-23_14-57-41/@
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] # Found 1 snapshot(s)
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] ###### - Grub-btrfs: Snapshot detection ended   - ######
[2020-08-23T14:57:44+0100] [ALPM-SCRIPTLET] done