Moving to Endeavour & dealing with current partition setup

Oh…ok - btrfs shapshots are actually done through Timeshift? I was under the impression that btrfs snapshots and Timeshift snapshots were a seperate thing.

So would this preclude me from ONLY setting up / as btrfs, and leaving /home as ext4? What would the advantage be of setting up /home as a btrfs subvolume?

Thanks for the writeup by the way, I have to admit it looks pretty daunting at first glance, but maybe I will be able to crack it with some more time.

Timeshift actually just uses standard btrfs tools for snapshot creation. You could also ceate these yourself without Timeshift from the cli. But Timeshift really simplifies this process immensely.

No, you could always still mount your ext4 partition to /home by changing the fstab later on.

Well you could also take snapshots of your home data and easily restore. I use this feature when I don’t want any “test residue” staying in /home after playing around with stuff.
I also run quite a few VM’s and haven’t had any issues regarding btrfs. Anyway, you could always store and run these from a mounted ext4 partition if need be.

[Edit] Another advantage: the subvolumes /root and /home share the same space. So your /root will never have too much (wasted) or too little space.

1 Like

Thanks.
It only looks that way; it’s meant to basically just be copied and pasted to the terminal. Should be (meant to) easy to set up.

2 Likes

Shows you how smooth Timeshift works that I didn’t even realise it was using btrfs (though now on thinking about it, I do remember something about this when I first configured it).

Ok, so here’s what I did - since I’m keeping my HDD as a place to symlink stuff like ~/Documents, etc. I just moved my VMs there for now, as well as any dotfiles from my home that might come in handy. That way, presumably, I can just do what you recommend and make one btrfs partition for / and /home. My HDD is on ext4, and it can stay there, and I can still play with btrfs on sda.

But in the Linux world that’s like taking candy from strangers! :laughing:

@2000 or @dalto - would you kindly advise me if I still need to carry out step #04 – Mount newly installed btrfs system to temporary directory /mnt/btrfs, even if I am not encrypting this install? I notice the instructions include:

sudo blkid -o device | grep luks

Edit:

From the guide below

Yes, you do need to mount, but the “grep luks” part won’t work if you’re going the unencrypted route.

I’d advise you follow the “quick” version (BTRFSonLUKS copy&paste) and change according to this post.
The outcome will be identical to the wiki article you’re referencing.

Adapting that Wiki article would of course also work; just haven’t done that yet.

[Edit] You can always add other partitions and mount via fstab after the install.

1 Like

Ah ok, switched over to the copy and paste guide.

For this step:

sudo grub-mkconfig -o /boot/grub/grub.cfg

I got:

grub-probe: error: cannot find a GRUB drive for /dev/sdd1.  Check your device.map.
done

sdd1 however is the USB stick I am installing from, so I suppose that’s probably expected.

This is the output I got from:

sudo mkinitcpio -p linux

=> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.8.1-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
loadkeys: Unable to open file: ie: No such file or directory
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.8.1-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
loadkeys: Unable to open file: ie: No such file or directory
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

Again, probably ok, but we’ll find out when I reboot and start step #6.

Well - looks like I am now on EndeavourOS! Everything seems to be working correctly, though I’m not sure how to check if that btrfs stuff is all working.

4 Likes

See below.
You’ll notice the benefits of btrfs snapshots once you create and restore them. Unlike rsync, restoring btrfs snapshots takes about 2 seconds!


Some useful btrfs related commands:

  • show mounted subvolumes
    findmnt -nt btrfs

  • show mountpoints (including fat32 and ext4 partitions)
    findmnt -D -t btrfs,vfat,ext4

  • list all subvolumes
    sudo btrfs sub list /

  • check size
    sudo btrfs filesystem usage /

3 Likes

Great, will check these tomorrow in more detail. Thanks so much for talking me through the setup, as well as for putting together that guide. It’s a very valuable resource it seems.

2 Likes

Coming back to my experience of installing in the light of the next day, since I switched to the copy and paste version halfway through the install yesterday, I just read through the verbose version again to get my head wrapped around what I did. I was feverishly copying and pasting stuff, probably a lot more casually than I should have been (seems I have less and less time lately after work), so I was worried I might have missed running a command or something that could cause problems further down the line.

I think the verbose version is great, though I can see that it’s not quite as clear with the action points, i.e. the commands aren’t always ready to run. It is still very useful having an explanation for what you’re doing with each command. It makes the whole think feel more ‘linuxy’ and less like you have to hide what you’re doing from the Arch purists watching over the fence in disgust :eyes:. A fusion of the two approaches would be perfect, or even just carrying over small explanatory paragraphs to the copy and paste version. Not being ungrateful though, I really appreciate all the time that went into it.

Running through the commands you provided me in the last post:

I get these results, all of which, at least to my inexperienced eye, seem alright:

/                       /dev/sda2[/@]                     btrfs  rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=256,subvol=/
├─/run/timeshift/backup /dev/sda2                         btrfs  rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
├─/swap                 /dev/sda2[/@swap]                 btrfs  rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=269,subvol=
├─/home                 /dev/sda2[/@home]                 btrfs  rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=257,subvol=/
└─/var/cache/pacman/pkg /dev/sda2[/@var-cache-pacman-pkg] btrfs  rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=268,subvol=/

It looks like all the various sub-volumes of / are accounted for and mounted in the right places:

findmnt -D -t btrfs,vfat,ext4
SOURCE                            FSTYPE   SIZE  USED  AVAIL USE% TARGET
/dev/sda2[/@]                     btrfs  232.4G 26.7G 204.6G  11% /
/dev/sda2[/@home]                 btrfs  232.4G 26.7G 204.6G  11% /home
/dev/sda2[/@swap]                 btrfs  232.4G 26.7G 204.6G  11% /swap
/dev/sda2[/@var-cache-pacman-pkg] btrfs  232.4G 26.7G 204.6G  11% /var/cache/pacman/pkg
/dev/sda1                         vfat     510M  696K 509.3M   0% /boot/efi
/dev/sdb1                         ext4     1.8T  624G   1.1T  34% /mnt/hdd
/dev/sda2                         btrfs  232.4G 26.7G 204.6G  11% /run/timeshift/backup

Here’s all the subvolumes listed (interesting to see that /var/lib/portables and /var/lib/machines are automatically created - seems to be something to do with systemd):

sudo btrfs sub list /
ID 256 gen 3183 top level 5 path @
ID 257 gen 3183 top level 5 path @home
ID 260 gen 22 top level 256 path var/lib/portables
ID 261 gen 23 top level 256 path var/lib/machines
ID 268 gen 3053 top level 5 path @var-cache-pacman-pkg
ID 269 gen 108 top level 5 path @swap

I couldn’t find anything concrete on why the ID skips from 261 to 268, but I’ll do more digging.

Here’s the size info:

 sudo btrfs filesystem usage /
 Overall:
     Device size:                 232.39GiB
     Device allocated:             28.02GiB
     Device unallocated:          204.37GiB
     Device missing:                  0.00B
     Used:                         26.64GiB
     Free (estimated):            204.59GiB      (min: 204.59GiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:               52.88MiB      (used: 0.00B)
     Multiple profiles:                  no
 
 Data,single: Size:26.01GiB, Used:25.79GiB (99.15%)
    /dev/sda2      26.01GiB
 
 Metadata,single: Size:2.01GiB, Used:877.45MiB (42.68%)
    /dev/sda2       2.01GiB
 
 System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
    /dev/sda2       4.00MiB
 
 Unallocated:
    /dev/sda2     204.37GiB

Lots of unallocated space, but from my reading into btrfs, seems routine.

Checking to see if my swapfile is there, it seems that’s all ok too:

              total        used        free      shared  buff/cache   available
Mem:          16003        1946       12476         222        1580       13539
Swap:          8191           0        8191

And also shows up here:

cat /proc/swaps
Filename                                Type            Size            Used            Priority
/swap/swapfile                          file            8388604         0               -2

Admittedly, even after going through this, there is the urge to reinstall and only run each command when I fully understand it, but I also don’t want to jinx what appears to be a perfectly functional system. Plus, if this was all incorporated in the regular install automatically, I would never noticed it doing all this in the background. I think it might just mean I’m ready for a more Arch-style install process next time around.

But hey, at the end of the day: I’m on EndeavourOS now, looking forward to learning more about my system, this distribution and the community. Thanks again for helping me get there @2000.

3 Likes

I wish i understood it too! My setup is less elaborate as it is the copy and paste only on a fresh install so i don’t have some of the mnt points you have.

[ricklinux@eos-xfce ~]$ findmnt -D -t btrfs,vfat,ext4
SOURCE                                                        FSTYPE   SIZE  USED  AVAIL USE% TARGET
/dev/mapper/luks-efc1098d-46e9-4d34-b4bf-a1b77b3a2361[/@]     btrfs  465.3G 84.5G 380.2G  18% /
/dev/nvme0n1p1                                                vfat     511M    2M   509M   0% /boot/efi
/dev/mapper/luks-efc1098d-46e9-4d34-b4bf-a1b77b3a2361[/@home] btrfs  465.3G 84.5G 380.2G  18% /home
/dev/mapper/luks-efc1098d-46e9-4d34-b4bf-a1b77b3a2361[/@swap] btrfs  465.3G 84.5G 380.2G  18% /swap
/dev/mapper/luks-efc1098d-46e9-4d34-b4bf-a1b77b3a2361[/@var-cache-pacman-pkg]
                                                              btrfs  465.3G 84.5G 380.2G  18% /var/cache/pacman/pkg
/dev/mapper/luks-efc1098d-46e9-4d34-b4bf-a1b77b3a2361         btrfs  465.3G 84.5G 380.2G  18% /run/timeshift/backup
[ricklinux@eos-xfce ~]$ 
1 Like

I think mine is different, mainly because I have a separate 2TB hard drive mounted on /mnt/hdd - that’s where I store all my image files, documents, etc. So at least theoretically, reinstalling and such should be easier.

One thing I noticed is I don’t seem to have the option in grub to boot into snapshots, though the “limit-timeshift-snapshots-update-grub.service” is loaded and active, and I also seem to have 4 snapshots (one created as 1st and three others before upgrading).

I also have a 4 TB hard drive but i don’t know how to add that as a mnt? It has to be done before using the copy and paste method? You had an existing setup so i’m not sure and under normal circumstances i think you just add it to the fstab? I’m not that good at some of this stuff without putting a lot of effort into it which i struggle with time and (effort)! :point_left:

Nope, I actually did it afterwards, I added it manually into my fstab.

I found the UUID of the drive (for me it was sdb1).

sudo blkid

Then I used nano to edit my fstab, adding in this line:

UUID="51596fec-9d35-4522-87c5-64bacce1314c" /mnt/hdd     ext4   errors=remount-ro 0 1

Obviously my UUID, my desired mountpoint, the filesystem (ext4 in this case) would be different than yours probably. I also can’t say for sure if the flags at the end should be the same for you.

I’m not an expert by any means, but that’s just how I did it.

EDIT:

On further inspection, I think the last field should be 2, rather than 1. But again, please don’t take that as gospel.

So i can add this after the copy & paste setup of BTRFSonLUKS? If so i might do that. Currently i have 2 m.2 drives an SSD and a Hard Drive. My setup is separate BTRFSonLUKS on 3 drives. Two being the m.2 and one being the SSD. I am using rEFInd and then decrypt the drive which goes to the grub menu. Each is separate and doesn’t interfere with each other. So the grub menu on each drive is only for that drive. I select which drive from rEFInd and i use grubx64.efi to boot them. So i could use the 4 TB drive to share on each i guess?

Wow, that’s a lot of drives! Honestly can’t give you advice there, sorry, I am still learning myself.

I have narrowed down why I am not seeing any btrfs snapshots in grub. When I run grub-mkconfig, I see this at the end of the output:

### BEGIN /etc/grub.d/41_snapshots-btrfs ###
###### - Grub-btrfs: Snapshot detection started - ######
# Info: Separate boot partition not detected 
# No snapshots found.
# If you think an error has occurred , please file a bug report at " https://github.com/Antynea/grub-btrfs "
# Nothing to do. Abort.
###### - Grub-btrfs: Snapshot detection ended   - ######
### END /etc/grub.d/41_snapshots-btrfs ###
done

@2000, perhaps you might have some insight into this? I can confirm there are snapshots present in /run/timeshift/backup/

This is what i get but i think @2000 will have to advise you on that as he is the expert here.

[ricklinux@eos-xfce ~]$ sudo grub-mkconfig -o /boot/grub/grub.cfg
[sudo] password for ricklinux: 
Generating grub configuration file ...
Found theme: /boot/grub/themes/EndeavourOS/theme.txt
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
###### - Grub-btrfs: Snapshot detection started - ######
# Info: Separate boot partition not detected 
# Found snapshot: 2020-08-22 10:16:28 | timeshift-btrfs/snapshots/2020-08-22_10-16-28/@
# Found snapshot: 2020-08-22 09:19:46 | timeshift-btrfs/snapshots/2020-08-22_09-19-46/@
# Found snapshot: 2020-08-21 20:22:44 | timeshift-btrfs/snapshots/2020-08-21_20-22-44/@
# Found snapshot: 2020-08-21 20:10:20 | timeshift-btrfs/snapshots/2020-08-21_20-10-20/@
# Found snapshot: 2020-08-21 14:22:27 | timeshift-btrfs/snapshots/2020-08-21_14-22-27/@
# Found snapshot: 2020-08-21 13:37:02 | timeshift-btrfs/snapshots/2020-08-21_13-37-02/@
# Found 6 snapshot(s)
###### - Grub-btrfs: Snapshot detection ended   - ######
done
[ricklinux@eos-xfce ~]$ 
1 Like

Hey, @ricklinux just out of curiosity, would you mind checking what the output of

cat /etc/timeshift.json

Is on your system?

[ricklinux@eos-xfce ~]$ cat /etc/timeshift.json
{
  "backup_device_uuid" : "0901c653-3f70-4f30-9ab9-f13818e07187",
  "parent_device_uuid" : "efc1098d-46e9-4d34-b4bf-a1b77b3a2361",
  "do_first_run" : "false",
  "btrfs_mode" : "true",
  "include_btrfs_home_for_backup" : "false",
  "include_btrfs_home_for_restore" : "false",
  "stop_cron_emails" : "true",
  "schedule_monthly" : "false",
  "schedule_weekly" : "false",
  "schedule_daily" : "true",
  "schedule_hourly" : "false",
  "schedule_boot" : "true",
  "count_monthly" : "2",
  "count_weekly" : "3",
  "count_daily" : "5",
  "count_hourly" : "6",
  "count_boot" : "3",
  "snapshot_size" : "0",
  "snapshot_count" : "0",
  "exclude" : [
  ],
  "exclude-apps" : [
  ]
}
[ricklinux@eos-xfce ~]$ 
1 Like