Best Approach for transformation from ext4 to BTRFS including replacement of rsync-backups

[milkytwix]

So, I want to change all my HardDisks and partitions to BTRFS. I want to have local snapshots and I want to have full remote Snapshots. My current Hard Disk Layout is the following:

sudo lsblk --output NAME,FSTYPE,LABEL,MODE,TYPE
NAME        FSTYPE LABEL    MODE       TYPE
sda                         brw-rw---- disk
└─sda1      ext4   Data     brw-rw---- part
sdb                         brw-rw---- disk
├─sdb1      ext4   Misc     brw-rw---- part
├─sdb2      ext4   Stuff    brw-rw---- part
└─sdb3      ext4   Backup   brw-rw---- part
nvme1n1                     brw-rw---- disk
└─nvme1n1p1 ext4   Virtual  brw-rw---- part
nvme0n1                     brw-rw---- disk
├─nvme0n1p1 vfat   NO_LABEL brw-rw---- part
├─nvme0n1p2 ext4   home     brw-rw---- part
└─nvme0n1p3 ext4   system   brw-rw---- part

So, basic installation will happen on nvme0n1, with BTRFS and the “ordinary” “partitions”, so I go with @home here. I have a folder namend “games” under /home, which I almost treated as seperate partition - can I just add an @games here? What would be your approach?
nvme1n1 will be split in to two “partitions” - one will be @virtual, the other will be @snapshots - here I want to store snapshots of /home and / locally - is this easily doable?
Data needs to have a seperate @, which I want to snapshot remotly - easy doable?
Same for everything on SDB.

I have a NAS available which I currently use for backups of the ext4 partitions via rsync. The NAS has both NFS as well as SMB shares available and is based on TrueNAS. Can I easily store BTRFS snapshots on this via NFS/SMB?

[dalto]

@milkytwix

can I just add an @games here

There are two common ways this is accomplished:

• You can created a nested subvolume inside your home named games.
• You can create a subvolume at the root of the partition called whatever you want. For example, @games if that is your preference. Then create an empty games directory. Lastly, edit /etc/fstab and mount the subvolume.

@milkytwix

What would be your approach?

For most use cases, the first option above is a lot simpler.

@milkytwix

nvme1n1 will be split in to two “partitions” - one will be @virtual, the other will be @snapshots

Eh, why are you splitting it like that? Couldn’t you combine them and use subvolumes to separate the data? That will be a lot easier to manage.

@milkytwix

here I want to store snapshots of /home and / locally - is this easily doable?

Unless you are planning to implement a manual snapshot system, that will not be “easily” doable. Both snapper and timeshift have their own locations for snapshots. It is better to just follow their defaults if you are using one of those tools.

Also, there is no real reason to store snapshots like that. The underlying storage for a snapshot is very different than a backup. They snapshot isn’t really “stored” there. It is just accessible via that location.

@milkytwix

Data needs to have a seperate @

I am not sure what this means. Can you explain?

@milkytwix

which I want to snapshot remotly - easy doable?

It would be a lot easier to take the snapshots locally and then replicate them to a remote device/location.

@milkytwix

Can I easily store BTRFS snapshots on this via NFS/SMB?

No. Btrfs snapshots can only be stored on a btrfs filesystem. They are part of the filesystem, not separate data.

You could copy the data out of snapshot to a NAS but it would lose all the benefits of being a snapshot.

[milkytwix]

@dalto

There are two common ways this is accomplished:

• You can created a nested subvolume inside your home named games.
• You can create a subvolume at the root of the partition called whatever you want. For example, @games if that is your preference. Then create an empty games directory. Lastly, edit /etc/fstab and mount the subvolume.

I wanted to avoid nested “@” because of how snapshots are working - hen I planed with seperate subvolumes with the respective “@” for snapper. I think that way it would be easier - wrong?
Reson why I want to it that way is nested “@” and I do it currently exactly the same way with the /etc/fstab. I want to manage snapshots with BTRFS assistant (what a lovely tool you have there… felt instant in love) on top of snapper and timeshift, and that way it seems to be more easy, but → *

@dalto

For most use cases, the first option above is a lot simpler.

The second does not hurt, beside the fstab shannanigans?
I want to have snapshots for /, /home and “games” (be it /games or /home/games or whatever) seperate from each other. Games will be snapshotted manually, home daily and / will be snapshotted manually, by timesahift and by (and most important, cause arch) btrfs-grub hook.

@dalto

Eh, why are you splitting it like that? Couldn’t you combine them and use subvolumes to separate the data? That will be a lot easier to manage.

I will use subvolumes - I just called them “partitions” because I was still searching for the best approach, which was clearly answered in your first paragraph. I learn from every post of you.

@dalto

Unless you are planning to implement a manual snapshot system, that will not be “easily” doable. Both snapper and timeshift have their own locations for snapshots. It is better to just follow their defaults if you are using one of those tools.

Also, there is no real reason to store snapshots like that. The underlying storage for a snapshot is very different than a backup. They snapshot isn’t really “stored” there. It is just accessible via that location.

Again understood. In my journey to learn BTRFS i stumpled over a video (which I cant find anymore) where the place of the snapshots where somewhere else. I intended to use that as snapshot AND backup at the same time, but I guess this is the wrong approach then.

@dalto

I am not sure what this means. Can you explain?

Well, that sentence was nonsense. Data is on it’s own physical disc, so it’s getting it’s own file system with automatically it’s own “@”…

@dalto

It would be a lot easier to take the snapshots locally and then replicate them to a remote device/location.

So, remote location must be BTRFS then? At least when I look at this:

@dalto

No. Btrfs snapshots can only be stored on a btrfs filesystem. They are part of the filesystem, not separate data.

You could copy the data out of snapshot to a NAS but it would lose all the benefits of being a snapshot.

Which is quite a bummer - what’s the best approach here? Just leave it be, use BTRFS ony for the main disk (/, /home and “games”) and be happy? How do you do backups?

[dalto]

@milkytwix

I wanted to avoid nested “@” because of how snapshots are working

Err…first @ is just a convention. You don’t actually need to name your subvolumes with @ unless you want to.

Second, snapshots don’t descend into nested subvolumes so this shouldn’t be an issue either way,

@milkytwix

hen I planed with seperate subvolumes with the respective “@” for snapper. I think that way it would be easier - wrong?

I don’t really understand what you mean at all with this statement.

@milkytwix

I want to manage snapshots with BTRFS assistant (what a lovely tool you have there… felt instant in love) on top of snapper and timeshift, and that way it seems to be more easy

If you want to use Btrfs Assistant you definitely shouldn’t create manual subvolumes for holding snapshots. That will definitely make it harder.

@milkytwix

The second does not hurt, beside the fstab shannanigans?

No, it works fine.

@milkytwix

I want to have snapshots for /, /home and “games” (be it /games or /home/games or whatever) seperate from each other.

They will always be separate. You are taking snapshots of a subvolume. How you nest or mount them won’t change that.

@milkytwix

Games will be snapshotted manually, home daily and / will be snapshotted manually, by timesahift and by (and most important, cause arch) btrfs-grub hook.

Above you mentioned you were going to use Snapper. Snapper and Timeshift are alternatives. You should not use them both. Timeshift can only take snapshots of @ and @home. Nothing else.

Snapper can take snapshots of any subvolume.

@milkytwix

Data is on it’s own physical disc, so it’s getting it’s own file system with automatically it’s own “@”

@ is convention that generally refers to the filesystem mounted at /. If you have a second disk, I would not recommend creating a subvolume named @ on it. That would be confusing in a lot of ways.

Don’t get hung up on the whole @ thing. It is just a name. For example, none of my installs have subvolumes named that way.

@milkytwix

So, remote location must be BTRFS then?

Yes.

@milkytwix

Which is quite a bummer - what’s the best approach here?

Is your TrueNAS box using zfs? If so, you could use btrfs for you main drive and zfs for your data drive. zfs supports snapshots and snapshot replication as well. It is a little harder to configure though.

A simpler option would be to use snapshots locally and deduplicated borg backups on your NAS. Those backups are nearly as space efficient as btrfs snapshots. The disadvantage is they are a lot slower to create.

@milkytwix

How do you do backups?

Personally? I use a combination of snapshots, snapshot replication, borg and rclone for a remote replica of my encrypted borg backups.

Any important data is stored at a minimum of 3 times and at least 2 geographically disparate locations. In many cases I have 4 copies of the data and also employ raid at most of the storage locations.

[milkytwix]

@dalto

Err…first @ is just a convention. You don’t actually need to name your subvolumes with @ unless you want to.

Second, snapshots don’t descend into nested subvolumes so this shouldn’t be an issue either way,

Wait what… That makes things WAY more easily. So I just create a subvolume via “sudo btrfs subvolume create /home/milkytwix/Games/” and I am good to go, snapshots of /home will not secend into it and everythings fine? Really that simple? Asking for a friend, ofc…

@dalto

I don’t really understand what you mean at all with this statement.

Because I wans’t aware of the question (and answer above) I planned to mount “games” from the “outside”, so to speak. But this is not required.

@dalto

If you want to use Btrfs Assistant you definitely shouldn’t create manual subvolumes for holding snapshots. That will definitely make it harder.

Understood, again: Maiking things easier!

@dalto

No, it works fine.

Well… I will go with the first, because… no descending into nested subvolumes

@dalto

They will always be separate. You are taking snapshots of a subvolume. How you nest or mount them won’t change that.

Well again: no descending makes things WAY more easier

@dalto

Above you mentioned you were going to use Snapper. Snapper and Timeshift are alternatives. You should not use them both. Timeshift can only take snapshots of @ and @home. Nothing else.

Snapper can take snapshots of any subvolume.

Well, I definately want this:
https://discovery.endeavouros.com/encrypted-installation/btrfs-with-timeshift-snapshots-on-the-grub-menu/2022/02/
And because snapper is included in BTRFS-Assistant I intended to use it as well… can I have grub snapshots without timeshift? Including auto-generated ones after updates? Because I just love that feature.

@dalto

@ is convention that generally refers to the filesystem mounted at /. If you have a second disk, I would not recommend creating a subvolume named @ on it. That would be confusing in a lot of ways.

Don’t get hung up on the whole @ thing. It is just a name. For example, none of my installs have subvolumes named that way.

Understood! The next questions are just out of curiosity:
Why don’t you use them? How do you avoid them being created during install?

@dalto

Yes.

Thx!

@dalto

Is your TrueNAS box using zfs? If so, you could use btrfs for you main drive and zfs for your data drive. zfs supports snapshots and snapshot replication as well. It is a little harder to configure though.

A simpler option would be to use snapshots locally and deduplicated borg backups on your NAS. Those backups are nearly as space efficient as btrfs snapshots. The disadvantage is they are a lot slower to create.

I already use snapshot replication for the backup of my NAS, that won’t be an issue. My original plan was to use ZFS on root (Remember? Another "Help, migrating from Manjaro" topic - #2 by dalto ), so I will stick to ext4 for the time being on my non nvme drives and slowly convert them to ZFS.

@dalto

Personally? I use a combination of snapshots, snapshot replication, borg and rclone for a remote replica of my encrypted borg backups.

Any important data is stored at a minimum of 3 times and at least 2 geographically disparate locations. In many cases I have 4 copies of the data and also employ raid at most of the storage locations.

The first part was the interesting (I was asking for the technical implementation, not the backup concept, sorry for asking in an unprecise manner). My backup concept is: Data on NAS, local Data is copied to NAS, together with backups of /home and stuff. NAS gets nightly backupped to secondary NAS (which resides in the garage), and Monthly backupped to a Harddisk which rotates to my sister.
Coming back to “how you do it”: I will read into rclone and borg and try to find something suitable, thank you VERY much!

[dalto]

@milkytwix

So I just create a subvolume via “sudo btrfs subvolume create /home/milkytwix/Games/” and I am good to go, snapshots of /home will not secend into it and everythings fine?

Yes. It is that easy.

@milkytwix

Well, I definately want this:
https://discovery.endeavouros.com/encrypted-installation/btrfs-with-timeshift-snapshots-on-the-grub-menu/2022/02/
And because snapper is included in BTRFS-Assistant I intended to use it as well… can I have grub snapshots without timeshift? Including auto-generated ones after updates? Because I just love that feature.

I really don’t recommend mixing snapper and timeshift. There is no benefit to that. Snapper also supports automatic snapshots before and after updates, booting from snapshots with grub, etc.

@milkytwix

Why don’t you use them?

Several reasons:

• I find it to be meaningless convention. The idea for using @ at the beginning of subvol names doesn’t add value the way most people use them.
• I do some complicated things with subvols where I use them to install multiple distros on the same filesystem. That naming convention just gets in the way for me.
• I don’t like doing things just because others do.

@milkytwix

How do you avoid them being created during install?

You can edit /etc/calamares/modules/mount.conf before starting the installer.

That being said, there is no real advantage to doing so. It doesn’t really matter either way. They are just names so unless you have a reason to change them I would leave them at the defaults.

[milkytwix]

@dalto

Yes. It is that easy.

I owe you a brewed cold one, Sir!

@dalto

I really don’t recommend mixing snapper and timeshift. There is no benefit to that. Snapper also supports automatic snapshots before and after updates, booting from snapshots with grub, etc.

So basically it is the following:

1. Install EOS with btrfs

2. sudo pacman -S btfs-assistant snapper grub-btrfs

3. sudo systemctl enable --now grub-btrfsd

And we are good to go?

@dalto

Several reasons:

• I find it to be meaningless convention. The idea for using @ at the beginning of subvol names doesn’t add value the way most people use them.
• I do some complicated things with subvols where I use them to install multiple distros on the same filesystem. That naming convention just gets in the way for me.
• I don’t like doing things just because others do.

Reasons 1 and 3 are reasons which I can EASILY identify with, reason 2 will make me create another topic when I am firmer with BTFS. Maybe.

@dalto

You can edit /etc/calamares/modules/mount.conf before starting the installer.

That being said, there is no real advantage to doing so. It doesn’t really matter either way. They are just names so unless you have a reason to change them I would leave them at the defaults.

I currently have no good reason other than they just make not much sense - no added value. Well, let’s see where I end up with

[dalto]

@milkytwix

sudo pacman -S btfs-assistant snapper grub-btrfs

You could also install snap-pac if you want snapshots before pacman transactions.

@milkytwix

And we are good to go?

You also need to open up btrfs assistant and create the snapper configs you need. Be sure to name the config for the / filesystem “root”.

This is should be easier to understand once you are in the software.

[milkytwix]

@dalto

You could also install snap-pac if you want snapshots before pacman transactions.

Thanks for mentioning that one!
I also missed yay -S btrfsmaintenance and the most important cmdline: sudo systemctl enable --now grub-btrfsd
But no it is up and running, I will play a bit with it, thanks again man!

@dalto

You also need to open up btrfs assistant and create the snapper configs you need. Be sure to name the config for the / filesystem “root”.

This is should be easier to understand once you are in the software.

Is there a comprehensive… guide for what btrfs-assistant is capable of? Will look for it myself, but this tool is bonkers! Case closed, btw…

[milkytwix]

Nubie reporting for Duty!
This shall be my last post on firefox/manjaro, eos (now with full Librewolf support thanks to being able to fully use AUR) is running quite fine and almost everything is done beside the games - this will take a ludicrous amount of time to get everything up and running, hence back to Manjaro for the evening.
I just wanted to post again to thank you, @dalto for your patience and support! Awesome, man, just awesome! I will stay in the forums to learn more (exypect more stupid questions and "mimimi"s by me) any maybe - just maybe - give some help back some time!
Cya and TYVM!