I have been a happy Endeavour user for the past years…back in the day I had attempted to setup a btrfs system, but at the end I settled with the old ext4.
Now as I am setting up my desktop, I’d like to take the bull from the horns and this time understand and make use of a btrfs filesystem.
In particular, as I have 2 SSDs, I’d like to have the main portion of the system - the standard subvolumes implemeted with Calamares - on the /sda, while I’d use /sdb as userdata storage (documents, music, downloads, pictures,…).
After an initial reading of the official docs and perusing the usual suspects (reddit, SO, Arch, Endeavour, some github repos) what seems to be a generally suggested solution involves:
creation of a system mountpoint /mnt/eOS for on-demand access
creation of a btrfs filesystem in /dev/sdb, and eventually an @userdata subvolume in it
mount top volume (or @userdata subvol) in /mnt/userdata, set its permission (755) and ownership and update fstab
mv/rsync -aAvh data dirs from @home to @userdata, symlinking then back and eventually updating user-dirs.dirs where needed
The other option I’ve found proposed here by @dalto (ref. Move BTRFS @home subvolume to hdd - #2 by dalto) is creating a subvolume for each of the major data dirs in /dev/sdb and then mount them in fstab to their respective location (eg. home/documents).
my questions about
a. among them, although I plan to leverage on borg rather than snapshots for user data, the second solution sounds more logical to me, considering it is a btrfs system; does it inherently have any drawbacks I am not able to see, though? my plan, for all this, is being able to keep my data and the OS independent.
b. incidentally, a concept I’ve realized not being really clear to me is the difference between subvolumes and mount points; when I attach a subvol or directory to /mnt/something in sda, my understanding is that one is simply the entry point to it, while its physical location might be in another block device, where it actually occupy the space, is it correct? hence, in the specific case, I can eventually mount the @userdatacreated in dev/sdb btrfs filesystem to /mnt/userdata/located within the / subvolume in /dev/sda
c. about swap: being a desktop computer with 16Gb RAM, I was thinking about trying zram. Except for giving up the possibility of hibernating the system (resorting to zswap if I’d really want to preserve that), are there any kind of subtle drawbacks to consider in a daily usage?
I thank you all for the observations you might want to contribute with and for the patience of bearing with my questions.
These aren’t mutually exclusive, I would recommend doing both. They serve different purposes. Use borg for long-term data backup and use snapshots for easy access to shorter term. You can painlessly take snapshots on an hourly basis. Even if you only retain snapshots for a week they are a great supplement to traditional backup solutions.
This approach is reasonable for traditional filesystems but none of this makes any sense in btrfs.
Just create the subvolumes you need.
If you are already using btrfs, subvolumes have only benefits IMO.
This is substantially confusing in btrfs due to the implementation. A subvolume is more than a directory or mountpoint. It is also a boundary for many other things. For example:
Each subvolume has it’s own snapshots
Some settings can be changed per subvolume
Snapshots, are themselves, subvolumes
However, subvolumes are located at a path inside the btrfs volume. You can choose to use that path or not. This brings us to the concept of nested and flat subvolumes. Essentially if you want your subvolumes inside each other.
first of all, thank you a lot @dalto for your thorough answer.
That’s in my plans, indeed…just I’d consider borg as the ‘proper’ backup, while snapshots, as you point out as well, as a short-term complement (I thought doing also a btrfs send of weekly snapshot to another non-btrfs HDD I use for backups).
that sounds really more logical, hence mkfs, then create the various subvolumes and finally mount them in fstab, pairing each /home/x with the corresponding subvol on the UUID of the second SSD
that’s kind of what I seem not to be able to wrap my head around…I am aware of the peculiarities of subvolumes you pointed out - snapshots, specific mount options -, but I think, as you said, it’s their nature that makes me confused…I remember having read that subvols do not need to be explicitly mounted to be used, for example…but, considered my preference for flat layouts rather than nested ones, I found mounting kind of mandatory; and hence the doubt about physical occupation of space vs. access to data
about this aspect, reading your past contributions on the forum @dalto and your preference for zram, may I ask, in your opinion, the performance difference between zswap and zram? Tuning the configuration of the first might be a viable solution to achieve a result almost on pair with zram, while preserving the option of hibernation?
I have tried to implement the scheme on a VM in libvirt, and I’d have two issues I’d need a clarification.
after creating a gpt partition table and a btrfs filesystem in /dev/sdb, I mount it temporarily in /mnt to create the various userdata subvolumes; that’s normal these have root:root ownership, requiring a chown -R $USER:$USER before mounting them in fstab (where I use the same mount options of @home subvol, except for the different UUID and the subvol itself, naturally)
on topic of subvolumes, while I see the evident value in creating one for each userdata dir, as well as for virtual machines in var/lib/machines, var/lib/portables and @HOME/.cache (I created them in a flat layout, right below subvol 5 rather than nested within /@ and /home respectively), I am asking myself if I’d have to do that for the flatpak dirs as well, $HOME/.var and the one under var/lib.
About zram/zswap, I tried just implementing the first one, and thanks to the generator, it’s quite straightforward; I’ll check how it performs on bare metal and then eventually create a swapfile if I’d want to switch to zswap.
Thanks again for your kind help, @dalto and wish you a good weekend!
I am not actually sure I understand what the question is related to the first bullet.
These are usually created automatically for you.
It is up to you. You can use “flat” subvolumes which is what the installer does with @ and @home and friends. You could also create nested subvolumes. i.e. you could just create a subvolume at /var/cache. You can also do a mix of both.
When you create nested subvolumes, they don’t need to be mounted in /etc/fstab. So if you created a subvolume at ~/.cache, that would be all you need to do.
I am sorry for my lack of clarity @dalto , I try to detail the process I follow in order to have each of the userdata directories within my /home under its own subvolume on the second drive sdb, with EndeavourOS being installed already in /dev/sda2
create gpt partition table in sdb and sudo mkfs.btrfs -L userdata /dev/sdb1
sudo mount /dev/sdb1 /mnt/userdata
create subvolumes in /mnt/userdata, eg. sudo btrfs subvolume create /mnt/userdata/@documents
edit /etc/fstab, associating each subvol with its own mountpoint UUID=... /home/me/Documents btrfs subvol=@documents,defaults,noatime,compress=zstd 0 0
finally, I systemctl daemon-reload and mount -a
The resulting mounted dirs are now naturally owned by root, but they need to store my documents, hence be accessible and owned by me; for this reason I chown -R me:me /home/me/Documents.
What I was asking is whether the procedure I follow is correct, or I should, for example, include uid-gid in the mountpoint options in fstab.
sorry, my bad, I meant /var/lib/libvirt/images…I don’t actually remember if it creates those automatically as well
I tended to prefer flat layouts as they seem to be easier to manage in terms of snapshots…I also opted for a flat for userdata, as those are created on another drive, with its own btrfs filesystem; on the other hand, the one you suggested about ~/.cache - and similarly might be said for ~/.var - seems far more reasonable than going full flat, hence a mixed approach might be the answer.
Thanks again, @dalto …also for the great btrfs-assistant which makes things far more clear!
I also tend to structure things to fit a snapshot/backup plan, so I can group and / or limit snapshots and backups, like generally avoiding snapshots and backup of /var/tmp as it is in a separate ZFS dataset, while /tmp is tmpfs anyway. Other slices of /var tend to mix and match, and /etc/libvirt and /var/lib/libvirt I tend to have on a separate pool, and the datasets just mount over the top. /var/lib/libvirt/qemu/save is also a good one to restrict snapshots/backup as I don’t care if the memory image is lost, and if I rollback the VM dataset containing the VM image, I just treat it like a crash. Handy to have a separate dataset for each VM so I can rollback, although I supposed same would be true if I used zvols. Works for me anyway, good for a small environment, probably need rethinking for large environment, but I don’t or think EOS as a large corporate environment. My point, I guess, is that it sounds like you have a reasonable plan to structure your BTRFS volumes and subvolumes around.
Snapshots in both nested and flat layouts only take a snapshot of the subvolume. In other words. If you have a subvolume at /home and a nested subvolume at /home/bob/.cache and you take a snapshot of /home, it will not contain /home/bob/.cache
Finally I had the chance to spend the whole Sunday in front of libvirt and experiment with btrfs.
I ultimately defined the layout, a mix of nested subvols where fstab isn’t really necessary (eg. /opt, libvirt and flatpak) and a flat solution for the user data directories, creating the subvols on the second SSD and mounting them to the respective folders in /home, defaulting to @home on the first SSD as per installation.
I also had the time to check a bit about snapshots, implementing both snapper and snap-pac workflows - in this btrfs-assistant is quite a great help, as it gives immediate feedback and the chance to restore easily a snapshot -, observing, as you said @dalto, how they behave both with flat and nested subvolumes.
I have only two unresoved questions left - sorry if they might sound silly, but as a non-IT person, I understand that, at times, I tend to make a mountain out of a molehill
when mounting subvolumes in fstab, what is - if any - the difference between subvol=@documents and subvol=/@documents? for example, I noticed all default subvolumes created by calamares are mounted as subvol=/@subvolume
about permissions, the user data directories are normally owned by the user, with 755 permission. In implementing the various subvolumes on the second drive is it a more suitable approach to change the ownership of the top level @userdata volume from root to user (ie. all children subvolumes will be already user-owned) or it’s better to make the single directories (Documents, Pictures, Music,…) user-owned, leaving the top-level root-owned?
I thank you again for the opportunity to learn more about btrfs and finally, after a long, long time, be able to put a flag on this as well
No difference. It is more standard to use the former but Calamares uses the latter. The end result is the same.
I suppose there is no absolute answer to this. Personally, I always leave the top level owned by root and change the permissions on the individual subvolumes. That way, if I ever create something under the root that shouldn’t be owned by my user account at a later date, I won’t accidentally create a security issue.
