Trouble booting from USB SSD - BTRFS with Subvolumes and LUKS2 Encryption

fsociety3765 · October 30, 2022, 6:48pm

Hi all,

I’ve been trying to clone my initial EndeavourOS setup from the SD card to a USB SSD. I’m having some success but I can’t get the Pi to fully boot from the SSD yet. Just looking for some assistance/advice if possible.

Some basic system info…

dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.15.74-2-rpi-ARCH (builduser@leming) (aarch64-unknown-linux-gnu-gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Thu Oct 27 17:19:59 MDT 2022
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5

The SSD is a Samsung T7 500GB.

My /boot/cmdline.txt…

cat /pi-setup/boot/cmdline.txt
ip=::::rpi-eos:eth0:dhcp cryptdevice=UUID=f5b54fb4-d46d-4d8c-bab4-eca960bac183:cryptroot root=/dev/mapper/cryptroot rootflags=subvol=@ rootfstype=btrfs fsck.repair=no rw modules-load=dwc2,g_ether rootwait console=serial0,115200 console=tty1 usbhid.mousepoll=8 usbhid.mousepoll=8

Block devices and mount points…

lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda             8:0    0 465.8G  0 disk
├─sda1          8:1    0   300M  0 part  /pi-setup/boot
└─sda2          8:2    0   320G  0 part
  └─cryptroot 253:0    0   320G  0 crypt /pi-setup/root/.snapshots
                                         /pi-setup/root/swap
                                         /pi-setup/root/var/cache
                                         /pi-setup/root/var/log
                                         /pi-setup/root/home
                                         /pi-setup/root
mmcblk0       179:0    0  59.5G  0 disk
├─mmcblk0p1   179:1    0   200M  0 part  /boot
└─mmcblk0p2   179:2    0  59.3G  0 part  /var/lib/docker/btrfs
                                         /var/log
                                         /var/cache
                                         /home
                                         /
zram0         252:0    0     1G  0 disk  [SWAP]

BTRFS subvolumes…

btrfs subvolume list /pi-setup/root
ID 256 gen 124 top level 5 path @
ID 257 gen 120 top level 5 path @home
ID 258 gen 123 top level 5 path @log
ID 259 gen 88 top level 5 path @cache
ID 260 gen 19 top level 5 path @swap
ID 262 gen 18 top level 5 path @snapshots

Block ID’s…

blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" UUID="C2E0-D02F" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="339da916-b8c9-4870-9510-32e73ff3a2ab"
/dev/mmcblk0p2: UUID="2a9818bf-eb50-4f31-85b2-bd40d6f82e06" UUID_SUB="d42346ac-290e-42ef-bdd9-c8139fde2854" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="primary" PARTUUID="69f35711-a7ec-4442-8ea0-d5fc28ac2ee2"
/dev/mapper/cryptroot: UUID="41875923-3e91-48d1-a2a6-b05d205521aa" UUID_SUB="bf7f93e2-8e51-4c0b-8014-fd160e30857d" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sda2: UUID="f5b54fb4-d46d-4d8c-bab4-eca960bac183" TYPE="crypto_LUKS" PARTUUID="bdc8446f-02"
/dev/sda1: SEC_TYPE="msdos" UUID="6FFD-E17B" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="bdc8446f-01"
/dev/zram0: UUID="f1fc2acc-a886-4b70-9755-eba4957b551b" TYPE="swap"

The /etc/fstab on the USB SSD looks like this…

cat /pi-setup/root/etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>

# /dev/sda1
UUID=6FFD-E17B  /boot  vfat  defaults  0  0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /               btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=256,subvol=/@            0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /home           btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=257,subvol=/@home        0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /var/log        btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=258,subvol=/@log         0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /var/cache      btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=259,subvol=/@cache       0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /swap           btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=260,subvol=/@swap        0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /.snapshots     btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=262,subvol=/@snapshots   0 0

During boot, everything appears to go as expected. LUKS2 encryption is working. File system and subvolumes are mounted etc… But then the screen goes blank and turns off instead of fully booting and showing the display manager. In my case gdm.

ricklinux · October 30, 2022, 6:53pm

Is this on a Pi4. On the Pi400 in order to boot from an SSD drive i had to add some quirks.

Edit: Not sure if this is required on the Pi4? I just plugged in the SSD and ran lsusb to get the drive uid.
Then i added the quirk in the /boot/commandline.txt

usb-storage.quirks=0080:a001:u (This is my particular drive UID)

fsociety3765 · October 30, 2022, 7:23pm

It is a Pi4 yes.

dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.15.74-2-rpi-ARCH (builduser@leming) (aarch64-unknown-linux-gnu-gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Thu Oct 27 17:19:59 MDT 2022
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5

It does seem to be booting from the USB device. With just the SSD connected, no SD card, Uboot loads the kernel, kernel hooks run and I am prompted to decrypt the root partition etc… I then see “OK” messages for each mount point that it was successfully mounted. But shortly after that something seems to go wrong.

I’m booted back on the SD card now. I am trying to look for where those boot logs are on the SSD but can’t seem to find them. There is stuff in the journal but it doesn’t appear to reflect everything I saw during the boot process.

fsociety3765 · October 30, 2022, 7:28pm

Given that Uboot loads the kernel and I am correctly prompted to decrypt the root partition, would you say I can eliminate a problem with Uboot or my /boot/cmdline.txt? In terms of the root filesystem params.

ricklinux · October 30, 2022, 7:32pm

Did you install from the EndeavourOS ISO?

Edit: I don’t have a Pi4 so I’m not sure. I just know that it’s related to the adapter that connects the ssd to usb on mine for Pi400.

fsociety3765 · October 30, 2022, 7:48pm

Yes. The initial setup on the SD card was done from the EndeavourOS ISO.

fsociety3765 · October 30, 2022, 7:58pm

I’ll give the USB quirk thing a try.

fsociety3765 · October 30, 2022, 8:13pm

lsusb
Bus 002 Device 002: ID 04e8:4001 Samsung Electronics Co., Ltd PSSD T7
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 046d:c548 Logitech, Inc. USB Receiver
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The SSD USB ID is 04e8:4001, but I note that yours has 5 digits on the first part. What does the first 0 mean on your ID, if anything? I also see that you have the u flag set which according to the kernel docs means IGNORE_UAS (don't bind to the uas driver). Is that a typical flag to use in general terms or was that specific for your setup?

ricklinux · October 30, 2022, 8:40pm

This is just how i found to make it work acording to RPI site.

Edit: Sorry i added one too many zeros. Should be this.

usb-storage.quirks=0080:a001:u

fsociety3765 · October 30, 2022, 8:52pm

Unfortunately, it doesn’t appear to make a difference.

ricklinux · October 30, 2022, 8:55pm

Well for me it does otherwise it doesn’t boot on a USB SSD.

ricklinux · October 30, 2022, 9:15pm

@fsociety3765
Have you set the bootloader to boot from SSD?

fsociety3765 · October 30, 2022, 9:23pm

How is that accomplished? In the cmdline.txt?

ricklinux · October 30, 2022, 9:42pm

fsociety3765 · October 30, 2022, 10:16pm

Well before I started any of this, I did set up Raspberry Pi OS and do the EEPROM update if that’s what that is in essence.

rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Tue 18 Oct 10:57:44 UTC 2022 (1666090664)
    LATEST: Tue 26 Apr 10:24:28 UTC 2022 (1650968668)
   RELEASE: critical (/lib/firmware/raspberrypi/bootloader/critical)

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1

fsociety3765 · October 30, 2022, 10:32pm

fsociety3765 · October 30, 2022, 10:33pm

fsociety3765 · October 30, 2022, 10:35pm

The above screenshots show what I am seeing onscreen.

Everything going well, then black screen.

Interestingly though, from searching around on the SSD in the journal, the only logs are from when it was running on the SD card. So, it would seem that none of what you see above is being logged in the journal.

While booted using the SD card and manually mounting the file system on the SSD, I’m using the journalctl --root /pi-setup/root to look at the journals on that device. The only logs there are what was already existing when I cloned the file system across using rsync.

ricklinux · October 30, 2022, 11:00pm

Pi400
Not using luks or btrfs

journalctl -b -0 | eos-sendlog
https://0x0.st/oYLj.txt

Screenshot_2022-10-30_19-03-59

fsociety3765 · October 31, 2022, 12:12am

There are no journal logs for any of the boots on the SSD. Like in the images above.

Is this indicative of anything in itself?

The fact it mounts the BTRFS subvolumes surely means that has mount the root subvolume right? Only by being able to read the fstab file would it even know about the other subvolumes.