Trouble booting from USB SSD - BTRFS with Subvolumes and LUKS2 Encryption

Hi all,

I’ve been trying to clone my initial EndeavourOS setup from the SD card to a USB SSD. I’m having some success but I can’t get the Pi to fully boot from the SSD yet. Just looking for some assistance/advice if possible.

Some basic system info…

dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.15.74-2-rpi-ARCH (builduser@leming) (aarch64-unknown-linux-gnu-gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Thu Oct 27 17:19:59 MDT 2022
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5

The SSD is a Samsung T7 500GB.

My /boot/cmdline.txt

cat /pi-setup/boot/cmdline.txt
ip=::::rpi-eos:eth0:dhcp cryptdevice=UUID=f5b54fb4-d46d-4d8c-bab4-eca960bac183:cryptroot root=/dev/mapper/cryptroot rootflags=subvol=@ rootfstype=btrfs fsck.repair=no rw modules-load=dwc2,g_ether rootwait console=serial0,115200 console=tty1 usbhid.mousepoll=8 usbhid.mousepoll=8

Block devices and mount points…

lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda             8:0    0 465.8G  0 disk
├─sda1          8:1    0   300M  0 part  /pi-setup/boot
└─sda2          8:2    0   320G  0 part
  └─cryptroot 253:0    0   320G  0 crypt /pi-setup/root/.snapshots
                                         /pi-setup/root/swap
                                         /pi-setup/root/var/cache
                                         /pi-setup/root/var/log
                                         /pi-setup/root/home
                                         /pi-setup/root
mmcblk0       179:0    0  59.5G  0 disk
├─mmcblk0p1   179:1    0   200M  0 part  /boot
└─mmcblk0p2   179:2    0  59.3G  0 part  /var/lib/docker/btrfs
                                         /var/log
                                         /var/cache
                                         /home
                                         /
zram0         252:0    0     1G  0 disk  [SWAP]

BTRFS subvolumes…

btrfs subvolume list /pi-setup/root
ID 256 gen 124 top level 5 path @
ID 257 gen 120 top level 5 path @home
ID 258 gen 123 top level 5 path @log
ID 259 gen 88 top level 5 path @cache
ID 260 gen 19 top level 5 path @swap
ID 262 gen 18 top level 5 path @snapshots

Block ID’s…

blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" UUID="C2E0-D02F" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="339da916-b8c9-4870-9510-32e73ff3a2ab"
/dev/mmcblk0p2: UUID="2a9818bf-eb50-4f31-85b2-bd40d6f82e06" UUID_SUB="d42346ac-290e-42ef-bdd9-c8139fde2854" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="primary" PARTUUID="69f35711-a7ec-4442-8ea0-d5fc28ac2ee2"
/dev/mapper/cryptroot: UUID="41875923-3e91-48d1-a2a6-b05d205521aa" UUID_SUB="bf7f93e2-8e51-4c0b-8014-fd160e30857d" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sda2: UUID="f5b54fb4-d46d-4d8c-bab4-eca960bac183" TYPE="crypto_LUKS" PARTUUID="bdc8446f-02"
/dev/sda1: SEC_TYPE="msdos" UUID="6FFD-E17B" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="bdc8446f-01"
/dev/zram0: UUID="f1fc2acc-a886-4b70-9755-eba4957b551b" TYPE="swap"

The /etc/fstab on the USB SSD looks like this…

cat /pi-setup/root/etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>

# /dev/sda1
UUID=6FFD-E17B  /boot  vfat  defaults  0  0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /               btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=256,subvol=/@            0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /home           btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=257,subvol=/@home        0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /var/log        btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=258,subvol=/@log         0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /var/cache      btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=259,subvol=/@cache       0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /swap           btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=260,subvol=/@swap        0 0

# /dev/sda2
UUID=41875923-3e91-48d1-a2a6-b05d205521aa       /.snapshots     btrfs           rw,noatime,compress=zstd:3,space_cache=v2,ssd,discard=async,commit=30,subvolid=262,subvol=/@snapshots   0 0

During boot, everything appears to go as expected. LUKS2 encryption is working. File system and subvolumes are mounted etc… But then the screen goes blank and turns off instead of fully booting and showing the display manager. In my case gdm.

Is this on a Pi4. On the Pi400 in order to boot from an SSD drive i had to add some quirks.

Edit: Not sure if this is required on the Pi4? I just plugged in the SSD and ran lsusb to get the drive uid.
Then i added the quirk in the /boot/commandline.txt

usb-storage.quirks=0080:a001:u (This is my particular drive UID)

It is a Pi4 yes.

dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.15.74-2-rpi-ARCH (builduser@leming) (aarch64-unknown-linux-gnu-gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Thu Oct 27 17:19:59 MDT 2022
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5

It does seem to be booting from the USB device. With just the SSD connected, no SD card, Uboot loads the kernel, kernel hooks run and I am prompted to decrypt the root partition etc… I then see “OK” messages for each mount point that it was successfully mounted. But shortly after that something seems to go wrong.

I’m booted back on the SD card now. I am trying to look for where those boot logs are on the SSD but can’t seem to find them. There is stuff in the journal but it doesn’t appear to reflect everything I saw during the boot process.

Given that Uboot loads the kernel and I am correctly prompted to decrypt the root partition, would you say I can eliminate a problem with Uboot or my /boot/cmdline.txt? In terms of the root filesystem params.

Did you install from the EndeavourOS ISO?

Edit: I don’t have a Pi4 so I’m not sure. I just know that it’s related to the adapter that connects the ssd to usb on mine for Pi400.

Yes. The initial setup on the SD card was done from the EndeavourOS ISO.

I’ll give the USB quirk thing a try.

lsusb
Bus 002 Device 002: ID 04e8:4001 Samsung Electronics Co., Ltd PSSD T7
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 046d:c548 Logitech, Inc. USB Receiver
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The SSD USB ID is 04e8:4001, but I note that yours has 5 digits on the first part. What does the first 0 mean on your ID, if anything? I also see that you have the u flag set which according to the kernel docs means IGNORE_UAS (don't bind to the uas driver). Is that a typical flag to use in general terms or was that specific for your setup?

This is just how i found to make it work acording to RPI site.

Edit: Sorry i added one too many zeros. Should be this.

usb-storage.quirks=0080:a001:u

Unfortunately, it doesn’t appear to make a difference.

Well for me it does otherwise it doesn’t boot on a USB SSD.

@fsociety3765
Have you set the bootloader to boot from SSD?

How is that accomplished? In the cmdline.txt?

Well before I started any of this, I did set up Raspberry Pi OS and do the EEPROM update if that’s what that is in essence.

rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Tue 18 Oct 10:57:44 UTC 2022 (1666090664)
    LATEST: Tue 26 Apr 10:24:28 UTC 2022 (1650968668)
   RELEASE: critical (/lib/firmware/raspberrypi/bootloader/critical)

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1

IMG_0326.PNG
IMG_0327.PNG
IMG_0328.PNG
IMG_0329.PNG
IMG_0330.PNG

IMG_0331.PNG
IMG_0332.PNG

The above screenshots show what I am seeing onscreen.

Everything going well, then black screen.

Interestingly though, from searching around on the SSD in the journal, the only logs are from when it was running on the SD card. So, it would seem that none of what you see above is being logged in the journal.

While booted using the SD card and manually mounting the file system on the SSD, I’m using the journalctl --root /pi-setup/root to look at the journals on that device. The only logs there are what was already existing when I cloned the file system across using rsync.

Pi400
Not using luks or btrfs

journalctl -b -0 | eos-sendlog
https://0x0.st/oYLj.txt

Screenshot_2022-10-30_19-03-59

There are no journal logs for any of the boots on the SSD. Like in the images above.

Is this indicative of anything in itself?

The fact it mounts the BTRFS subvolumes surely means that has mount the root subvolume right? Only by being able to read the fstab file would it even know about the other subvolumes.