Resized LUKS+Btrfs (through GUI) now cannot boot

[hook]

I guess I was asking for trouble …but here we are now and I hope someone has a good idea how to fix it. I’d like to avoid reinstalling, but ultimately, it would not be a huge loss, if I had to.

First off, this is about two SSDs:

• SSD1 is a 1TB Goodram PX500
• SSD2 is a 1TB Samsung 970 Evo Plus

Initial set-up

On my first install, I used Calamares to set up SSD as follows:

• 1 GB – ESP
• 999 GB – LUKS, and within it:
  • 990 GB – Btrfs (default subvolumes)
  • 9 GB – swap

So far so good, all went well for several weeks.

What stupid thing I did

Then I added a second drive into my laptop, to change Btrfs into a Btrfs RAID1.

This is when I noticed that SSD2 is 20 GiB smaller than SSD1. Which means even if I ignore swap and ESP, the whole SSD2 is smaller than the Btrfs partition on SSD1.

Even though the smart people on #Btrfs IRC channel told me the two Btrfs partitions in RAID do not have to be the same size, if I don’t care about some unused space, I got greedy and wanted to optimise my disk use.

I also got lazy and used KDE Partition Manager and before rebooting did:

• resize Btrfs+LUKS on SSD1 to make it 20 GB smaller
• rezize Swap+LUKS on SSD1 to make it 20 GB larger
• create 1 GB ESP on SSD2
• create 970 GB Btrfs+LUKS on SSD2
• create 10 GB Swap+LUKS on SSD2
• I got extra frisky, so while I was at it, I also (re)named the labels of a few partitions

As it all seemed to go well, I felt pretty good and rebooted. Which, of course led to …

The inevitable problem

At first everything looked fine, GRUB and LUKS password, both OK.

SystemD started loading too (at least from initramfs, it seems).

But at this point it stayed still at an uncomfortable amount of time, and worse still, eventually led into many many screens-full of:

… which after what felt minutes ended up with:

Halp!

I brought this upon myself, I confess. I did feel and act a bit too brave.

But if there is reasonable way to fix it, I would be very thankful.

If I need to re-install from scratch, that’s OK, but if there is a (safe) work-around that takes less work, that would be lovely!

More info

I documented my original plans here:

Installation plans for my new laptop - CAN HAS LUKS BTRFS RAID!?

And this is how the actual install went:

First steps of the installation - Leza is born!

There is a whole ongoing blog post series about this install

[dalto]

Can you boot off the live ISO and share the contents of /etc/fstab and lsblk -o name,type,fstype,size,uuid

[hook]

OK, I managed to get it back up. A very helpful bloke emerged on Matrix and walked me through fixing it through the emergency shell.

That was quite a learning experience. I will record it in my next blog post, but it basically boils down to:

The thing that messed up everything was that I deleted and re-created the encrypted swap partition, as then it had a new UUID.

The solution was therefore to enter the emergency shell to figure things out (could have done that from LiveUSB, but it was more fun this way and I had help in a chat) and then:

1. reboot and in GRUB edit the boot command to remove the non-existing swap partition from the kernel line
2. wait during boot that SystemD gives up on the non-existing swap partition
3. sudoedit /mnt/rootfs/etc/crypttab and sudoedit /mnt/rootfs/etc/fstab to change the UUID of the encrypted swap partition to the new partition’s UUID
4. sudoedit /etc/dracut.conf.d/calamares-luks.conf to use the new swap partitions UUID
5. sudo dracut-rebuild
6. edit /etc/default/grub to use the new swap partitions UUID
7. sudo grub-mkconfig
8. reboot

(Super thanks @dalto for offering your help.)

For some reason it now asks me for the decryption password twice:

• first at the very start (as it did before)
• then again during boot, when Plymouth is already running (this is new)

It seems the second time it asks it’s about the swap partition.

[dalto]

@hook

For some reason it now asks me for the decryption password twice:

You need to add the existing keyfile to your new swap partition using cryptsetup. The keyfile is in the root.

[hook]

Can I add the same keyfile also to the newly created Btrfs and Swap partitions on SSD2?

[dalto]

You can add it to any luks device.

[hook]

Nice, I’ll try that then

And if the two root Btrfs partitions are in RAID1, I don’t need to worry about one disappearing, because both will have a copy of the keyfile, right?

[dalto]

Did you create a separate btrfs filesystem on the other disk already? If you want them in raid 1, that isn’t how you do that. You add the partition to your existing filesystem. Then there is only one filesystem with two copies of the data.

[hook]

@dalto

Did you create a separate btrfs filesystem on the other disk already?

I did, yes.

If you want them in raid 1, that isn’t how you do that. You add the partition to your existing filesystem. Then there is only one filesystem with two copies of the data.

I noticed this is an issue, yes:

btrfs device add /dev/disk/by-uuid/a19847bc-d137-4443-9cd5-9f311a5d8636 / says:

ERROR: /dev/disk/by-uuid/a19847bc-d137-4443-9cd5-9f311a5d8636 appears to contain an existing filesystem (btrfs)
ERROR: use the -f option to force overwrite of /dev/disk/by-uuid/a19847bc-d137-4443-9cd5-9f311a5d8636

I guess I should btrfs device add /dev/disk/by-uuid/a19847bc-d137-4443-9cd5-9f311a5d8636 / --force and it should be fine? Given that it is an empty unused btrfs partition. I am following some sane instructions from #btrfs IRC channel, will update here once finished.

[hook]

I managed to get the Btrfs RAID1 working (will update above post later).

But after reboot, I boot fails me yet again (I waited >10 minutes and then gave up).

I suppose at this stage it is just easier to boot from LiveUSB and see what went wrong now.

/dev/disk/by-uuid/e1d4963d-… is the root= disk in GRUB.

lsblk -f after

mkdir --parents /mnt/rootfs
cryptsetup open /dev/disk/by-uuid/1a45a072-[…] rootfs
cryptsetup open /dev/disk/by-uuid/c82fca05-[…] rootfs2

• nvme0n1 = Goodram SSD (a.k.a. the SSD1 that was in the laptop from the beginning)
  • nvme0n1p1 = UUID=B33A-4C29 = EFI
  • nvme0n1p2 = UUID=1a45a072-[…] = LUKS partition with Btrfs in RAID1
    • UUID=e1d4963d = Btrfs in RAID1
  • nvme0n1p3 = UUID=2d90af35-[…] = LUKS partition with Swap
• nvme1n1 = Samsung SSD (a.k.a. the newly added SSD2)
  • nvme1n1p1 = UUID=44D2-04AD = backup EFI (to be rsync’d from actual EFI)
  • nvme1n1p2 = UUID=c82fca05-[…] = LUKS partition with Btrfs in RAID1
    • UUID=e1d4963d = Btrfs in RAID1
  • nvme1n1p3 = UUID=2e91342f-[…] = LUKS partition with Swap

btrfs filesystem usage -T /mnt/rootfs after

mount /dev/mapper/rootfs /mnt/rootfs

[dalto]

Did you add the UUID of the second luks partition to be unlocked as part of the kernel cmdline?

Are you using grub or systemd-boot?

[hook]

Using GRUB.

I managed to get to the point where I can manually boot it, if I edit the entry from GRUB during boot and add rd.luks.uuid= for the SSD2 LUKS partition(s). Apart from removing loglevel=3 splash quiet, that is the only change I made in the GRUB entry.

But for some reason if I grub-install it produces an error saying:

grub-install: error: attempt to install to encrypted disk without cryptodisk enabled. Set GRUB_ENABLE_CRYPTODISK=y in file /etc/default/grub.

But that option is already enabled in that file.

[dalto]

Can we see the contents of /etc/default/grub

[hook]

Of course (now that I can boot into it, it’s much easier):

# GRUB boot loader configuration

GRUB_DEFAULT='0'
GRUB_TIMEOUT='5'
GRUB_DISTRIBUTOR='EndeavourOS'
GRUB_CMDLINE_LINUX_DEFAULT='nowatchdog nvme_load=YES rd.luks.uuid=1a45a072-e9ed-4416-ac7e-04b69f11a9cc rd.luks.uuid=2d90af35-7e6a-40f8-8353-f20433d0f994 resume=/dev/mapper/luks-2d90af35-7e6a-40f8-8353-f20433d0f994 loglevel=3 splash quiet
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
GRUB_ENABLE_CRYPTODISK=y

# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY='true'

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
GRUB_BACKGROUND='/usr/share/endeavouros/splash.png'
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT=true

# Uncomment to disable submenus in boot menu
GRUB_DISABLE_SUBMENU='false'

# Probing for other operating systems is disabled for security reasons. Read
# documentation on GRUB_DISABLE_OS_PROBER, if still want to enable this
# functionality install os-prober and uncomment to detect and include other
# operating systems.
#GRUB_DISABLE_OS_PROBER=false

[hook]

I managed to fix the (main) booting problem. I had to add the other UUID behind LUKS as well as close the quote in the line (silly me):

GRUB_CMDLINE_LINUX_DEFAULT='nowatchdog nvme_load=YES rd.luks.uuid=1a45a072-e9ed-4416-ac7e-04b69f11a9cc rd.luks.uuid=c82fca05-59d3-4595-969b-c1c4124d8559 rd.luks.uuid=2d90af35-7e6a-40f8-8353-f20433d0f994 rd.luks.uuid=2e91342f-3d19-4f75-a9a6-fc3f9798cb30 resume=/dev/mapper/luks-2d90af35-7e6a-40f8-8353-f20433d0f994 loglevel=3 splash quiet'

Now it boots, but asks me three times to provide the LUKS password:

• twice before GRUB, once for each SSD
• once during Plymouth/SystemD

[hook]

@hook

Now it boots, but asks me three times to provide the LUKS password:

• twice before GRUB, once for each SSD
• once during Plymouth/SystemD

This is currently the only gripe I have after I set up Btrfs RAID1.