"unknown filesystem" grub rescue at boot after system lockup

Agh!
Moved laptop earlier, heard a succession of beeps, next time I look the capslock is blinking - oh oh. System unresponsive, REISUB didn’t help (sure I set it up), did a hard reset.

On next boot I get “Unknown filesystem” & rescue prompt. System on NVMe (~1.5 years old) of 3 partitions: ESP VFAT, Root BTRFS, swap. Booted to Ganymede live, did some checks shown below.

lsblk -f

NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1
│ vfat FAT32 6A4B-DB32
├─nvme0n1p2

└─nvme0n1p3
swap 1 swap 112c0258-1344-4a02-8451-918ffacd3052

sudo modprobe btrfs

sudo blkid /dev/nvme0n1p2
/dev/nvme0n1p2: PARTLABEL=“endeavouros” PARTUUID=“b96e852a-4a54-48e7-9a4b-ea8762994c65”
sudo btrfs filesystem show /dev/nvme0n1p2
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
sudo btrfs inspect-internal dump-super -f /dev/nvme0n1p2

superblock: bytenr=65536, device=/dev/nvme0n1p2

csum_type 0 (crc32c)
csum_size 4
csum 0xe67d9754 [DON’T MATCH]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 981f6265-bd51-4cdb-ab52-216221398977
metadata_uuid 00000000-0000-0000-0000-000000000000
label endeavouros
generation 215966
root 104708931584
sys_array_size 129
chunk_root_generation 199704
root_level 1
chunk_root 23642112
chunk_root_level 0
log_root 104673148928
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 236668776448
bytes_used 92219805696
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x3
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 215966
dev_item.uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
dev_item.fsid 981f6265-bd51-4cdb-ab52-216221398977 [match]
dev_item.type 0
dev_item.total_bytes 236668776448
dev_item.bytes_used 110620573696
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 274886295552 owner 274877906946 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
stripe 1 devid 1 offset 30408704
dev_uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
backup_roots[4]:
backup 0:
backup_tree_root: 104703639552 gen: 215965 level: 1
backup_chunk_root: 23642112 gen: 199704 level: 0
backup_extent_root: 104650047488 gen: 215965 level: 2
backup_fs_root: 104286453760 gen: 215812 level: 1
backup_dev_root: 104207532032 gen: 199704 level: 0
csum_root: 104650457088 gen: 215965 level: 2
backup_total_bytes: 236668776448
backup_bytes_used: 92218572800
backup_num_devices: 1

backup 1:
backup_tree_root: 104708931584 gen: 215966 level: 1
backup_chunk_root: 23642112 gen: 199704 level: 0
backup_extent_root: 104655486976 gen: 70368744393630 level: 2
backup_fs_root: 70473030631424 gen: 215812 level: 1
backup_dev_root: 104207532032 gen: 199704 level: 0
csum_root: 104623800320 gen: 215966 level: 2
backup_total_bytes: 236668776448
backup_bytes_used: 70460963983360
backup_num_devices: 1

backup 2:
backup_tree_root: 104682831872 gen: 215963 level: 1
backup_chunk_root: 23642112 gen: 199704 level: 0
backup_extent_root: 104632975360 gen: 215963 level: 2
backup_fs_root: 104286453760 gen: 215812 level: 1
backup_dev_root: 104207532032 gen: 199704 level: 0
csum_root: 104630681600 gen: 215963 level: 2
backup_total_bytes: 236668776448
backup_bytes_used: 92218445824
backup_num_devices: 1

backup 3:
backup_tree_root: 104695316480 gen: 215964 level: 1
backup_chunk_root: 23642112 gen: 199704 level: 0
backup_extent_root: 104652996608 gen: 215964 level: 2
backup_fs_root: 104286453760 gen: 215812 level: 1
backup_dev_root: 104207532032 gen: 199704 level: 0
csum_root: 104652144640 gen: 215964 level: 2
backup_total_bytes: 236668776448
backup_bytes_used: 92218318848
backup_num_devices: 1

sudo btrfs inspect-internal dump-super --all /dev/nvme0n1p2

superblock: bytenr=65536, device=/dev/nvme0n1p2

csum_type 0 (crc32c)
csum_size 4
csum 0xe67d9754 [DON’T MATCH]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 981f6265-bd51-4cdb-ab52-216221398977
metadata_uuid 00000000-0000-0000-0000-000000000000
label endeavouros
generation 215966
root 104708931584
sys_array_size 129
chunk_root_generation 199704
root_level 1
chunk_root 23642112
chunk_root_level 0
log_root 104673148928
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 236668776448
bytes_used 92219805696
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x3
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 215966
dev_item.uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
dev_item.fsid 981f6265-bd51-4cdb-ab52-216221398977 [match]
dev_item.type 0
dev_item.total_bytes 236668776448
dev_item.bytes_used 110620573696
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

superblock: bytenr=67108864, device=/dev/nvme0n1p2

csum_type 0 (crc32c)
csum_size 4
csum 0x6dc085e6 [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 981f6265-bd51-4cdb-ab52-216221398977
metadata_uuid 00000000-0000-0000-0000-000000000000
label endeavouros
generation 215966
root 104708931584
sys_array_size 129
chunk_root_generation 199704
root_level 1
chunk_root 23642112
chunk_root_level 0
log_root 0
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 236668776448
bytes_used 92219805696
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x3
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 215966
dev_item.uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
dev_item.fsid 981f6265-bd51-4cdb-ab52-216221398977 [match]
dev_item.type 0
dev_item.total_bytes 236668776448
dev_item.bytes_used 110620573696
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

sudo btrfs rescue super-recover /dev/nvme0n1p2

ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: n
Aborted to recover bad superblocks

Was anxious about the triple superblock crc mismatch output so I cancelled. Backup superblock 1 seems fine when inspected, backup superblock 2 seems absent (nil output from command):

sudo btrfs inspect-internal dump-super --super 1 /dev/nvme0n1p2

superblock: bytenr=67108864, device=/dev/nvme0n1p2

csum_type 0 (crc32c)
csum_size 4
csum 0x6dc085e6 [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 981f6265-bd51-4cdb-ab52-216221398977
metadata_uuid 00000000-0000-0000-0000-000000000000
label endeavouros
generation 215966
root 104708931584
sys_array_size 129
chunk_root_generation 199704
root_level 1
chunk_root 23642112
chunk_root_level 0
log_root 0
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 236668776448
bytes_used 92219805696
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x3
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 215966
dev_item.uuid 9ee7a29f-9711-4178-8c78-a5b412517e8d
dev_item.fsid 981f6265-bd51-4cdb-ab52-216221398977 [match]
dev_item.type 0
dev_item.total_bytes 236668776448
dev_item.bytes_used 110620573696
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

I tried a RO mount but this happened:

sudo mount -o ro,subvol=@,rescue=usebackuproot /dev/nvme0n1p2 /mnt

mount: /mnt: fsconfig() failed: vfat: Unknown parameter ‘subvol’.
dmesg(1) may have more information after failed mount system call.

I’m tempted to try sudo btrfs rescue super-recover /dev/nvme0n1p2 & see what happens but I don’t want to lose this filsystem if I can help it, recovery is prefereable.

Can anyone help?

only to mention.. doing rescue operations on the only copy you have is never the best idea.
I can not help much on BTRFS repair.
But BTRFS calls for a backup solution, even a little more compared to ext4.

1 Like

For anyone wondering if the NVMe is failing (hint: nope):

sudo smartctl -a /dev/nvme0n1

smartctl 7.5 2025-04-30 r5714 [x86_64-linux-6.17.8-arch1-1] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: SAMSUNG MZVLQ256HAJD-000H1
Serial Number: S4UJNX3RA78390
Firmware Version: HPS4NFXV
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 256,060,514,304 [256 GB]
Unallocated NVM Capacity: 0
Controller ID: 5
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 256,060,514,304 [256 GB]
Namespace 1 Utilization: 108,551,933,952 [108 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 aa11b9d2b0
Local Time is: Sat Feb 14 20:08:08 2026 AEST
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 82 Celsius
Critical Comp. Temp. Threshold: 85 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 4.86W - - 0 0 0 0 0 0
1 + 4.49W - - 1 1 1 1 0 0
2 + 2.37W - - 2 2 2 2 0 500
3 - 0.0500W - - 3 3 3 3 210 1200
4 - 0.0050W - - 4 4 4 4 1000 9000

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
1 - 4096 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0x1)
Critical Warning: 0x00
Temperature: 24 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 1%
Data Units Read: 807,789 [413 GB]
Data Units Written: 6,278,928 [3.21 TB]
Host Read Commands: 11,466,072
Host Write Commands: 189,618,994
Controller Busy Time: 672
Power Cycles: 418
Power On Hours: 208
Unsafe Shutdowns: 32
Media and Data Integrity Errors: 0
Error Information Log Entries: 225
Warning Comp. Temperature Time: 4
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 24 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message
0 225 0 0x2019 0x4004 - 0 0 - Invalid Field in Command

Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code
0 Short Completed without error 0 - - - - -

Also, verbose btrfs rescue super-recover output:

sudo btrfs rescue super-recover -v /dev/nvme0n1p2
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
All Devices:
       Device: id = 1, name = /dev/nvme0n1p2
ERROR: superblock checksum mismatch: wanted 0xe67d9754 found 0xd136d0c3
Before Recovering:
       [All good supers]:
               device name = /dev/nvme0n1p2
               superblock bytenr = 67108864
[All bad supers]:
               device name = /dev/nvme0n1p2
               superblock bytenr = 65536
Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: y
Recovered bad superblocks successful

Yep, I went for it.

Next step - check filesystem isn’t a potato:

sudo mkdir -p /mnt/rootfs 
sudo mount -o ro /dev/nvme0n1p2 /mnt/rootfs/
sudo btrfs subvolume list /mnt/rootfs  
ID 256 gen 214190 top level 5 path timeshift-btrfs/snapshots/2024-08-04_10-11-10/@ 
ID 257 gen 6776 top level 5 path timeshift-btrfs/snapshots/2024-10-20_16-27-20/@home 
ID 258 gen 214429 top level 5 path @cache 
ID 259 gen 215957 top level 5 path @log
…etc…
ID 300 gen 215966 top level 5 path @home 
ID 301 gen 215966 top level 5 path @
…etc…

We have fs, yay.

Next, let’s sudo umount /mnt/rootfs/ & try mounting our subvolumes (@ + @home) & ESP:

sudo mkdir -p /mnt/home
sudo mount -o subvol=@ /dev/nvme0n1p2 /mnt 
sudo mount -o subvol=@home /dev/nvme0n1p2 /mnt/home 
sudo mount /dev/nvme0n1p1 /mnt/boot/efi

After browsing to confirm valid dirs & files including /home/bettertryreallyfixingsystem we can chroot & check initramfs contains btrfs modules:

sudo arch-chroot /mnt
dracut --list-modules /mnt/boot/initramfs-linux.img | grep btrfs    
btrfs 
btrfs-snapshot-overlay

W007! We have readable initramfs img files, hopefully also modules & grub config. Let’s check our modules:

ls /mnt/usr/lib/modules/*/

/mnt/usr/lib/modules/6.12.63-1-lts/:
build              modules.builtin            modules.dep      modules.softdep      pkgbase
kernel             modules.builtin.alias.bin  modules.dep.bin  modules.symbols      updates
modules.alias      modules.builtin.bin        modules.devname  modules.symbols.bin  vmlinuz
modules.alias.bin  modules.builtin.modinfo    modules.order    modules.weakdep

/mnt/usr/lib/modules/6.18.2-arch2-1/:
build              modules.builtin            modules.dep      modules.softdep      pkgbase
kernel             modules.builtin.alias.bin  modules.dep.bin  modules.symbols      updates
modules.alias      modules.builtin.bin        modules.devname  modules.symbols.bin  vdso
modules.alias.bin  modules.builtin.modinfo    modules.order    modules.weakdep      vmlinuz


And grub config at less /boot/grub/grub.cfg seems ok, let’s check if GRUB is installed in the boot sector:

sudo dd if=/dev/nvme0n1 bs=512 count=1 2>/dev/null | strings | grep -i grub
sudo dd if=/dev/nvme0n1 bs=2 count=1 2>/dev/null | od -Ax -tx1 | head -n1 | grep eb 

Ah, well, seems GRUB isn’t there anymore so let’s install it & regenerate the config:

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=endeavouros
Installing for x86_64-efi platform. 
Installation finished. No error reported.

grub-mkconfig -o /boot/grub/grub.cfg 
Generating grub configuration file ... 
Found theme: /boot/grub/themes/poly-dark/theme.txt 
Found linux image: /boot/vmlinuz-linux-lts 
Found initrd image: /boot/initramfs-linux-lts.img 
Found fallback initrd image(s) in /boot:  initramfs-linux-lts-fallback.img 
Found linux image: /boot/vmlinuz-linux 
Found initrd image: /boot/initramfs-linux.img 
Found fallback initrd image(s) in /boot:  initramfs-linux-fallback.img 
Warning: os-prober will not be executed to detect other bootable partitions. 
Systems on them will not be added to the GRUB boot configuration. 
Check GRUB_DISABLE_OS_PROBER documentation entry. 
Adding boot menu entry for UEFI Firmware Settings ... 
Detecting snapshots ... 
Found snapshot:   ..etc…
Found 20 snapshot(s) 
Unmount /tmp/grub-btrfs.486j4LP6Ma .. Success 
done

Well, let’s exit & sudo umount -R /mnt then try a reboot - see you on the flipside :crossed_fingers:

This says it all: https://www.youtube.com/watch?v=lX8GG3dnsp8

So glad the EnOS live image is so feature rich, including KDE Connect - that made throwing things between phone/etc & affected laptop so much easier.

And yeah, as Joe said :backhand_index_pointing_up:

“doing rescue operations on the only copy you have is never the best idea”.

- Der Doktor

First time for everything I guess, hopefully (likely not) the last time.

Now, to double check REISUB implementation…

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.