BTRFS: File system unmountable, no boot and access possible

General system Newbie
1

Trying to boot this morning resulted in this message:

Starting systemd-udevd, version 252.2-3-arch
mount: /new_root: mount(2) system call failed: File exists.
dmesg(1) may have more information after failed mount system call
You are now dropped into an emergency shell
sh: can't access ty: job control turned off

Strange, but what the heck, I have snapshots, always created before I upgrade the system. Trying to boot into any of the snapshots gives the same message.

Booting into a USB and trying to figure out what the problem is, I see this message in journalctl whenever I try to mount (sudo mount -t btrfs /dev/nvme0n1p2 /mnt) my NVME SSD:

Dec 06 09:02:52 EndeavourOS kernel: BTRFS info (device nvme0n1p2): flagging fs with big metadata feature
Dec 06 09:02:52 EndeavourOS kernel: BTRFS info (device nvme0n1p2): disk space caching is enabled
Dec 06 09:02:52 EndeavourOS kernel: BTRFS info (device nvme0n1p2): has skinny extents
Dec 06 09:02:52 EndeavourOS kernel: BTRFS info (device nvme0n1p2): enabling ssd optimizations
Dec 06 09:02:52 EndeavourOS kernel: BTRFS info (device nvme0n1p2): start tree-log replay
Dec 06 09:02:53 EndeavourOS kernel: ------------[ cut here ]------------
Dec 06 09:02:53 EndeavourOS kernel: BTRFS: Transaction aborted (error -17)
Dec 06 09:02:53 EndeavourOS kernel: WARNING: CPU: 3 PID: 5733 at fs/btrfs/delayed-inode.c:1088 __btrfs_run_delayed_items+0x16d/0x2a0 [btrfs]
Dec 06 09:02:53 EndeavourOS kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq rfkill nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) snd_hda_codec_realtek intel_rapl_msr intel_rapl_common snd_hda_codec_generic ledtrig_audio sn>
Dec 06 09:02:53 EndeavourOS kernel: CPU: 3 PID: 5733 Comm: mount Tainted: P        W  OE     5.15.8-arch1-1 #1 53ba9287dd5ea74b4b3d62267b6afa78f9f903d2
Dec 06 09:02:53 EndeavourOS kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C91/MAG B550 TOMAHAWK (MS-7C91), BIOS A.40 10/29/2020
Dec 06 09:02:53 EndeavourOS kernel: RIP: 0010:__btrfs_run_delayed_items+0x16d/0x2a0 [btrfs]
Dec 06 09:02:53 EndeavourOS kernel: Code: 48 0f ba 2a 03 72 25 41 83 ff fb 0f 84 16 01 00 00 41 83 ff e2 0f 84 0c 01 00 00 44 89 fe 48 c7 c7 d8 cc 65 c0 e8 87 76 96 d7 <0f> 0b 44 89 f9 ba 40 04 00 00 48 c7 c6 70 de 64 c0 48 89 df e8 d1
Dec 06 09:02:53 EndeavourOS kernel: RSP: 0018:ffff9fe855fc7738 EFLAGS: 00010246
Dec 06 09:02:53 EndeavourOS kernel: RAX: 0000000000000000 RBX: ffff8be30a6daf08 RCX: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: RBP: ffff8be37ba4c270 R08: 0000000000000000 R09: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8be37ba4c2c8
Dec 06 09:02:53 EndeavourOS kernel: R13: ffff8be37e0d6d20 R14: ffff8be377046800 R15: 00000000ffffffef
Dec 06 09:02:53 EndeavourOS kernel: FS:  00007f4c628bd740(0000) GS:ffff8bea0eac0000(0000) knlGS:0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 06 09:02:53 EndeavourOS kernel: CR2: 00007feed23cfe98 CR3: 00000001722be000 CR4: 0000000000750ee0
Dec 06 09:02:53 EndeavourOS kernel: PKRU: 55555554
Dec 06 09:02:53 EndeavourOS kernel: Call Trace:
Dec 06 09:02:53 EndeavourOS kernel:  <TASK>
Dec 06 09:02:53 EndeavourOS kernel:  drop_one_dir_item+0x153/0x170 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  add_inode_ref+0xc05/0x10b0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  replay_one_buffer+0x1d2/0x4c0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  ? find_extent_buffer+0x56/0x90 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  walk_down_log_tree+0x181/0x3a0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  walk_log_tree+0xde/0x220 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  btrfs_recover_log_trees+0x217/0x3d0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  ? log_new_dir_dentries+0x510/0x510 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  open_ctree+0x1454/0x16e3 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  btrfs_mount_root.cold+0x13/0xfa [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  legacy_get_tree+0x27/0x50
Dec 06 09:02:53 EndeavourOS kernel:  vfs_get_tree+0x25/0xc0
Dec 06 09:02:53 EndeavourOS kernel:  vfs_kern_mount.part.0+0x79/0xc0
Dec 06 09:02:53 EndeavourOS kernel:  btrfs_mount+0x15f/0x430 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  ? legacy_get_tree+0x27/0x50
Dec 06 09:02:53 EndeavourOS kernel:  legacy_get_tree+0x27/0x50
Dec 06 09:02:53 EndeavourOS kernel:  vfs_get_tree+0x25/0xc0
Dec 06 09:02:53 EndeavourOS kernel:  path_mount+0x485/0xaf0
Dec 06 09:02:53 EndeavourOS kernel:  __x64_sys_mount+0x11f/0x160
Dec 06 09:02:53 EndeavourOS kernel:  do_syscall_64+0x5c/0x90
Dec 06 09:02:53 EndeavourOS kernel:  ? syscall_exit_to_user_mode+0x23/0x50
Dec 06 09:02:53 EndeavourOS kernel:  ? do_syscall_64+0x69/0x90
Dec 06 09:02:53 EndeavourOS kernel:  ? do_syscall_64+0x69/0x90
Dec 06 09:02:53 EndeavourOS kernel:  ? syscall_exit_to_user_mode+0x23/0x50
Dec 06 09:02:53 EndeavourOS kernel:  ? do_syscall_64+0x69/0x90
Dec 06 09:02:53 EndeavourOS kernel:  ? exc_page_fault+0x72/0x180
Dec 06 09:02:53 EndeavourOS kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Dec 06 09:02:53 EndeavourOS kernel: RIP: 0033:0x7f4c62a4344e
Dec 06 09:02:53 EndeavourOS kernel: Code: 48 8b 0d 25 0a 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f2 09 0c 00 f7 d8 64 89 01 48
Dec 06 09:02:53 EndeavourOS kernel: RSP: 002b:00007ffee7449f98 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
Dec 06 09:02:53 EndeavourOS kernel: RAX: ffffffffffffffda RBX: 00007f4c62b6f264 RCX: 00007f4c62a4344e
Dec 06 09:02:53 EndeavourOS kernel: RDX: 000055cdadd01780 RSI: 000055cdadd017c0 RDI: 000055cdadd017a0
Dec 06 09:02:53 EndeavourOS kernel: RBP: 000055cdadd01550 R08: 0000000000000000 R09: 00007f4c62b04a60
Dec 06 09:02:53 EndeavourOS kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: R13: 000055cdadd017a0 R14: 000055cdadd01780 R15: 000055cdadd01550
Dec 06 09:02:53 EndeavourOS kernel:  </TASK>
Dec 06 09:02:53 EndeavourOS kernel: ---[ end trace 30b74faa20f6ad76 ]---
Dec 06 09:02:53 EndeavourOS kernel: BTRFS: error (device nvme0n1p2) in __btrfs_run_delayed_items:1088: errno=-17 Object already exists
Dec 06 09:02:53 EndeavourOS kernel: BTRFS: error (device nvme0n1p2) in btrfs_replay_log:2415: errno=-17 Object already exists (Failed to recover log tree)
Dec 06 09:02:53 EndeavourOS kernel: ------------[ cut here ]------------
Dec 06 09:02:53 EndeavourOS kernel: WARNING: CPU: 4 PID: 5770 at fs/btrfs/transaction.c:130 btrfs_put_transaction+0x127/0x130 [btrfs]
Dec 06 09:02:53 EndeavourOS kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq rfkill nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) snd_hda_codec_realtek intel_rapl_msr intel_rapl_common snd_hda_codec_generic ledtrig_audio sn>
Dec 06 09:02:53 EndeavourOS kernel: CPU: 4 PID: 5770 Comm: btrfs-transacti Tainted: P        W  OE     5.15.8-arch1-1 #1 53ba9287dd5ea74b4b3d62267b6afa78f9f903d2
Dec 06 09:02:53 EndeavourOS kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C91/MAG B550 TOMAHAWK (MS-7C91), BIOS A.40 10/29/2020
Dec 06 09:02:53 EndeavourOS kernel: RIP: 0010:btrfs_put_transaction+0x127/0x130 [btrfs]
Dec 06 09:02:53 EndeavourOS kernel: Code: 5d e9 7d 96 16 d7 0f 0b 5b be 03 00 00 00 5d 41 5c 41 5d e9 5b b9 39 d7 0f 0b e9 fd fe ff ff 0f 0b eb d5 0f 0b e9 3d ff ff ff <0f> 0b e9 46 ff ff ff 66 90 0f 1f 44 00 00 41 54 4c 8d a7 60 04 00
Dec 06 09:02:53 EndeavourOS kernel: RSP: 0018:ffff9fe8562c7e28 EFLAGS: 00010286
Dec 06 09:02:53 EndeavourOS kernel: RAX: ffff8be3673964c0 RBX: ffff8be32b01e488 RCX: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: RDX: ffff8be3672b3628 RSI: 0000000000000000 RDI: ffff8be3672b3610
Dec 06 09:02:53 EndeavourOS kernel: RBP: ffff8be3672b3600 R08: 0000000000000000 R09: 0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8be32b01e000
Dec 06 09:02:53 EndeavourOS kernel: R13: ffff8be32b01e460 R14: ffff8be36285a800 R15: ffff8be3672b3628
Dec 06 09:02:53 EndeavourOS kernel: FS:  0000000000000000(0000) GS:ffff8bea0eb00000(0000) knlGS:0000000000000000
Dec 06 09:02:53 EndeavourOS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 06 09:02:53 EndeavourOS kernel: CR2: 00007f3096ba1500 CR3: 00000001ce010000 CR4: 0000000000750ee0
Dec 06 09:02:53 EndeavourOS kernel: PKRU: 55555554
Dec 06 09:02:53 EndeavourOS kernel: Call Trace:
Dec 06 09:02:53 EndeavourOS kernel:  <TASK>
Dec 06 09:02:53 EndeavourOS kernel:  btrfs_cleanup_transaction.isra.0+0x10f/0x5a0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  ? try_to_wake_up+0x213/0x590
Dec 06 09:02:53 EndeavourOS kernel:  transaction_kthread+0x18e/0x1a0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  ? btrfs_cleanup_transaction.isra.0+0x5a0/0x5a0 [btrfs 7a43803856b6703d145c653b8b38db14276f7e69]
Dec 06 09:02:53 EndeavourOS kernel:  kthread+0x132/0x160
Dec 06 09:02:53 EndeavourOS kernel:  ? set_kthread_struct+0x50/0x50
Dec 06 09:02:53 EndeavourOS kernel:  ret_from_fork+0x22/0x30
Dec 06 09:02:53 EndeavourOS kernel:  </TASK>
Dec 06 09:02:53 EndeavourOS kernel: ---[ end trace 30b74faa20f6ad77 ]---
Dec 06 09:02:53 EndeavourOS kernel: BTRFS error (device nvme0n1p2): open_ctree failed

Then I try to check using btrfs check what is going on with the SSD. But no error occurs:

[liveuser@eos-2021.12.17 ~]$ sudo btrfs check --check-data-csum -p /dev/nvme0n1p2
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 21b3cc23-6e76-45bf-acd2-d99dc24cc0d8
[1/7] checking root items                      (0:00:02 elapsed, 2083036 items checked)
[2/7] checking extents                         (0:00:13 elapsed, 244780 items checked)
[3/7] checking free space cache                (0:00:00 elapsed, 1148 items checked)
[4/7] checking fs roots                        (0:00:29 elapsed, 147302 items checked)
[5/7] checking csums against data              (0:10:28 elapsed, 712198 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 30 items checked)
[7/7] checking quota groups                    (0:00:24 elapsed, 1843328 items checked)
found 1174419738624 bytes used, no error found
total csum bytes: 1141823932
total tree bytes: 4009918464
total fs tree bytes: 2423521280
total extent tree bytes: 225329152
btree space waste bytes: 732619798
file data blocks allocated: 2454781140992
 referenced 1403791265792
[liveuser@eos-2021.12.17 ~]$

I’m kind of at a loss at what to do now. Any ideas on how to proceed?

2

Not sure if this is related? I’m no expert on Btrfs. Maybe you wait for a reply from @dalto

https://wiki.archlinux.org/title/Btrfs#BTRFS:_open_ctree_failed

3

Thanks for the link, although its probably not the same problem. My system is not a multi-device BTRFS, and my error message looks different, the part

mount: wrong fs type, bad option, bad superblock on /dev/sdb2, missing codepage or helper program, or other error

is not there for me.

1 Like
4

Is it something to do with this then?

https://www.suse.com/support/kb/doc/?id=000018761

5

I’m sceptical, as the error message in the support database

mount: wrong fs type, bad option, bad superblock on /dev/<devicename>, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so

does not happen for me. Here’s how a try of mounting the BTRFS partition using the rescue USB looks for me:

[liveuser@eos-2021.12.17 ~]$ sudo mount -t btrfs /dev/nvme0n1p2 /mnt
mount: /mnt: mount(2) system call failed: File exists.
[liveuser@eos-2021.12.17 ~]$

I usually assume that the error message should match perfectly, although with newer versions of the tools messages can change.

Edit: The stacktrace looks different as well.

6

Shouldn’t that be:

sudo mount /dev/nvme0n1p2  /mnt -o subvol=@

? From memory, so could be wrong…

7

Think It is

sudo mount -o subvol=@ /dev/nvme0n1p2 /mnt
8

Same thing, different order :smiley:

9

Oh…the order don’t matter? :man_shrugging:

10

Well, at least that’s not the problem:

[liveuser@eos-2021.12.17 ~]$ sudo mount -o subvol=@ /dev/nvme0n1p2 /mnt
mount: /mnt: mount(2) system call failed: File exists.
[liveuser@eos-2021.12.17 ~]$

and

[liveuser@eos-2021.12.17 ~]$ sudo mount -t btrfs /dev/nvme0n1p2 /mnt -o subvol=@
mount: /mnt: mount(2) system call failed: File exists.
[liveuser@eos-2021.12.17 ~]$
11

Nope :rofl:

12

Reboot live media or try unmounting /mnt first.

13

I am on live media. The device is not mounted anywhere:

[liveuser@eos-2021.12.17 ~]$ sudo umount /mnt
umount: /mnt: not mounted.
[liveuser@eos-2021.12.17 ~]$ sudo mount -t btrfs /dev/nvme0n1p2 /mnt -o subvol=@
mount: /mnt: mount(2) system call failed: File exists.
[liveuser@eos-2021.12.17 ~]$
14

Have you checked fstab?

15

The live media /etc/fstab is pretty empty:

[liveuser@eos-2021.12.17 ~]$ cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
[liveuser@eos-2021.12.17 ~]$
16

That’s on the live ISO though? :thinking:

17

I would still reboot the live media just in case, then try the command again.

18

Yes, thats on the live media. I cannot check any other fstab as I cannot access my system, it doesn’t boot. Which fstab should I check?

19

I understand. Wasn’t thinking. :laughing:

Edit: I’m not that smart. :rofl:

1 Like
20

@DromundKaas
Have you tried the btrfs-zero-log command?

Edit: I think it’s changed?

btrfs rescue zero-log

Edit: I don’t want to cause you any data loss or issues because i am no expert. So I’m just asking questions about what you have tried.