Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Unfortunately my Fedora's btrfs partition failed again. Yet no idea of 
the culprit because memtest passes and the drive is good.
The system freezed and I had to reset (magic Sysrq keys didn't work 
either), being welcomed by an unmountable fs.

This is what I get when I try to mount it:

$ sudo mount /dev/nvme0n1p6 /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on 
/dev/nvme0n1p6, missing codepage or helper program, or other error.

[  375.964495] BTRFS info (device nvme0n1p6): disk space caching is 
enabled
[  375.964499] BTRFS info (device nvme0n1p6): has skinny extents
[  375.977169] BTRFS warning (device nvme0n1p6): checksum verify failed 
on 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
[  375.977179] BTRFS error (device nvme0n1p6): failed to read block 
groups: -5
[  375.978953] BTRFS error (device nvme0n1p6): open_ctree failed

Check fails to run:

$ sudo btrfs check /dev/nvme0n1p6
Opening filesystem to check...
checksum verify failed on 21348679680 wanted 0xd05bf9be found 0x2874489b
checksum verify failed on 21348679680 wanted 0xd05bf9be found 0x2874489b
Csum didn't match
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system

usebackuproot didn't help either:

$ sudo mount -o rescue=usebackuproot /dev/nvme0n1p6 /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on 
/dev/nvme0n1p6, missing codepage or helper program, or other error.

I tried btrfs rescue but it didn't lead to a mountable fs:

$ sudo btrfs rescue super-recover /dev/nvme0n1p6
All supers are valid, no need to recover

$ sudo btrfs rescue zero-log /dev/nvme0n1p6
Clearing log on /dev/nvme0n1p6, previous log_root 21344239616, level 0

$ sudo btrfs rescue chunk-recover /dev/nvme0n1p6
Scanning: DONE in dev0
Check chunks successfully with no orphans
Chunk tree recovered successfully

I did manage to recover some data with btrfs restore (no idea how much 
of it):

$ sudo btrfs restore /dev/nvme0n1p6 
/run/media/liveuser/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6_restore/
Skipping snapshot snapshot
[...lots of snapper snapshots]
Skipping snapshot root

I really did want to use rescue=skipbg 
(https://lwn.net/Articles/822242/) or rescue=onlyfs 
(https://lwn.net/ml/linux-btrfs/20200701144438.7613-1-josef@toxicpanda.com/) 
but it seems that neither managed to reach upstream :(

btrfs restore really sucks compared to the previous recovery options 
because it gives you no way to list your subvolumes or to recover a 
specific snapshot.

I've also looked at 
https://en.opensuse.org/SDB:BTRFS#How_to_repair_a_broken.2Funmountable_btrfs_filesystem 
to see if I had any other options left, but it seems I will have to 
reinstall from scratch.

We truly need a better way to recovery-mount partitions, along w/ better 
tools to at least *try* fixing them.

Niccolo'

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-12 06:27 Niccolò Belli ha scritto:
> I did manage to recover some data with btrfs restore (no idea how much 
> of it):
> 
> $ sudo btrfs restore /dev/nvme0n1p6
> /run/media/liveuser/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6_restore/
> Skipping snapshot snapshot
> [...lots of snapper snapshots]
> Skipping snapshot root

I just noticed it completely skipped to restore my root subvolume.
Maybe because it was originally restored from a snapshot (during a 
previous btrfs corruption issue) and so somehow it thinks it's still a 
snapshot?
How can I restore it? I cannot afford to restore all the snapshots, I 
don't have enough free space to do so.

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-12 06:44 Niccolò Belli ha scritto:
> Il 2021-09-12 06:27 Niccolò Belli ha scritto:
>> I did manage to recover some data with btrfs restore (no idea how much 
>> of it):
>> 
>> $ sudo btrfs restore /dev/nvme0n1p6
>> /run/media/liveuser/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6_restore/
>> Skipping snapshot snapshot
>> [...lots of snapper snapshots]
>> Skipping snapshot root
> 
> I just noticed it completely skipped to restore my root subvolume.
> Maybe because it was originally restored from a snapshot (during a
> previous btrfs corruption issue) and so somehow it thinks it's still a
> snapshot?
> How can I restore it? I cannot afford to restore all the snapshots, I
> don't have enough free space to do so.

Even better would be to restore a snapper snapshot of my root subvolume 
(that way it would be less likely to be corrupted).

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Qu Wenruo]

On 2021/9/12 下午6:27, Niccolò Belli wrote:
> Unfortunately my Fedora's btrfs partition failed again. Yet no idea of
> the culprit because memtest passes and the drive is good.
> The system freezed and I had to reset (magic Sysrq keys didn't work
> either), being welcomed by an unmountable fs.
>
> This is what I get when I try to mount it:
>
> $ sudo mount /dev/nvme0n1p6 /mnt/
> mount: /mnt: wrong fs type, bad option, bad superblock on
> /dev/nvme0n1p6, missing codepage or helper program, or other error.
>
> [  375.964495] BTRFS info (device nvme0n1p6): disk space caching is enabled
> [  375.964499] BTRFS info (device nvme0n1p6): has skinny extents
> [  375.977169] BTRFS warning (device nvme0n1p6): checksum verify failed
> on 21348679680 wanted 0xd05bf9be found 0x2874489b level 1

This is very strange.

Not transid error, not tree-checker, but purely csum mismatch.

Furthermore, it's csum mismatch means, its bytenr and fsid is correct,
just csum mismatch.

It's not something wrong in the content (in runtime memory), but really
just csum mismatch.

This really means, something in the tree block content is not correct,
thus not matching the inlined csum. Exactly the thing csum is designed
to detect.

Normally for metadata, we use DUP thus you have your chance to recovery
from such real data corruption, but unfortunately for NVME SSD, we
default to SINGLE, without any extra copies.

> [  375.977179] BTRFS error (device nvme0n1p6): failed to read block
> groups: -5

This is again in extent tree.

> [  375.978953] BTRFS error (device nvme0n1p6): open_ctree failed
>
> Check fails to run:
>
> $ sudo btrfs check /dev/nvme0n1p6
> Opening filesystem to check...
> checksum verify failed on 21348679680 wanted 0xd05bf9be found 0x2874489b
> checksum verify failed on 21348679680 wanted 0xd05bf9be found 0x2874489b
> Csum didn't match
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
> usebackuproot didn't help either:
>
> $ sudo mount -o rescue=usebackuproot /dev/nvme0n1p6 /mnt/
> mount: /mnt: wrong fs type, bad option, bad superblock on
> /dev/nvme0n1p6, missing codepage or helper program, or other error.
>
> I tried btrfs rescue but it didn't lead to a mountable fs:
>
> $ sudo btrfs rescue super-recover /dev/nvme0n1p6
> All supers are valid, no need to recover
>
> $ sudo btrfs rescue zero-log /dev/nvme0n1p6
> Clearing log on /dev/nvme0n1p6, previous log_root 21344239616, level 0

None of these really helps.
>
> $ sudo btrfs rescue chunk-recover /dev/nvme0n1p6
> Scanning: DONE in dev0
> Check chunks successfully with no orphans
> Chunk tree recovered successfully

This can be even a little dangerous.

>
> I did manage to recover some data with btrfs restore (no idea how much
> of it):
>
> $ sudo btrfs restore /dev/nvme0n1p6
> /run/media/liveuser/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6_restore/
> Skipping snapshot snapshot
> [...lots of snapper snapshots]
> Skipping snapshot root
>
> I really did want to use rescue=skipbg
> (https://lwn.net/Articles/822242/) or rescue=onlyfs
> (https://lwn.net/ml/linux-btrfs/20200701144438.7613-1-josef@toxicpanda.com/)
> but it seems that neither managed to reach upstream :(

No, it's already in the upstream, in v5.11.

Just a different name, "rescue=ibadroots" or "rescue=ignorebadroots".
And it get enhanced to handle the exact case you're hitting, in v5.14.


So please try first using "rescue=ibadroots" (need to be mounted with
RO), then check your fs.

The core thing is to ensure everything, include csum tree (by reading
all your data), all your files/directories are fine.

Then you can try to rebuild extent tree.
But for now I prefer to do the rescue mount first, then consider the
next step.


Really, this looks like a data corruption not detected by SSD controller
but detected by btrfs.

Thanks,
Qu

>
> btrfs restore really sucks compared to the previous recovery options
> because it gives you no way to list your subvolumes or to recover a
> specific snapshot.
>
> I've also looked at
> https://en.opensuse.org/SDB:BTRFS#How_to_repair_a_broken.2Funmountable_btrfs_filesystem
> to see if I had any other options left, but it seems I will have to
> reinstall from scratch.
>
> We truly need a better way to recovery-mount partitions, along w/ better
> tools to at least *try* fixing them.
>
> Niccolo'

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-12 07:14 Qu Wenruo ha scritto:
> No, it's already in the upstream, in v5.11.
> 
> Just a different name, "rescue=ibadroots" or "rescue=ignorebadroots".
> And it get enhanced to handle the exact case you're hitting, in v5.14.
> 
> 
> So please try first using "rescue=ibadroots" (need to be mounted with
> RO), then check your fs.

$ sudo mount -o ro,rescue=ibadroots /dev/nvme0n1p6 /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on 
/dev/nvme0n1p6, missing codepage or helper program, or other error.

$ btrfs --version
btrfs-progs v5.14

$ uname -a
Linux localhost-live 5.14.0-60.fc35.x86_64 #1 SMP Mon Aug 30 16:45:32 
UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[10496.832659] BTRFS info (device nvme0n1p6): ignoring bad roots
[10496.832663] BTRFS info (device nvme0n1p6): disk space caching is 
enabled
[10496.832664] BTRFS info (device nvme0n1p6): has skinny extents
[10496.845607] BTRFS warning (device nvme0n1p6): checksum verify failed 
on 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
[10496.845616] BTRFS error (device nvme0n1p6): failed to read block 
groups: -5
[10496.846085] BTRFS error (device nvme0n1p6): open_ctree failed

Before doing so I restored a previous dd copy of the partition since you 
said that chunk-recover could have been somewhat dangerous.

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Qu Wenruo]

On 2021/9/12 下午7:41, Niccolò Belli wrote:
> Il 2021-09-12 07:14 Qu Wenruo ha scritto:
>> No, it's already in the upstream, in v5.11.
>>
>> Just a different name, "rescue=ibadroots" or "rescue=ignorebadroots".
>> And it get enhanced to handle the exact case you're hitting, in v5.14.
>>
>>
>> So please try first using "rescue=ibadroots" (need to be mounted with
>> RO), then check your fs.
>
> $ sudo mount -o ro,rescue=ibadroots /dev/nvme0n1p6 /mnt/
> mount: /mnt: wrong fs type, bad option, bad superblock on
> /dev/nvme0n1p6, missing codepage or helper program, or other error.
>
> $ btrfs --version
> btrfs-progs v5.14
>
> $ uname -a
> Linux localhost-live 5.14.0-60.fc35.x86_64 #1 SMP Mon Aug 30 16:45:32
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

My bad, the commit is 2b29726c473b ("btrfs: rescue: allow ibadroots to
skip bad extent tree when reading block group items"), which is only
merged into the incoming v5.15-rc1.
>
> [10496.832659] BTRFS info (device nvme0n1p6): ignoring bad roots
> [10496.832663] BTRFS info (device nvme0n1p6): disk space caching is enabled
> [10496.832664] BTRFS info (device nvme0n1p6): has skinny extents
> [10496.845607] BTRFS warning (device nvme0n1p6): checksum verify failed
> on 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
> [10496.845616] BTRFS error (device nvme0n1p6): failed to read block
> groups: -5
> [10496.846085] BTRFS error (device nvme0n1p6): open_ctree failed
>
> Before doing so I restored a previous dd copy of the partition since you
> said that chunk-recover could have been somewhat dangerous.

If you have dd copy, then things can be much easier to handle.

The easiest way to make ibadroots to work in such situation is, to
manually destroy extent tree root manually.

# btrfs ins dump-tree -t root <dev> | \
   grep "(EXTENT_TREE ROOT_ITEM 0)" -A3
	item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
		generation 21 root_dirid 0 bytenr 1359085568 level 0 refs 1
		lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
		uuid 00000000-0000-0000-0000-000000000000

Grab the bytenr ("1359085568" in my example).

Then use the btrfs-map-logical from the following branch of btrfs-progs:
https://github.com/adam900710/btrfs-progs/tree/map_logical_rework

(The current btrfs-map-logical can't handle extent tree well)

$ btrfs-map-logical -l 1359085568 <dev>
mirror 1 logical 1359085568 physical 1367474176 device /dev/test/test
mirror 2 logical 1359085568 physical 1635909632 device /dev/test/test

Then corrupt the first 4 bytes of each copy (in your case, there should
be only one copy) using its physical bytenr"

$ xfs_io -f -c "pwrite 1367474176 4" <dev>

Then rescue=ibadroots should work as expected.

Thanks,
Qu

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-12 15:35 Qu Wenruo ha scritto:
> My bad, the commit is 2b29726c473b ("btrfs: rescue: allow ibadroots to
> skip bad extent tree when reading block group items"), which is only
> merged into the incoming v5.15-rc1.

I've compiled linux-git master and tried again:

$ uname -a
Linux arch-laptop 5.14.0-1-git-11152-g78e709522d2c #1 SMP PREEMPT Sun, 
12 Sep 2021 12:53:13 +0000 x86_64 GNU/Linux

$ sudo mount -o ro,rescue=ibadroots 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6.img.copy 
/mnt2/
mount: /mnt2: can't read superblock on /dev/loop0.

[  196.277414] loop: module loaded
[  196.278228] loop0: detected capacity change from 0 to 207618048
[  196.736456] BTRFS: device label fedora_localhost-live devid 1 transid 
1029644 /dev/loop0 scanned by mount (2730)
[  196.736819] BTRFS info (device loop0): flagging fs with big metadata 
feature
[  196.736825] BTRFS info (device loop0): ignoring bad roots
[  196.736826] BTRFS info (device loop0): disk space caching is enabled
[  196.736827] BTRFS info (device loop0): has skinny extents
[  197.408704] BTRFS warning (device loop0): checksum verify failed on 
21348679680 wanted 0xd05bf9be found 0x2874489b level 1
[  197.408843] BTRFS info (device loop0): start tree-log replay
[  197.408847] BTRFS warning (device loop0): log replay required on RO 
media
[  197.409458] BTRFS error (device loop0): open_ctree failed

So I've added nologreplay and it did the trick:

$ sudo mount -o ro,rescue=nologreplay:ibadroots 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6.img.copy 
/mnt2/

[  416.913016] loop0: detected capacity change from 0 to 207618048
[  416.918895] BTRFS info (device loop0): flagging fs with big metadata 
feature
[  416.918903] BTRFS info (device loop0): disabling log replay at mount 
time
[  416.918905] BTRFS info (device loop0): ignoring bad roots
[  416.918907] BTRFS info (device loop0): disk space caching is enabled
[  416.918908] BTRFS info (device loop0): has skinny extents
[  416.929887] BTRFS warning (device loop0): checksum verify failed on 
21348679680 wanted 0xd05bf9be found 0x2874489b level 1

$ sudo btrfs subvolume list /mnt2 | grep -v snapshot | grep -v docker
ID 256 gen 1029644 top level 5 path home
ID 265 gen 1029641 top level 256 path home/niko/.cache
ID 912 gen 1029406 top level 5 path images
ID 2428 gen 359129 top level 5 path var
ID 2429 gen 1026054 top level 2428 path var/tmp
ID 2430 gen 1029044 top level 2428 path var/cache
ID 2433 gen 1029641 top level 5 path root
ID 2653 gen 1029641 top level 5 path avd

$ ls -al /mnt2
totale 16
drwxr-xr-x 1 root root  58 16 giu 19.09 .
drwxr-xr-x 1 root root 198 12 set 17.50 ..
drwxr-xr-x 1 niko niko  72 26 ago 10.22 avd
drwxr-xr-x 1 root root  28 16 apr 10.25 home
drwxrwxrwx 1 niko niko  96 20 ago 17.23 images
dr-xr-xr-x 1 root root 196 24 ago 14.58 root
drwxr-xr-x 1 root root  28 16 apr 10.21 snapshots
drwxr-xr-x 1 root root  16 13 giu 00.10 var

Apart from backing my files up, what else should I be able to do at this 
point?

I've tried scrubbing but it's a no go:

$ sudo btrfs scrub start /dev/loop0
scrub started on /dev/loop0, fsid d3d387d6-eb5e-4b4b-9a9c-581d74fb56b4 
(pid=3255)

$ sudo btrfs scrub status /dev/loop0
UUID:             d3d387d6-eb5e-4b4b-9a9c-581d74fb56b4
Scrub started:    Sun Sep 12 18:00:30 2021
Status:           aborted
Duration:         0:00:00
Total to scrub:   97.27GiB
Rate:             0.00B/s
Error summary:    no errors found

Any chance to recover the partition?

Niccolò

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

I think I've managed to recover the vast majority of my files:

$ sudo cp -a avd var 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/
cp: avd/Pixel_4_API_30.avd/userdata-qemu.img.qcow2: recupero delle 
informazioni degli extent non riuscito: Errore di input/output

$ sudo cp -a snapshots/home/375/snapshot 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/home
cp: 
snapshots/home/375/snapshot/niko/devel/beach/client/android/.gradle/checksums/sha1-checksums.bin: 
recupero delle informazioni degli extent non riuscito: Errore di 
input/output

$ sudo cp -a snapshots/root/3004/snapshot 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/root

$ sudo cp -a snapshots/images/3/snapshot 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/images
cp: snapshots/images/3/snapshot/OSX-KVM/mac_hdd_ng.img: recupero delle 
informazioni degli extent non riuscito: Errore di input/output
cp: snapshots/images/3/snapshot/OSX-KVM/BaseSystem.img: recupero delle 
informazioni degli extent non riuscito: Errore di input/output

The only valuable thing I've lost seems to be an hackintosh vm I use for 
development, but I can create another one.

[ 2213.532522] BTRFS warning (device loop0): Skipping commit of aborted 
transaction.
[ 2213.532526] ------------[ cut here ]------------
[ 2213.532527] BTRFS: Transaction aborted (error -28)
[ 2213.532565] WARNING: CPU: 2 PID: 3061 at fs/btrfs/transaction.c:1946 
btrfs_commit_transaction.cold+0xf2/0x2ef [btrfs]
[ 2213.532609] Modules linked in: loop ext4 mbcache jbd2 ses enclosure 
scsi_transport_sas uas usb_storage cmac ccm xt_conntrack xt_MASQUERADE 
nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat 
nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter overlay 
bridge stp llc snd_hda_codec_hdmi bnep uvcvideo videobuf2_vmalloc 
videobuf2_memops videobuf2_v4l2 videobuf2_common btusb videodev btrtl 
btbcm btintel mc bluetooth ecdh_generic ecc crc16 joydev 
intel_spi_platform intel_spi mousedev spi_nor iTCO_wdt mei_wdt 
intel_pmc_bxt mtd mei_hdcp iTCO_vendor_support intel_rapl_msr wmi_bmof 
dell_laptop i915 dell_wmi dell_smbios dell_wmi_descriptor iwlmvm 
x86_pkg_temp_thermal intel_powerclamp coretemp sparse_keymap snd_ctl_led 
dcdbas snd_hda_codec_realtek i2c_algo_bit snd_hda_codec_generic ttm 
ledtrig_audio mac80211 kvm_intel snd_hda_intel libarc4 snd_intel_dspcfg 
dell_smm_hwmon kvm irqbypass rapl intel_cstate iwlwifi 
snd_intel_sdw_acpi drm_kms_helper intel_uncore snd_hda_codec
[ 2213.532656]  pcspkr psmouse processor_thermal_device_pci_legacy 
i2c_i801 cec processor_thermal_device snd_hda_core 
processor_thermal_rfim cfg80211 i2c_smbus processor_thermal_mbox 
intel_gtt intel_pch_thermal agpgart snd_hwdep processor_thermal_rapl 
syscopyarea lpc_ich rfkill snd_pcm sysfillrect intel_rapl_common mei_me 
snd_timer sysimgblt snd soundcore fb_sys_fops intel_soc_dts_iosf mei wmi 
int3403_thermal i2c_hid_acpi video dw_dmac i2c_hid int3402_thermal 
int3400_thermal int340x_thermal_zone acpi_thermal_rel acpi_pad mac_hid 
nls_iso8859_1 vfat fat crypto_user drm fuse ip_tables x_tables btrfs 
blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc 
encrypted_keys trusted asn1_encoder tee hid_multitouch usbhid dm_mod 
serio_raw rtsx_pci_sdmmc mmc_core atkbd libps2 crct10dif_pclmul 
crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd 
cryptd rtsx_pci xhci_pci xhci_pci_renesas tpm_crb i8042 serio tpm_tis 
tpm_tis_core tpm rng_core pl2303 mos7720 parport
[ 2213.532705] CPU: 2 PID: 3061 Comm: btrfs-transacti Not tainted 
5.14.0-1-git-11152-g78e709522d2c #1 
4337bfa7fd23c500813a836e2184863944fac299
[ 2213.532708] Hardware name: Dell Inc. XPS 13 9343/0X2GW3, BIOS A20 
06/06/2019
[ 2213.532709] RIP: 0010:btrfs_commit_transaction.cold+0xf2/0x2ef 
[btrfs]
[ 2213.532746] Code: 28 48 89 04 24 48 89 c2 49 8b 44 24 28 48 39 c2 75 
30 0f 0b 0f 0b 48 8b 45 50 eb a1 89 de 48 c7 c7 c8 e7 6e c0 e8 2f 4b 3a 
d9 <0f> 0b eb a9 48 8b 7d 50 89 da 48 c7 c6 f8 e7 6e c0 e8 62 ba ff ff
[ 2213.532748] RSP: 0018:ffffaa6401eebde0 EFLAGS: 00010282
[ 2213.532750] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: 
0000000000000027
[ 2213.532751] RDX: ffff9be716518728 RSI: 0000000000000001 RDI: 
ffff9be716518720
[ 2213.532752] RBP: ffff9be577d1c8f0 R08: 0000000000000000 R09: 
ffffaa6401eebc10
[ 2213.532754] R10: ffffaa6401eebc08 R11: ffffffff9aaccca8 R12: 
ffff9be55293b200
[ 2213.532755] R13: ffff9be6d303b000 R14: ffff9be577d1c948 R15: 
ffffaa6401eebe78
[ 2213.532756] FS:  0000000000000000(0000) GS:ffff9be716500000(0000) 
knlGS:0000000000000000
[ 2213.532758] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2213.532759] CR2: 000055d4b013fbd0 CR3: 0000000065a10005 CR4: 
00000000003706e0
[ 2213.532760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[ 2213.532761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[ 2213.532763] Call Trace:
[ 2213.532767]  transaction_kthread+0x12e/0x1a0 [btrfs 
1ccea181fc519646bf0a24aa8f854515e5e21f83]
[ 2213.532799]  ? btrfs_cleanup_transaction.isra.0+0x590/0x590 [btrfs 
1ccea181fc519646bf0a24aa8f854515e5e21f83]
[ 2213.532830]  kthread+0x132/0x160
[ 2213.532834]  ? set_kthread_struct+0x40/0x40
[ 2213.532836]  ret_from_fork+0x22/0x30
[ 2213.532841] ---[ end trace a9ee4fb88980a146 ]---
[ 2213.532843] BTRFS: error (device loop0) in cleanup_transaction:1946: 
errno=-28 No space left
[ 2224.518838] BTRFS warning (device loop0): checksum verify failed on 
21348679680 wanted 0xd05bf9be found 0x2874489b level 1
[ 2227.041189] BTRFS warning (device loop0): checksum verify failed on 
21348679680 wanted 0xd05bf9be found 0x2874489b level 1

This error is weird instead, considering it's mounted ro it shouldn't 
complain about no space being left.

By the way I didn't get any kind of I/O error while restoring avd, home 
or images with btrfs restore: what's the difference with ibadroots?

Niccolò

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Qu Wenruo]

On 2021/9/13 上午5:23, Niccolò Belli wrote:
> I think I've managed to recover the vast majority of my files:
> 
> $ sudo cp -a avd var 
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/
> cp: avd/Pixel_4_API_30.avd/userdata-qemu.img.qcow2: recupero delle 
> informazioni degli extent non riuscito: Errore di input/output
> 
> $ sudo cp -a snapshots/home/375/snapshot 
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/home
> cp: 
> snapshots/home/375/snapshot/niko/devel/beach/client/android/.gradle/checksums/sha1-checksums.bin: 
> recupero delle informazioni degli extent non riuscito: Errore di 
> input/output
> 
> $ sudo cp -a snapshots/root/3004/snapshot 
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/root
> 
> $ sudo cp -a snapshots/images/3/snapshot 
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/images
> cp: snapshots/images/3/snapshot/OSX-KVM/mac_hdd_ng.img: recupero delle 
> informazioni degli extent non riuscito: Errore di input/output
> cp: snapshots/images/3/snapshot/OSX-KVM/BaseSystem.img: recupero delle 
> informazioni degli extent non riuscito: Errore di input/output

During the backup, have you checked the dmesg?

I just want to make sure that, the errors are all csum mismatch, not 
something more series like csum tree corruption.

> 
> The only valuable thing I've lost seems to be an hackintosh vm I use for 
> development, but I can create another one.
> 
> [ 2213.532522] BTRFS warning (device loop0): Skipping commit of aborted 
> transaction.
> [ 2213.532526] ------------[ cut here ]------------
> [ 2213.532527] BTRFS: Transaction aborted (error -28)
> [ 2213.532565] WARNING: CPU: 2 PID: 3061 at fs/btrfs/transaction.c:1946 
> btrfs_commit_transaction.cold+0xf2/0x2ef [btrfs]
> [ 2213.532609] Modules linked in: loop ext4 mbcache jbd2 ses enclosure 
> scsi_transport_sas uas usb_storage cmac ccm xt_conntrack xt_MASQUERADE 
> nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat 
> nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter overlay 
> bridge stp llc snd_hda_codec_hdmi bnep uvcvideo videobuf2_vmalloc 
> videobuf2_memops videobuf2_v4l2 videobuf2_common btusb videodev btrtl 
> btbcm btintel mc bluetooth ecdh_generic ecc crc16 joydev 
> intel_spi_platform intel_spi mousedev spi_nor iTCO_wdt mei_wdt 
> intel_pmc_bxt mtd mei_hdcp iTCO_vendor_support intel_rapl_msr wmi_bmof 
> dell_laptop i915 dell_wmi dell_smbios dell_wmi_descriptor iwlmvm 
> x86_pkg_temp_thermal intel_powerclamp coretemp sparse_keymap snd_ctl_led 
> dcdbas snd_hda_codec_realtek i2c_algo_bit snd_hda_codec_generic ttm 
> ledtrig_audio mac80211 kvm_intel snd_hda_intel libarc4 snd_intel_dspcfg 
> dell_smm_hwmon kvm irqbypass rapl intel_cstate iwlwifi 
> snd_intel_sdw_acpi drm_kms_helper intel_uncore snd_hda_codec
> [ 2213.532656]  pcspkr psmouse processor_thermal_device_pci_legacy 
> i2c_i801 cec processor_thermal_device snd_hda_core 
> processor_thermal_rfim cfg80211 i2c_smbus processor_thermal_mbox 
> intel_gtt intel_pch_thermal agpgart snd_hwdep processor_thermal_rapl 
> syscopyarea lpc_ich rfkill snd_pcm sysfillrect intel_rapl_common mei_me 
> snd_timer sysimgblt snd soundcore fb_sys_fops intel_soc_dts_iosf mei wmi 
> int3403_thermal i2c_hid_acpi video dw_dmac i2c_hid int3402_thermal 
> int3400_thermal int340x_thermal_zone acpi_thermal_rel acpi_pad mac_hid 
> nls_iso8859_1 vfat fat crypto_user drm fuse ip_tables x_tables btrfs 
> blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc 
> encrypted_keys trusted asn1_encoder tee hid_multitouch usbhid dm_mod 
> serio_raw rtsx_pci_sdmmc mmc_core atkbd libps2 crct10dif_pclmul 
> crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd 
> cryptd rtsx_pci xhci_pci xhci_pci_renesas tpm_crb i8042 serio tpm_tis 
> tpm_tis_core tpm rng_core pl2303 mos7720 parport
> [ 2213.532705] CPU: 2 PID: 3061 Comm: btrfs-transacti Not tainted 
> 5.14.0-1-git-11152-g78e709522d2c #1 
> 4337bfa7fd23c500813a836e2184863944fac299
> [ 2213.532708] Hardware name: Dell Inc. XPS 13 9343/0X2GW3, BIOS A20 
> 06/06/2019
> [ 2213.532709] RIP: 0010:btrfs_commit_transaction.cold+0xf2/0x2ef [btrfs]
> [ 2213.532746] Code: 28 48 89 04 24 48 89 c2 49 8b 44 24 28 48 39 c2 75 
> 30 0f 0b 0f 0b 48 8b 45 50 eb a1 89 de 48 c7 c7 c8 e7 6e c0 e8 2f 4b 3a 
> d9 <0f> 0b eb a9 48 8b 7d 50 89 da 48 c7 c6 f8 e7 6e c0 e8 62 ba ff ff
> [ 2213.532748] RSP: 0018:ffffaa6401eebde0 EFLAGS: 00010282
> [ 2213.532750] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: 
> 0000000000000027
> [ 2213.532751] RDX: ffff9be716518728 RSI: 0000000000000001 RDI: 
> ffff9be716518720
> [ 2213.532752] RBP: ffff9be577d1c8f0 R08: 0000000000000000 R09: 
> ffffaa6401eebc10
> [ 2213.532754] R10: ffffaa6401eebc08 R11: ffffffff9aaccca8 R12: 
> ffff9be55293b200
> [ 2213.532755] R13: ffff9be6d303b000 R14: ffff9be577d1c948 R15: 
> ffffaa6401eebe78
> [ 2213.532756] FS:  0000000000000000(0000) GS:ffff9be716500000(0000) 
> knlGS:0000000000000000
> [ 2213.532758] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2213.532759] CR2: 000055d4b013fbd0 CR3: 0000000065a10005 CR4: 
> 00000000003706e0
> [ 2213.532760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [ 2213.532761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> [ 2213.532763] Call Trace:
> [ 2213.532767]  transaction_kthread+0x12e/0x1a0 [btrfs 
> 1ccea181fc519646bf0a24aa8f854515e5e21f83]
> [ 2213.532799]  ? btrfs_cleanup_transaction.isra.0+0x590/0x590 [btrfs 
> 1ccea181fc519646bf0a24aa8f854515e5e21f83]
> [ 2213.532830]  kthread+0x132/0x160
> [ 2213.532834]  ? set_kthread_struct+0x40/0x40
> [ 2213.532836]  ret_from_fork+0x22/0x30
> [ 2213.532841] ---[ end trace a9ee4fb88980a146 ]---
> [ 2213.532843] BTRFS: error (device loop0) in cleanup_transaction:1946: 
> errno=-28 No space left
> [ 2224.518838] BTRFS warning (device loop0): checksum verify failed on 
> 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
> [ 2227.041189] BTRFS warning (device loop0): checksum verify failed on 
> 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
> 
> This error is weird instead, considering it's mounted ro it shouldn't 
> complain about no space being left.

The ENOSPC is caused by the fact that we fill dummy block groups by 
using up all its bytes.

But it's still a problem that why the kernel even tries to commit 
transaction.

> 
> By the way I didn't get any kind of I/O error while restoring avd, home 
> or images with btrfs restore: what's the difference with ibadroots?

For btrfs-restore, it just ignore all csum errors.
While for ibadroots, it still does csum check (with all the other sanity 
checks in kernel)

Thanks,
Qu

> 
> Niccolò
> 

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-12 19:55 Qu Wenruo ha scritto:
> During the backup, have you checked the dmesg?

Yes, the only error was the "No space left" I mentioned in the previous 
message.

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Qu Wenruo]

On 2021/9/13 下午3:16, Niccolò Belli wrote:
> Il 2021-09-12 19:55 Qu Wenruo ha scritto:
>> During the backup, have you checked the dmesg?
> 
> Yes, the only error was the "No space left" I mentioned in the previous 
> message.
> 

Then it means no csum error.

But didn't you hit -EIO errors?

 > $ sudo cp -a avd var 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/
 > cp: avd/Pixel_4_API_30.avd/userdata-qemu.img.qcow2: recupero delle 
informazioni degli extent non riuscito: Errore di input/output

And

 > $ sudo cp -a snapshots/home/375/snapshot 
/run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6/home
cp: 
snapshots/home/375/snapshot/niko/devel/beach/client/android/.gradle/checksums/sha1-checksums.bin: 
recupero delle informazioni degli extent non riuscito: Errore di 
input/output

For those errors, not even a single dmesg error?


Anyway, if you have most data recovered and have dd copy at hand, then 
you may want to try fully recovery the fs to RW state, by some 
aggressive method:

# btrfs check --init-extent-tree --repair <dev>

This command will try to rebuild the extent tree completely, it uses 
existing trees to rebuild, thus requires the fs has nothing else but 
only extent tree corrupted.

This is *dangerous*, so the final call is still on you.
But if you really hit no other error messages during the full fs 
recovery, then it looks like only extent tree is corrupted, and may 
worthy a try.

BTW, since you're already using rescue=nologreplay, you may want to zero 
the log before rebuilding the extent tree.

Thanks,
Qu

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-13 04:05 Qu Wenruo ha scritto:
> For those errors, not even a single dmesg error?

Nope.

> Anyway, if you have most data recovered and have dd copy at hand, then
> you may want to try fully recovery the fs to RW state, by some
> aggressive method:
> 
> # btrfs check --init-extent-tree --repair <dev>
> 
> This command will try to rebuild the extent tree completely, it uses
> existing trees to rebuild, thus requires the fs has nothing else but
> only extent tree corrupted.
> 
> This is *dangerous*, so the final call is still on you.
> But if you really hit no other error messages during the full fs
> recovery, then it looks like only extent tree is corrupted, and may
> worthy a try.
> 
> BTW, since you're already using rescue=nologreplay, you may want to
> zero the log before rebuilding the extent tree.

3 and a half hours later and it's still building the extent tree.
I gave up and made a new fs from scratch.

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Zygo Blaxell]

On Sun, Sep 12, 2021 at 05:51:25PM +0200, Niccolò Belli wrote:
> Il 2021-09-12 15:35 Qu Wenruo ha scritto:
> > My bad, the commit is 2b29726c473b ("btrfs: rescue: allow ibadroots to
> > skip bad extent tree when reading block group items"), which is only
> > merged into the incoming v5.15-rc1.
> 
> I've compiled linux-git master and tried again:
> 
> $ uname -a
> Linux arch-laptop 5.14.0-1-git-11152-g78e709522d2c #1 SMP PREEMPT Sun, 12
> Sep 2021 12:53:13 +0000 x86_64 GNU/Linux
> 
> $ sudo mount -o ro,rescue=ibadroots
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6.img.copy
> /mnt2/
> mount: /mnt2: can't read superblock on /dev/loop0.
> 
> [  196.277414] loop: module loaded
> [  196.278228] loop0: detected capacity change from 0 to 207618048
> [  196.736456] BTRFS: device label fedora_localhost-live devid 1 transid
> 1029644 /dev/loop0 scanned by mount (2730)
> [  196.736819] BTRFS info (device loop0): flagging fs with big metadata
> feature
> [  196.736825] BTRFS info (device loop0): ignoring bad roots
> [  196.736826] BTRFS info (device loop0): disk space caching is enabled
> [  196.736827] BTRFS info (device loop0): has skinny extents
> [  197.408704] BTRFS warning (device loop0): checksum verify failed on
> 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
> [  197.408843] BTRFS info (device loop0): start tree-log replay
> [  197.408847] BTRFS warning (device loop0): log replay required on RO media
> [  197.409458] BTRFS error (device loop0): open_ctree failed
> 
> So I've added nologreplay and it did the trick:
> 
> $ sudo mount -o ro,rescue=nologreplay:ibadroots
> /run/media/niko/3ea0705c-21c9-4ba9-80ee-5a511cb2a093/nvme0n1p6.img.copy
> /mnt2/
> 
> [  416.913016] loop0: detected capacity change from 0 to 207618048
> [  416.918895] BTRFS info (device loop0): flagging fs with big metadata
> feature
> [  416.918903] BTRFS info (device loop0): disabling log replay at mount time
> [  416.918905] BTRFS info (device loop0): ignoring bad roots
> [  416.918907] BTRFS info (device loop0): disk space caching is enabled
> [  416.918908] BTRFS info (device loop0): has skinny extents
> [  416.929887] BTRFS warning (device loop0): checksum verify failed on
> 21348679680 wanted 0xd05bf9be found 0x2874489b level 1
> 
> $ sudo btrfs subvolume list /mnt2 | grep -v snapshot | grep -v docker
> ID 256 gen 1029644 top level 5 path home
> ID 265 gen 1029641 top level 256 path home/niko/.cache
> ID 912 gen 1029406 top level 5 path images
> ID 2428 gen 359129 top level 5 path var
> ID 2429 gen 1026054 top level 2428 path var/tmp
> ID 2430 gen 1029044 top level 2428 path var/cache
> ID 2433 gen 1029641 top level 5 path root
> ID 2653 gen 1029641 top level 5 path avd
> 
> $ ls -al /mnt2
> totale 16
> drwxr-xr-x 1 root root  58 16 giu 19.09 .
> drwxr-xr-x 1 root root 198 12 set 17.50 ..
> drwxr-xr-x 1 niko niko  72 26 ago 10.22 avd
> drwxr-xr-x 1 root root  28 16 apr 10.25 home
> drwxrwxrwx 1 niko niko  96 20 ago 17.23 images
> dr-xr-xr-x 1 root root 196 24 ago 14.58 root
> drwxr-xr-x 1 root root  28 16 apr 10.21 snapshots
> drwxr-xr-x 1 root root  16 13 giu 00.10 var
> 
> Apart from backing my files up, what else should I be able to do at this
> point?

I have questions and a suggestion:

1.  What is the device model and firmware revision of the SSD?  e.g.

	smartctl -i /dev/nvme0n1 | grep -v 'Serial Number'

2.  What kernel was running at the time of the failure?  Was it 5.14,
or an earlier kernel?

Try dup metadata ('btrfs balance start -mconvert=dup,soft /' if you
already created the fs).  If you have dup metadata and the filesystem
starts self-repairing metadata csum errors, we'll know it's a failed
drive.  If both copies of metadata are corrupted identically, the problem
is probably in some layer above the drive (or the drive does deduplication
internally, but that is why we want to know the model number).

> I've tried scrubbing but it's a no go:
> 
> $ sudo btrfs scrub start /dev/loop0
> scrub started on /dev/loop0, fsid d3d387d6-eb5e-4b4b-9a9c-581d74fb56b4
> (pid=3255)
> 
> $ sudo btrfs scrub status /dev/loop0
> UUID:             d3d387d6-eb5e-4b4b-9a9c-581d74fb56b4
> Scrub started:    Sun Sep 12 18:00:30 2021
> Status:           aborted
> Duration:         0:00:00
> Total to scrub:   97.27GiB
> Rate:             0.00B/s
> Error summary:    no errors found
> 
> Any chance to recover the partition?
> 
> Niccolò

Re: Unmountable / uncheckable Fedora 34 btrfs: failed to read block groups: -5 open_ctree failed

[Niccolò Belli]

Il 2021-09-13 16:50 Zygo Blaxell ha scritto:
> 1.  What is the device model and firmware revision of the SSD?  e.g.
> 
> 	smartctl -i /dev/nvme0n1 | grep -v 'Serial Number'

$ sudo smartctl -i /dev/nvme0n1 | grep -v 'Serial Number'
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.15-200.fc34.x86_64] 
(local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, 
www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 970 EVO 250GB
Firmware Version:                   2B2QEXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 250,059,350,016 [250 GB]
Unallocated NVM Capacity:           0
Controller ID:                      4
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          250,059,350,016 [250 GB]
Namespace 1 Utilization:            171,265,327,104 [171 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5781b1927e
Local Time is:                      Mon Sep 13 22:51:00 2021 CEST

> 2.  What kernel was running at the time of the failure?  Was it 5.14,
> or an earlier kernel?

It was 5.13.14 or 5.13.15.