volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1

volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1

[berodual_xyz]

Dear list,

following from earlier this weeks case of attempting data rescue on a corrupt FS, we have now cloned all devices and can do potentially dangerous rescue attempts (since we can re-clone the original disks).

On kernel 4.20.17 and btrfs-progs 4.20.2:


##
$ btrfs inspect-internal tree-stats /dev/sdh
parent transid verify failed on 1048576 wanted 60234 found 60230
parent transid verify failed on 1048576 wanted 60234 found 60230
Ignoring transid failure
volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
btrfs[0x426fdc]
btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
btrfs(btrfs_setup_all_roots+0x368)[0x416540]
btrfs[0x416a8a]
btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
btrfs(open_ctree+0x75)[0x416c61]
btrfs(cmd_inspect_tree_stats+0xfa)[0x4706ad]
btrfs(handle_command_group+0x5d)[0x40c7b3]
btrfs(cmd_inspect+0x15)[0x4499f1]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0f26c7cb35]
btrfs[0x40c509]
[1]    15273 abort      btrfs inspect-internal tree-stats /dev/sdh


###


$ btrfs rescue zero-log /dev/sdh
parent transid verify failed on 1048576 wanted 60234 found 60230
parent transid verify failed on 1048576 wanted 60234 found 60230
Ignoring transid failure
volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
btrfs[0x426fdc]
btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
btrfs(btrfs_setup_all_roots+0x368)[0x416540]
btrfs[0x416a8a]
btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
btrfs(open_ctree+0x75)[0x416c61]
btrfs[0x4657ff]
btrfs(handle_command_group+0x5d)[0x40c7b3]
btrfs(cmd_rescue+0x15)[0x465b98]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4507ab1b35]
btrfs[0x40c509]
[1]    30211 abort      btrfs rescue zero-log /dev/sdh

###

Even if we will not be able to recover the data, adding these traces here hopefully helps enhancing the tools in the future as a crash is most definitely not the "expected behavior".


Have a nice weekend and I hope to receive some feedback.

Thanks in advance,

Marcel

Re: volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1

[berodual_xyz]

Adding to last email, being more brave to test out commands now:

Different crash than on last report.

Command:

# btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh

#########

$ btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh
WARNING: low-memory mode repair support is only partial
Opening filesystem to check...
Checking filesystem on /dev/sdh
UUID: 8b19ff46-3f42-4f51-be6b-5fc8a7d8f2cd
Creating a new extent tree
Failed to find [6815744, 168, 16384]
btrfs unable to find ref byte nr 55431977238528 parent 0 root 10  owner 2 offset 0
Failed to find [6815744, 168, 16384]
btrfs unable to find ref byte nr 55431977254912 parent 0 root 10  owner 1 offset 0
Failed to find [6815744, 168, 16384]
btrfs unable to find ref byte nr 55431977271296 parent 0 root 10  owner 0 offset 0
Failed to find [6815744, 168, 16384]
btrfs unable to find ref byte nr 55431977287680 parent 0 root 10  owner 1 offset 0
Failed to find [6815744, 168, 16384]
btrfs unable to find ref byte nr 55431977304064 parent 0 root 10  owner 0 offset 0
parent transid verify failed on 55431977353216 wanted 60235 found 60236
Ignoring transid failure
Failed to find [55431977238528, 168, 16384]
btrfs unable to find ref byte nr 55431977320448 parent 0 root 1  owner 1 offset 0
Failed to find [55431977238528, 168, 16384]
btrfs unable to find ref byte nr 55431977336832 parent 0 root 1  owner 0 offset 0
Failed to find [55431977353216, 168, 16384]
btrfs unable to find ref byte nr 55431977369600 parent 0 root 1  owner 0 offset 0
[1/7] checking root items... skipped
ERROR: extent[1048576 16384] backref lost (owner: 3, level: 1) root 3

## (shortened - same output repeating several times)

Added an extent item [1048576 16384]
Added one tree block ref start 1048576 root 3
ERROR: extent[1064960 16384] backref lost (owner: 3, level: 0) root 3
Added an extent item [1064960 16384]
Added one tree block ref start 1064960 root 3
ERROR: Dev extent's total-byte 15123666173952 is not equal to bytes-used 15121518690304 in dev[1, 204, 1]
ERROR: Dev extent's total-byte 15123687145472 is not equal to bytes-used 15121539661824 in dev[1, 204, 2]
ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 3]
ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 4]
ERROR: extent[1163264 16384] backref lost (owner: 3, level: 0) root 3
Added an extent item [1163264 16384]
Added one tree block ref start 1163264 root 3
ERROR: extent[1179648 16384] backref lost (owner: 3, level: 0) root 3
Added an extent item [1179648 16384]
Added one tree block ref start 1179648 root 3
ERROR: extent[1196032 16384] backref lost (owner: 3, level: 0) root 3
Added an extent item [55431977877504 16384]
Added one tree block ref start 55431977877504 root 1
ERROR: extent[55431977861120 16384] backref lost (owner: 10, level: 0) root 1
Added an extent item [55431977861120 16384]
Added one tree block ref start 55431977861120 root 1
ERROR: extent[54142438047744 16384] backref lost (owner: 10, level: 0) root 1
Added an extent item [54142438047744 16384]
Added one tree block ref start 54142438047744 root 1
ERROR: extent [54142438047744 16384] referencer bytenr mismatch, wanted: 54142438047744, have: 55432766324736
Delete backref in extent [54142438047744 16384]
Failed to find [55432521760768, 168, 16384]    (0:00:06 elapsed, 225 items checked)
btrfs unable to find ref byte nr 55432762015744 parent 0 root 10  owner 0 offset 0
transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
btrfs[0x43ebef]
btrfs(btrfs_commit_transaction+0x89)[0x43f07a]
btrfs[0x47baec]
btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2]
btrfs(cmd_check+0x1e34)[0x460cd4]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6ce4825b35]
btrfs[0x40c509]
[1]    1049 abort      btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh


###

And on another run of the same command, yet another crash:

###



ERROR: extent[34096057876480 16384] backref lost (owner: 1, level: 0) root 1
Added an extent item [34096057876480 16384]
Added one tree block ref start 34096057876480 root 1
ERROR: extent[3543721410560 16384] backref lost (owner: 1, level: 0) root 1
Added an extent item [3543721410560 16384]
Added one tree block ref start 3543721410560 root 1
ERROR: extent[55432521760768 16384] backref lost (owner: 10, level: 0) root 1
Added an extent item [55432521760768 16384]
Added one tree block ref start 55432521760768 root 1
ERROR: extent[55431977877504 16384] backref lost (owner: 10, level: 0) root 1
Added an extent item [55431977877504 16384]
Added one tree block ref start 55431977877504 root 1
check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1
btrfs[0x478fb4]
btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
btrfs(cmd_check+0x1e34)[0x460cd4]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
btrfs[0x40c509]
check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1
btrfs[0x478fb4]
btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
btrfs(cmd_check+0x1e34)[0x460cd4]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
btrfs[0x40c509]
ERROR: extent[55432747909120 16384] backref lost (owner: 10, level: 1) root 1
Added an extent item [55432747909120 16384]
Added one tree block ref start 55432747909120 root 1
ctree.c:1725: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1
btrfs[0x40cb13]
btrfs(btrfs_leaf_free_space+0x7f)[0x40ee06]
btrfs[0x474092]
btrfs[0x47906f]
btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
btrfs(cmd_check+0x1e34)[0x460cd4]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
btrfs[0x40c509]
leaf is not a leaf 55432747909120
ERROR: extent [55431977877504 16384] referencer bytenr mismatch, wanted: 55431977877504, have: 55432766324736
Delete backref in extent [55431977877504 16384]
ERROR: extent [55432747909120 16384] referencer bytenr mismatch, wanted: 55432747909120, have: 55432766324736
Delete backref in extent [55432747909120 16384]
Failed to find [55432521760768, 168, 16384]
btrfs unable to find ref byte nr 55432746516480 parent 0 root 10  owner 0 offset 0
transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
btrfs[0x43ebef]
btrfs(btrfs_commit_transaction+0x89)[0x43f07a]
btrfs[0x47baec]
btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2]
btrfs(cmd_check+0x1e34)[0x460cd4]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
btrfs[0x40c509]
[1]    1652 abort      btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, March 31, 2019 4:21 PM, berodual_xyz <berodual_xyz@protonmail.com> wrote:

> Dear list,
>
> following from earlier this weeks case of attempting data rescue on a corrupt FS, we have now cloned all devices and can do potentially dangerous rescue attempts (since we can re-clone the original disks).
>
> On kernel 4.20.17 and btrfs-progs 4.20.2:
>
> --
>
> $ btrfs inspect-internal tree-stats /dev/sdh
> parent transid verify failed on 1048576 wanted 60234 found 60230
> parent transid verify failed on 1048576 wanted 60234 found 60230
> Ignoring transid failure
> volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
> btrfs[0x426fdc]
> btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
> btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
> btrfs(btrfs_setup_all_roots+0x368)[0x416540]
> btrfs[0x416a8a]
> btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
> btrfs(open_ctree+0x75)[0x416c61]
> btrfs(cmd_inspect_tree_stats+0xfa)[0x4706ad]
> btrfs(handle_command_group+0x5d)[0x40c7b3]
> btrfs(cmd_inspect+0x15)[0x4499f1]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0f26c7cb35]
> btrfs[0x40c509]
> [1] 15273 abort btrfs inspect-internal tree-stats /dev/sdh
>
> ###
>
> $ btrfs rescue zero-log /dev/sdh
> parent transid verify failed on 1048576 wanted 60234 found 60230
> parent transid verify failed on 1048576 wanted 60234 found 60230
> Ignoring transid failure
> volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
> btrfs[0x426fdc]
> btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
> btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
> btrfs(btrfs_setup_all_roots+0x368)[0x416540]
> btrfs[0x416a8a]
> btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
> btrfs(open_ctree+0x75)[0x416c61]
> btrfs[0x4657ff]
> btrfs(handle_command_group+0x5d)[0x40c7b3]
> btrfs(cmd_rescue+0x15)[0x465b98]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4507ab1b35]
> btrfs[0x40c509]
> [1] 30211 abort btrfs rescue zero-log /dev/sdh
>
> ###
>
> Even if we will not be able to recover the data, adding these traces here hopefully helps enhancing the tools in the future as a crash is most definitely not the "expected behavior".
>
> Have a nice weekend and I hope to receive some feedback.
>
> Thanks in advance,
>
> Marcel

Re: volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1

[berodual_xyz]

Dear all,

"btrfs rescue chunk-recover" also crashes. I wonder if having this crash fixed in the tools is preventing me from restoring the filesystem? Or maybe manually fixing the data corruption that causes this?

##

 Chunk: start = 61070229110784, len = 4294967296, type = 9, num_stripes = 4
      Stripes list:
      [ 0] Stripe: devid = 0, offset = 0
      [ 1] Stripe: devid = 0, offset = 0
      [ 2] Stripe: devid = 0, offset = 0
      [ 3] Stripe: devid = 0, offset = 0
      Block Group: start = 61070229110784, len = 4294967296, flag = 9
      Device extent list:
          [ 0]Device extent: devid = 4, start = 15122580897792, len = 1073741824, chunk offset = 61070229110784
          [ 1]Device extent: devid = 3, start = 15122580897792, len = 1073741824, chunk offset = 61070229110784
          [ 2]Device extent: devid = 1, start = 15122635423744, len = 1073741824, chunk offset = 61070229110784
          [ 3]Device extent: devid = 2, start = 15122614452224, len = 1073741824, chunk offset = 61070229110784
Unrecoverable Chunks:
  Chunk: start = 6329730072576, len = 33554432, type = 2, num_stripes = 0
      Stripes list:
      Block Group: start = 6329730072576, len = 33554432, flag = 2
      No device extent.

Total Chunks:                      14126
  Recoverable:                    14125
  Unrecoverable: 1

Orphan Block Groups:

Orphan Device Extents:

No block group[61074524078080, 4294967296]
No block group[61065934143488, 4294967296]
No block group[61070229110784, 4294967296]
No block group[6329730072576, 33554432]
Failed to find [55432747909120, 168, 16384]
btrfs unable to find ref byte nr 55432747909120 parent 0 root 10  owner 1 offset 0
transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
btrfs[0x43ebef]
btrfs(btrfs_commit_transaction+0x89)[0x43f07a]
btrfs(btrfs_recover_chunk_tree+0x327a)[0x46a85f]
btrfs[0x465b08]
btrfs(handle_command_group+0x5d)[0x40c7b3]
btrfs(cmd_rescue+0x15)[0x465b98]
btrfs(main+0x24a)[0x40ca02]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc9a26c3b35]
btrfs[0x40c509]
[1]    2750 abort      btrfs rescue chunk-recover -y -vv /dev/sdh
###

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, 31. March 2019 16:38, berodual_xyz <berodual_xyz@protonmail.com> wrote:

> Adding to last email, being more brave to test out commands now:
>
> Different crash than on last report.
>
> Command:
>
> btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh
>
> =========================================================
>
> #########
>
> $ btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh
> WARNING: low-memory mode repair support is only partial
> Opening filesystem to check...
> Checking filesystem on /dev/sdh
> UUID: 8b19ff46-3f42-4f51-be6b-5fc8a7d8f2cd
> Creating a new extent tree
> Failed to find [6815744, 168, 16384]
> btrfs unable to find ref byte nr 55431977238528 parent 0 root 10 owner 2 offset 0
> Failed to find [6815744, 168, 16384]
> btrfs unable to find ref byte nr 55431977254912 parent 0 root 10 owner 1 offset 0
> Failed to find [6815744, 168, 16384]
> btrfs unable to find ref byte nr 55431977271296 parent 0 root 10 owner 0 offset 0
> Failed to find [6815744, 168, 16384]
> btrfs unable to find ref byte nr 55431977287680 parent 0 root 10 owner 1 offset 0
> Failed to find [6815744, 168, 16384]
> btrfs unable to find ref byte nr 55431977304064 parent 0 root 10 owner 0 offset 0
> parent transid verify failed on 55431977353216 wanted 60235 found 60236
> Ignoring transid failure
> Failed to find [55431977238528, 168, 16384]
> btrfs unable to find ref byte nr 55431977320448 parent 0 root 1 owner 1 offset 0
> Failed to find [55431977238528, 168, 16384]
> btrfs unable to find ref byte nr 55431977336832 parent 0 root 1 owner 0 offset 0
> Failed to find [55431977353216, 168, 16384]
> btrfs unable to find ref byte nr 55431977369600 parent 0 root 1 owner 0 offset 0
> [1/7] checking root items... skipped
> ERROR: extent[1048576 16384] backref lost (owner: 3, level: 1) root 3
>
> (shortened - same output repeating several times)
>
> --------------------------------------------------
>
> Added an extent item [1048576 16384]
> Added one tree block ref start 1048576 root 3
> ERROR: extent[1064960 16384] backref lost (owner: 3, level: 0) root 3
> Added an extent item [1064960 16384]
> Added one tree block ref start 1064960 root 3
> ERROR: Dev extent's total-byte 15123666173952 is not equal to bytes-used 15121518690304 in dev[1, 204, 1]
> ERROR: Dev extent's total-byte 15123687145472 is not equal to bytes-used 15121539661824 in dev[1, 204, 2]
> ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 3]
> ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 4]
> ERROR: extent[1163264 16384] backref lost (owner: 3, level: 0) root 3
> Added an extent item [1163264 16384]
> Added one tree block ref start 1163264 root 3
> ERROR: extent[1179648 16384] backref lost (owner: 3, level: 0) root 3
> Added an extent item [1179648 16384]
> Added one tree block ref start 1179648 root 3
> ERROR: extent[1196032 16384] backref lost (owner: 3, level: 0) root 3
> Added an extent item [55431977877504 16384]
> Added one tree block ref start 55431977877504 root 1
> ERROR: extent[55431977861120 16384] backref lost (owner: 10, level: 0) root 1
> Added an extent item [55431977861120 16384]
> Added one tree block ref start 55431977861120 root 1
> ERROR: extent[54142438047744 16384] backref lost (owner: 10, level: 0) root 1
> Added an extent item [54142438047744 16384]
> Added one tree block ref start 54142438047744 root 1
> ERROR: extent [54142438047744 16384] referencer bytenr mismatch, wanted: 54142438047744, have: 55432766324736
> Delete backref in extent [54142438047744 16384]
> Failed to find [55432521760768, 168, 16384] (0:00:06 elapsed, 225 items checked)
> btrfs unable to find ref byte nr 55432762015744 parent 0 root 10 owner 0 offset 0
> transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
> btrfs[0x43ebef]
> btrfs(btrfs_commit_transaction+0x89)[0x43f07a]
> btrfs[0x47baec]
> btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2]
> btrfs(cmd_check+0x1e34)[0x460cd4]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6ce4825b35]
> btrfs[0x40c509]
> [1] 1049 abort btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh
>
> ###
>
> And on another run of the same command, yet another crash:
>
> ###
>
> ERROR: extent[34096057876480 16384] backref lost (owner: 1, level: 0) root 1
> Added an extent item [34096057876480 16384]
> Added one tree block ref start 34096057876480 root 1
> ERROR: extent[3543721410560 16384] backref lost (owner: 1, level: 0) root 1
> Added an extent item [3543721410560 16384]
> Added one tree block ref start 3543721410560 root 1
> ERROR: extent[55432521760768 16384] backref lost (owner: 10, level: 0) root 1
> Added an extent item [55432521760768 16384]
> Added one tree block ref start 55432521760768 root 1
> ERROR: extent[55431977877504 16384] backref lost (owner: 10, level: 0) root 1
> Added an extent item [55431977877504 16384]
> Added one tree block ref start 55431977877504 root 1
> check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1
> btrfs[0x478fb4]
> btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
> btrfs(cmd_check+0x1e34)[0x460cd4]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
> btrfs[0x40c509]
> check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1
> btrfs[0x478fb4]
> btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
> btrfs(cmd_check+0x1e34)[0x460cd4]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
> btrfs[0x40c509]
> ERROR: extent[55432747909120 16384] backref lost (owner: 10, level: 1) root 1
> Added an extent item [55432747909120 16384]
> Added one tree block ref start 55432747909120 root 1
> ctree.c:1725: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1
> btrfs[0x40cb13]
> btrfs(btrfs_leaf_free_space+0x7f)[0x40ee06]
> btrfs[0x474092]
> btrfs[0x47906f]
> btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759]
> btrfs(cmd_check+0x1e34)[0x460cd4]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
> btrfs[0x40c509]
> leaf is not a leaf 55432747909120
> ERROR: extent [55431977877504 16384] referencer bytenr mismatch, wanted: 55431977877504, have: 55432766324736
> Delete backref in extent [55431977877504 16384]
> ERROR: extent [55432747909120 16384] referencer bytenr mismatch, wanted: 55432747909120, have: 55432766324736
> Delete backref in extent [55432747909120 16384]
> Failed to find [55432521760768, 168, 16384]
> btrfs unable to find ref byte nr 55432746516480 parent 0 root 10 owner 0 offset 0
> transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
> btrfs[0x43ebef]
> btrfs(btrfs_commit_transaction+0x89)[0x43f07a]
> btrfs[0x47baec]
> btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2]
> btrfs(cmd_check+0x1e34)[0x460cd4]
> btrfs(main+0x24a)[0x40ca02]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35]
> btrfs[0x40c509]
> [1] 1652 abort btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Sunday, March 31, 2019 4:21 PM, berodual_xyz berodual_xyz@protonmail.com wrote:
>
> > Dear list,
> > following from earlier this weeks case of attempting data rescue on a corrupt FS, we have now cloned all devices and can do potentially dangerous rescue attempts (since we can re-clone the original disks).
> > On kernel 4.20.17 and btrfs-progs 4.20.2:
> > --
> > $ btrfs inspect-internal tree-stats /dev/sdh
> > parent transid verify failed on 1048576 wanted 60234 found 60230
> > parent transid verify failed on 1048576 wanted 60234 found 60230
> > Ignoring transid failure
> > volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
> > btrfs[0x426fdc]
> > btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
> > btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
> > btrfs(btrfs_setup_all_roots+0x368)[0x416540]
> > btrfs[0x416a8a]
> > btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
> > btrfs(open_ctree+0x75)[0x416c61]
> > btrfs(cmd_inspect_tree_stats+0xfa)[0x4706ad]
> > btrfs(handle_command_group+0x5d)[0x40c7b3]
> > btrfs(cmd_inspect+0x15)[0x4499f1]
> > btrfs(main+0x24a)[0x40ca02]
> > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0f26c7cb35]
> > btrfs[0x40c509]
> > [1] 15273 abort btrfs inspect-internal tree-stats /dev/sdh
> >
> > ###
> >
> > $ btrfs rescue zero-log /dev/sdh
> > parent transid verify failed on 1048576 wanted 60234 found 60230
> > parent transid verify failed on 1048576 wanted 60234 found 60230
> > Ignoring transid failure
> > volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
> > btrfs[0x426fdc]
> > btrfs(btrfs_chunk_readonly+0x98)[0x429acd]
> > btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44]
> > btrfs(btrfs_setup_all_roots+0x368)[0x416540]
> > btrfs[0x416a8a]
> > btrfs(open_ctree_fs_info+0xd0)[0x416bcc]
> > btrfs(open_ctree+0x75)[0x416c61]
> > btrfs[0x4657ff]
> > btrfs(handle_command_group+0x5d)[0x40c7b3]
> > btrfs(cmd_rescue+0x15)[0x465b98]
> > btrfs(main+0x24a)[0x40ca02]
> > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4507ab1b35]
> > btrfs[0x40c509]
> > [1] 30211 abort btrfs rescue zero-log /dev/sdh
> >
> > ###
> >
> > Even if we will not be able to recover the data, adding these traces here hopefully helps enhancing the tools in the future as a crash is most definitely not the "expected behavior".
> > Have a nice weekend and I hope to receive some feedback.
> > Thanks in advance,
> > Marcel

Re: volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1

[Chris Murphy]

On Mon, Apr 1, 2019 at 6:40 AM berodual_xyz <berodual_xyz@protonmail.com> wrote:
>
> Dear all,
>
> "btrfs rescue chunk-recover" also crashes. I wonder if having this crash fixed in the tools is preventing me from restoring the filesystem?

The crashing could in fact make the problem worse if it's also writing
any changes to disk that are interrupted by the crash. So yeah, until
the crasher is fixed, it's not going to improve the situation.

>Or maybe manually fixing the data corruption that causes this?

I don't understand the question. If you use --init-extent-tree it
almost immediately starts writing a new extent tree. And if that's
interrupted at all or does not otherwise succeed, the file system is
useless for anything else. You'd need to reimage your clones from
source.

Over on the linux-raid@ list, they do this, which I think also works
for LVM. It definitely doesn't work with Btrfs.
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

The reason why is because the overlays and original block devices are
indistinguishable to Btrfs. And the user space tools have no way of
explicitly specifying the block device. You pick one device, and it
uses UUID to infer the rest of the members, and it's uncontrollable
which ones it'll pick. If you have a separate computer, you could
export only the rw overlays via iSCSI to another computer, and that
would likely work. The one thing I'd add to the overlay recipe, is
using `blockdev --setro` on all the original drives, just to make
certain they aren't written to.

It's more than a little frustrating we don't have non-destructive
repair on Btrfs. Btrfs has a full volume snapshot via the seed device
feature, so a future repair feature could just do that automatically
and write out the repair to another volume or to a file. And then that
repair attempt could be sanity checked just by mounting the sprout
device containing the repaired file system. Even if it's not a perfect
repair it might allow a read only mount, and permit normal extraction
methods. It would also be better for developers to always have access
to the before repair state, and after repair state. But right now
we've got numerous cases for a long time of `btrfs check --repair`
blowing up file systems worse than they were before the repair was
tried, and in the process the file system is so badly damaged it's not
even useful for user or developer.


-- 
Chris Murphy