Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

[Marc MERLIN <marc@merlins.org>]

On Tue, May 10, 2022 at 12:14:37PM -0400, Josef Bacik wrote:
> Ah duh, we delete it the first time, but then we find another
> overlapping extent and we try to delete it again and it's not there.
> I've fixed this up, try again please.  Thanks,

good news, it worked! :)

I started a simple ./btrfs check --repair /dev/mapper/dshelf1
did you want me to add options to it or run as is?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0]
misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328
misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

        Do not use --repair unless you are advised to do so by a developer
        or an experienced user, and then only after having accepted that no
        fsck can successfully repair all types of filesystem corruption. Eg.
        some software or hardware bugs can fatally damage a volume.
        The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56119a61efd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110
incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180
backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210
incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60
backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720
incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880
backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0
incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0
backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690
incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0
backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0
incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0
backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10
incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0
backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750
incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0
backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780
incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0
backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50
incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660
backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08