[bug] btrfs check clear-space-cache v1 corrupted file system

[bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

Problem happens with btrfsprogs 4.19.1

https://bugzilla.kernel.org/show_bug.cgi?id=202717

That bug report includes the trace in user space; but I've also
processed the resulting coredump file with gdb and attached that to
the bug report.

Before using --clear-space-cache v1, btrfs scrub and btrfs check came
up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
and ran 'btrfs check' which finds corruption.

I've taken no further action, and the btrfs check output gives me no
useful plain language information what the next steps are. So I'm just
gonna leave it alone since it's a backup volume anyway.



-- 
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 973 bytes --]



On 2019/3/2 上午8:20, Chris Murphy wrote:
> Problem happens with btrfsprogs 4.19.1
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=202717
> 
> That bug report includes the trace in user space; but I've also
> processed the resulting coredump file with gdb and attached that to
> the bug report.
> 
> Before using --clear-space-cache v1, btrfs scrub and btrfs check came
> up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
> and ran 'btrfs check' which finds corruption.
> 
> I've taken no further action, and the btrfs check output gives me no
> useful plain language information what the next steps are. So I'm just
> gonna leave it alone since it's a backup volume anyway.

Is the backup volume binary dump of the original fs?

I'm interesting why clear space cache v1 doesn't work, to find out that,
we need tree dump of the original fs.

And is the original check done by latest btrfs-progs or v4.19.1?

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

On Fri, Mar 1, 2019 at 6:05 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/3/2 上午8:20, Chris Murphy wrote:
> > Problem happens with btrfsprogs 4.19.1
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=202717
> >
> > That bug report includes the trace in user space; but I've also
> > processed the resulting coredump file with gdb and attached that to
> > the bug report.
> >
> > Before using --clear-space-cache v1, btrfs scrub and btrfs check came
> > up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
> > and ran 'btrfs check' which finds corruption.
> >
> > I've taken no further action, and the btrfs check output gives me no
> > useful plain language information what the next steps are. So I'm just
> > gonna leave it alone since it's a backup volume anyway.
>
> Is the backup volume binary dump of the original fs?

No. Only btrfs send/receive.


>
> I'm interesting why clear space cache v1 doesn't work, to find out that,
> we need tree dump of the original fs.

btrfs insp dump-t -t 2 ? Or the whole thing?

>
> And is the original check done by latest btrfs-progs or v4.19.1?

The original check is 4.19.1
The clear cache is 4.19.1
The subsequent check showing corruption is 4.20.2 but I've found that
it's the same output for 4.19.1 as well.


-- 
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

OK this is screwy. The super has a totally different generation for
the root tree than any of the backup roots.

$ sudo btrfs rescue super -v /dev/mapper/sdd
All Devices:
    Device: id = 1, name = /dev/mapper/sdc
    Device: id = 2, name = /dev/mapper/sdd

Before Recovering:
    [All good supers]:
        device name = /dev/mapper/sdc
        superblock bytenr = 65536

        device name = /dev/mapper/sdc
        superblock bytenr = 67108864

        device name = /dev/mapper/sdc
        superblock bytenr = 274877906944

        device name = /dev/mapper/sdd
        superblock bytenr = 65536

        device name = /dev/mapper/sdd
        superblock bytenr = 67108864

        device name = /dev/mapper/sdd
        superblock bytenr = 274877906944

    [All bad supers]:

All supers are valid, no need to recover

$ sudo btrfs insp dump-s -f /dev/mapper/sdd
superblock: bytenr=65536, device=/dev/mapper/sdd
---------------------------------------------------------
csum_type        0 (crc32c)
csum_size        4
csum            0x312a8b5e [match]
bytenr            65536
flags            0x1
            ( WRITTEN )
magic            _BHRfS_M [match]
fsid            f5adc913-bbea-4340-8b5f-3411e2cda642
label            second
generation        4514
root            29392896
sys_array_size        258
chunk_root_generation    3823
root_level        1
chunk_root        822549151744
chunk_root_level    1
log_root        0
log_root_transid    0
log_root_level        0
total_bytes        1500308553728
bytes_used        735327125504
sectorsize        4096
nodesize        16384
leafsize (deprecated)        16384
stripesize        4096
root_dir        6
num_devices        2
compat_flags        0x0
compat_ro_flags        0x0
incompat_flags        0x161
            ( MIXED_BACKREF |
              BIG_METADATA |
              EXTENDED_IREF |
              SKINNY_METADATA )
cache_generation    4076
uuid_tree_generation    4076
dev_item.uuid        2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
dev_item.fsid        f5adc913-bbea-4340-8b5f-3411e2cda642 [match]
dev_item.type        0
dev_item.total_bytes    750154276864
dev_item.bytes_used    737702576128
dev_item.io_align    4096
dev_item.io_width    4096
dev_item.sector_size    4096
dev_item.devid        2
dev_item.dev_group    0
dev_item.seek_speed    0
dev_item.bandwidth    0
dev_item.generation    0
sys_chunk_array[2048]:
    item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
        length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 0
            stripe 0 devid 2 offset 1048576
            dev_uuid 2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
            stripe 1 devid 1 offset 20971520
            dev_uuid d071b0ca-1d27-48cb-826e-63c5e9972339
    item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 822549151744)
        length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
        io_align 65536 io_width 65536 sector_size 4096
        num_stripes 2 sub_stripes 1
            stripe 0 devid 2 offset 737670070272
            dev_uuid 2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
            stripe 1 devid 1 offset 737689993216
            dev_uuid d071b0ca-1d27-48cb-826e-63c5e9972339
backup_roots[4]:
    backup 0:
        backup_tree_root:    931053568    gen: 4075    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1
        backup_extent_root:    931086336    gen: 4075    level: 2
        backup_fs_root:        537116672    gen: 3806    level: 0
        backup_dev_root:    721174528    gen: 3823    level: 1
        backup_csum_root:    932249600    gen: 4075    level: 2
        backup_total_bytes:    1500308553728
        backup_bytes_used:    735441764352
        backup_num_devices:    2

    backup 1:
        backup_tree_root:    930086912    gen: 4076    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1
        backup_extent_root:    930447360    gen: 4076    level: 2
        backup_fs_root:        537116672    gen: 3806    level: 0
        backup_dev_root:    930856960    gen: 4076    level: 1
        backup_csum_root:    932118528    gen: 4076    level: 2
        backup_total_bytes:    1500308553728
        backup_bytes_used:    735441764352
        backup_num_devices:    2

    backup 2:
        backup_tree_root:    930070528    gen: 4073    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1
        backup_extent_root:    930365440    gen: 4073    level: 2
        backup_fs_root:        537116672    gen: 3806    level: 0
        backup_dev_root:    721174528    gen: 3823    level: 1
        backup_csum_root:    931086336    gen: 4073    level: 2
        backup_total_bytes:    1500308553728
        backup_bytes_used:    735441764352
        backup_num_devices:    2

    backup 3:
        backup_tree_root:    930086912    gen: 4074    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1
        backup_extent_root:    930447360    gen: 4074    level: 2
        backup_fs_root:        537116672    gen: 3806    level: 0
        backup_dev_root:    721174528    gen: 3823    level: 1
        backup_csum_root:    932052992    gen: 4074    level: 2
        backup_total_bytes:    1500308553728
        backup_bytes_used:    735441764352
        backup_num_devices:    2
$


Super says root tree generation 4514; but the most recent backup says
4076. What?

# btrfs insp dump-t -b 930086912 /dev/mapper/sdd
btrfs-progs v4.19.1
node 930086912 level 1 items 19 free 474 generation 4076 owner ROOT_TREE
fs uuid f5adc913-bbea-4340-8b5f-3411e2cda642
chunk uuid 4af0d265-00fa-4a2b-9ea9-59c06a372284
    key (EXTENT_TREE ROOT_ITEM 0) block 931102720 gen 4076
    key (279 INODE_ITEM 0) block 535216128 gen 3802
    key (307 INODE_ITEM 0) block 715111579648 gen 3511
    key (368 EXTENT_DATA 0) block 715103928320 gen 3510
    key (430 INODE_ITEM 0) block 715111612416 gen 3511
    key (491 EXTENT_DATA 0) block 715111677952 gen 3511
    key (553 INODE_ITEM 0) block 715111841792 gen 3511
    key (652 EXTENT_DATA 0) block 714560716800 gen 3523
    key (711 INODE_ITEM 0) block 934019072 gen 4076
    key (773 INODE_ITEM 0) block 934887424 gen 4076
    key (829 INODE_ITEM 0) block 671989760 gen 3808
    key (884 INODE_ITEM 0) block 714619404288 gen 3525
    key (939 ROOT_ITEM 1889) block 715035377664 gen 3506
    key (981 ROOT_ITEM 3330) block 933658624 gen 4067
    key (1034 INODE_ITEM 0) block 172621824 gen 3546
    key (1095 EXTENT_DATA 0) block 934232064 gen 4076
    key (FREE_SPACE UNTYPED 11840520192) block 934248448 gen 4076
    key (FREE_SPACE UNTYPED 275981008896) block 934658048 gen 4076
    key (FREE_SPACE UNTYPED 568038785024) block 933724160 gen 4067
# btrfs insp dump-t -b 29392896 /dev/mapper/sdd
btrfs-progs v4.19.1
node 29392896 level 1 items 10 free 483 generation 4514 owner ROOT_TREE
fs uuid f5adc913-bbea-4340-8b5f-3411e2cda642
chunk uuid 4af0d265-00fa-4a2b-9ea9-59c06a372284
    key (EXTENT_TREE ROOT_ITEM 0) block 712994078720 gen 4514
    key (773 INODE_ITEM 0) block 29442048 gen 4470
    key (829 INODE_ITEM 0) block 671989760 gen 3808
    key (884 INODE_ITEM 0) block 714619404288 gen 3525
    key (939 ROOT_ITEM 1889) block 715035377664 gen 3506
    key (981 ROOT_ITEM 3330) block 933658624 gen 4067
    key (1034 INODE_ITEM 0) block 172621824 gen 3546
    key (1095 EXTENT_DATA 0) block 29409280 gen 4514
    key (FREE_SPACE UNTYPED 559448850432) block 29376512 gen 4483
    key (FREE_SPACE UNTYPED 568038785024) block 933724160 gen 4067
#

I guess I don't understand why 'btrfs check' would have successfully
written six updated copies of the super, when the operation hadn't yet
succeeded. I guess this isn't an atomic operation, and perhaps it
can't be.

---
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

Two more strange things:

# btrfs check -r 930086912 /dev/mapper/sdd
Opening filesystem to check...
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
Ignoring transid failure
Checking filesystem on /dev/mapper/sdd
UUID: f5adc913-bbea-4340-8b5f-3411e2cda642
[1/7] checking root items
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
Ignoring transid failure
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=672006144 item=178 parent
level=1 child level=1
ERROR: failed to repair root items: Input/output error
#

dmesg most recent messages are four hours ago, nothing at the time of
this input/output error.
Mar 01 16:19:24 fnuc.local kernel: BTRFS info (device dm-2): disk
space caching is enabled
Mar 01 16:19:24 fnuc.local kernel: BTRFS info (device dm-2): has skinny extents



And next, why are backup 1 and backup 3 the same except for the gen
for tree, extent, and csum roots? The addresses are all the same. How
is that COW?

backup 1:
        backup_tree_root:    930086912    gen: 4076    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1

    backup 3:
        backup_tree_root:    930086912    gen: 4074    level: 1
        backup_chunk_root:    822549151744    gen: 3823    level: 1

When I go inspect that address, it's of course generation 4076. So
this backup 3 slot is actually invalid, it's already been overwritten.

Anyway, if you need more than just the extent tree, I need a hint on
how to sanitize file names. I tried looking through the archives, I
thought you posted a sed trick to do that with debug-tree but I can't
find it. And I know you don't like btrfs-image -s or -ss.


---
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1531 bytes --]



On 2019/3/2 上午10:29, Chris Murphy wrote:
> On Fri, Mar 1, 2019 at 6:05 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/3/2 上午8:20, Chris Murphy wrote:
>>> Problem happens with btrfsprogs 4.19.1
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=202717
>>>
>>> That bug report includes the trace in user space; but I've also
>>> processed the resulting coredump file with gdb and attached that to
>>> the bug report.
>>>
>>> Before using --clear-space-cache v1, btrfs scrub and btrfs check came
>>> up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
>>> and ran 'btrfs check' which finds corruption.
>>>
>>> I've taken no further action, and the btrfs check output gives me no
>>> useful plain language information what the next steps are. So I'm just
>>> gonna leave it alone since it's a backup volume anyway.
>>
>> Is the backup volume binary dump of the original fs?
> 
> No. Only btrfs send/receive.
> 
> 
>>
>> I'm interesting why clear space cache v1 doesn't work, to find out that,
>> we need tree dump of the original fs.
> 
> btrfs insp dump-t -t 2 ? Or the whole thing?

extent tree and root tree.
As those are the only trees modified in clear extent tree.

Thanks,
Qu

> 
>>
>> And is the original check done by latest btrfs-progs or v4.19.1?
> 
> The original check is 4.19.1
> The clear cache is 4.19.1
> The subsequent check showing corruption is 4.20.2 but I've found that
> it's the same output for 4.19.1 as well.
>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> Sending URL for dump-tree output offlist. Conversation should still be on-list.


Any more information required from me at this point?


-- 
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > Sending URL for dump-tree output offlist. Conversation should still be on-list.
>
>
> Any more information required from me at this point?

This file system has been on a shelf since the problem happened. Is
the lack of COW repairs with btrfs check solved? Can this file system
be fixed? Maybe --init-extent-tree is worth a shot?


-- 
Chris Murphy

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 964 bytes --]



On 2019/8/12 下午12:24, Chris Murphy wrote:
> On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
>>
>> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
>>>
>>> Sending URL for dump-tree output offlist. Conversation should still be on-list.
>>
>>
>> Any more information required from me at this point?
> 
> This file system has been on a shelf since the problem happened. Is
> the lack of COW repairs with btrfs check solved? Can this file system
> be fixed? Maybe --init-extent-tree is worth a shot?
> 
> 
Latest btrfs check has solved the problem of bad CoW. In fact it's
solved in btrfs-progs v5.1 release.

So, if you run btrfs check --clear-space-cache v1 again, it shouldn't
cause transid mismatch problem any more.

Although IIRC that fs already got transid error, thus enhanced
btrfs-check won't help much to solve the existing transid mismatch problem.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bug] btrfs check clear-space-cache v1 corrupted file system

[Chris Murphy]

On Sun, Aug 11, 2019 at 10:54 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/8/12 下午12:24, Chris Murphy wrote:
> > On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
> >>
> >> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
> >>>
> >>> Sending URL for dump-tree output offlist. Conversation should still be on-list.
> >>
> >>
> >> Any more information required from me at this point?
> >
> > This file system has been on a shelf since the problem happened. Is
> > the lack of COW repairs with btrfs check solved? Can this file system
> > be fixed? Maybe --init-extent-tree is worth a shot?
> >
> >
> Latest btrfs check has solved the problem of bad CoW. In fact it's
> solved in btrfs-progs v5.1 release.
>
> So, if you run btrfs check --clear-space-cache v1 again, it shouldn't
> cause transid mismatch problem any more.

That's good news.

>
> Although IIRC that fs already got transid error, thus enhanced
> btrfs-check won't help much to solve the existing transid mismatch problem.

OK and the other bug report I just posted is this same file system
failing to ro scrub. It's not obvious that the failure is due to
transid mismatch.

If the file system can't be fixed, I'll wipe it.

-- 
Chris Murphy