kernel 5.2 read time tree block corruption

kernel 5.2 read time tree block corruption

[José Luis]

Dear devs,

I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
filesystems. I can work as intended on 4.19 which is an LTS version,
previously using 5.1 but Manjaro removed it from their repositories.

More info:
· dmesg:
> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
> [  +0,044643] BTRFS error (device sdb2): open_ctree failed



· sudo mount  /dev/sdb2 /mnt/
> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.

(cannot read superblock on /dev...)

· sudo btrfs rescue super-recover /dev/sdb2
> All supers are valid, no need to recover


· sudo btrfs check /dev/sdb2
> Opening filesystem to check...
> Checking filesystem on /dev/sdb2
> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> root 5 inode 431 errors 1040, bad file extent, some csum missing
> root 5 inode 755 errors 1040, bad file extent, some csum missing
> root 5 inode 2379 errors 1040, bad file extent, some csum missing
> root 5 inode 11721 errors 1040, bad file extent, some csum missing
> root 5 inode 12211 errors 1040, bad file extent, some csum missing
> root 5 inode 15368 errors 1040, bad file extent, some csum missing
> root 5 inode 35329 errors 1040, bad file extent, some csum missing
> root 5 inode 960427 errors 1040, bad file extent, some csum missing
> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
> root 388 inode 1245 errors 1040, bad file extent, some csum missing
> root 388 inode 1288 errors 1040, bad file extent, some csum missing
> root 388 inode 1292 errors 1040, bad file extent, some csum missing
> root 388 inode 1313 errors 1040, bad file extent, some csum missing
> root 388 inode 11870 errors 1040, bad file extent, some csum missing
> root 388 inode 68126 errors 1040, bad file extent, some csum missing
> root 388 inode 88051 errors 1040, bad file extent, some csum missing
> root 388 inode 88255 errors 1040, bad file extent, some csum missing
> root 388 inode 88455 errors 1040, bad file extent, some csum missing
> root 388 inode 88588 errors 1040, bad file extent, some csum missing
> root 388 inode 88784 errors 1040, bad file extent, some csum missing
> root 388 inode 88916 errors 1040, bad file extent, some csum missing
> ERROR: errors found in fs roots
> found 37167415296 bytes used, error(s) found
> total csum bytes: 33793568
> total tree bytes: 1676722176
> total fs tree bytes: 1540243456
> total extent tree bytes: 81510400
> btree space waste bytes: 306327457
> file data blocks allocated: 42200928256
>  referenced 52868354048




---

Regards,
José Luis.

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4202 bytes --]



On 2019/10/15 下午6:15, José Luis wrote:
> Dear devs,
> 
> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
> filesystems. I can work as intended on 4.19 which is an LTS version,
> previously using 5.1 but Manjaro removed it from their repositories.
> 
> More info:
> · dmesg:
>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604

In fs tree, you are hitting a free space cache inode?
That doesn't sound good.

Please provide the following dump:

# btrfs ins dump-tree -b 30622793728 /dev/sdb2

The output may contain filename, feel free to remove filenames.

>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
> 
> 
> 
> · sudo mount  /dev/sdb2 /mnt/
>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
> 
> (cannot read superblock on /dev...)
> 
> · sudo btrfs rescue super-recover /dev/sdb2
>> All supers are valid, no need to recover
> 
> 
> · sudo btrfs check /dev/sdb2
>> Opening filesystem to check...
>> Checking filesystem on /dev/sdb2
>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
>> [1/7] checking root items
>> [2/7] checking extents
>> [3/7] checking free space cache
>> cache and super generation don't match, space cache will be invalidated
>> [4/7] checking fs roots
>> root 5 inode 431 errors 1040, bad file extent, some csum missing
>> root 5 inode 755 errors 1040, bad file extent, some csum missing
>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref

Check is reporting the same problem of the inode.

We need to make sure what's going wrong on that leaf, based on the
mentioned dump.

For the csum missing error and bad file extent, it should be a big problem.
if you want to make sure what's going wrong, please provide the
following dump:

# btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7

Also feel free the censor the filenames.

Thanks,
Qu

>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
>> ERROR: errors found in fs roots
>> found 37167415296 bytes used, error(s) found
>> total csum bytes: 33793568
>> total tree bytes: 1676722176
>> total fs tree bytes: 1540243456
>> total extent tree bytes: 81510400
>> btree space waste bytes: 306327457
>> file data blocks allocated: 42200928256
>>  referenced 52868354048
> 
> 
> 
> 
> ---
> 
> Regards,
> José Luis.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4400 bytes --]



On 2019/10/15 下午8:24, Qu Wenruo wrote:
> 
> 
> On 2019/10/15 下午6:15, José Luis wrote:
>> Dear devs,
>>
>> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
>> filesystems. I can work as intended on 4.19 which is an LTS version,
>> previously using 5.1 but Manjaro removed it from their repositories.
>>
>> More info:
>> · dmesg:
>>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
>>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
>>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
> 
> In fs tree, you are hitting a free space cache inode?
> That doesn't sound good.
> 
> Please provide the following dump:
> 
> # btrfs ins dump-tree -b 30622793728 /dev/sdb2
> 
> The output may contain filename, feel free to remove filenames.
> 
>>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
>>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
>>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
>>
>>
>>
>> · sudo mount  /dev/sdb2 /mnt/
>>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
>>
>> (cannot read superblock on /dev...)
>>
>> · sudo btrfs rescue super-recover /dev/sdb2
>>> All supers are valid, no need to recover
>>
>>
>> · sudo btrfs check /dev/sdb2
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdb2
>>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
>>> [1/7] checking root items
>>> [2/7] checking extents
>>> [3/7] checking free space cache
>>> cache and super generation don't match, space cache will be invalidated
>>> [4/7] checking fs roots
>>> root 5 inode 431 errors 1040, bad file extent, some csum missing
>>> root 5 inode 755 errors 1040, bad file extent, some csum missing
>>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
>>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
>>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
>>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
>>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
>>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
>>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
>>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
> 
> Check is reporting the same problem of the inode.
> 
> We need to make sure what's going wrong on that leaf, based on the
> mentioned dump.
> 
> For the csum missing error and bad file extent, it should be a big problem.

s/should/should not/

> if you want to make sure what's going wrong, please provide the
> following dump:
> 
> # btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7
> 
> Also feel free the censor the filenames.
> 
> Thanks,
> Qu
> 
>>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
>>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
>>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
>>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
>>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
>>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
>>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
>>> ERROR: errors found in fs roots
>>> found 37167415296 bytes used, error(s) found
>>> total csum bytes: 33793568
>>> total tree bytes: 1676722176
>>> total fs tree bytes: 1540243456
>>> total extent tree bytes: 81510400
>>> btree space waste bytes: 306327457
>>> file data blocks allocated: 42200928256
>>>  referenced 52868354048
>>
>>
>>
>>
>> ---
>>
>> Regards,
>> José Luis.
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 6693 bytes --]



On 2019/10/15 下午11:03, José Luis wrote:
> Thanks for fast response Qu.
> 
> I booted into a pendrive live system for the test cause I'm using the
> involving fylesystem with kernel 4.19. This time when I mount
>> [manjaro@manjaro ~]$ sudo mount /dev/sdb2 /mnt
>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
> and in the dmesg:
> [ +30,866472] BTRFS info (device sdb2): disk space caching is enabled
> [  +0,017443] BTRFS info (device sdb2): enabling ssd optimizations
> [  +0,000637] BTRFS critical (device sdb2): corrupt leaf: root=5
> block=32145457152 slot=99, invalid key objectid: has
> 18446744073709551605 expect 6 or [256, 18446744073709551360] or
> 18446744073709551604
> [  +0,000002] BTRFS error (device sdb2): block=32145457152 read time
> tree block corruption detected
> [  +0,000012] BTRFS warning (device sdb2): failed to read fs tree: -5
> [  +0,061995] BTRFS error (device sdb2): open_ctree failed
> 
> So I suppose you need dump output from the block 32145457152 so I pastebin that:
> sudo btrfs ins dump-tree -b 32145457152 /dev/sdb2
> output --> https://pastebin.com/ssB5HTn7

The output is way crazier than I thought...

I was only expecting some strange inode number, but what I got is
completely ridiculous.

From item 96, we are having completely impossible inodes.
From FREE_INO to DATA_RELOC, even EXTENT_CSUM.

All of these are impossible to exist in fs tree.
The most strange thing is, they are all last modified in 2019-2-15.

Anyway, the tree-checker is doing completely valid behavior for this
case. The data is really ridiculous.

Any history about the kernel used in that time?
I see something only possible in Windows, any clue?

> 
> Please provide the parameter to the grep redirection for: "btrfs ins
> dump-tree -t 5 /dev/sdb2 | grep -A 7"

My bad, the parameter is "(431"

It will output all info about inode 431, so we can make sure what's
going wrong.

Thanks,
Qu
> 
> El mar., 15 oct. 2019 a las 14:38, Qu Wenruo
> (<quwenruo.btrfs@gmx.com>) escribió:
>>
>>
>>
>> On 2019/10/15 下午8:24, Qu Wenruo wrote:
>>>
>>>
>>> On 2019/10/15 下午6:15, José Luis wrote:
>>>> Dear devs,
>>>>
>>>> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
>>>> filesystems. I can work as intended on 4.19 which is an LTS version,
>>>> previously using 5.1 but Manjaro removed it from their repositories.
>>>>
>>>> More info:
>>>> · dmesg:
>>>>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
>>>>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
>>>>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
>>>
>>> In fs tree, you are hitting a free space cache inode?
>>> That doesn't sound good.
>>>
>>> Please provide the following dump:
>>>
>>> # btrfs ins dump-tree -b 30622793728 /dev/sdb2
>>>
>>> The output may contain filename, feel free to remove filenames.
>>>
>>>>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
>>>>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
>>>>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
>>>>
>>>>
>>>>
>>>> · sudo mount  /dev/sdb2 /mnt/
>>>>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
>>>>
>>>> (cannot read superblock on /dev...)
>>>>
>>>> · sudo btrfs rescue super-recover /dev/sdb2
>>>>> All supers are valid, no need to recover
>>>>
>>>>
>>>> · sudo btrfs check /dev/sdb2
>>>>> Opening filesystem to check...
>>>>> Checking filesystem on /dev/sdb2
>>>>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
>>>>> [1/7] checking root items
>>>>> [2/7] checking extents
>>>>> [3/7] checking free space cache
>>>>> cache and super generation don't match, space cache will be invalidated
>>>>> [4/7] checking fs roots
>>>>> root 5 inode 431 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 755 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
>>>>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
>>>>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
>>>
>>> Check is reporting the same problem of the inode.
>>>
>>> We need to make sure what's going wrong on that leaf, based on the
>>> mentioned dump.
>>>
>>> For the csum missing error and bad file extent, it should be a big problem.
>>
>> s/should/should not/
>>
>>> if you want to make sure what's going wrong, please provide the
>>> following dump:
>>>
>>> # btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7
>>>
>>> Also feel free the censor the filenames.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
>>>>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
>>>>> ERROR: errors found in fs roots
>>>>> found 37167415296 bytes used, error(s) found
>>>>> total csum bytes: 33793568
>>>>> total tree bytes: 1676722176
>>>>> total fs tree bytes: 1540243456
>>>>> total extent tree bytes: 81510400
>>>>> btree space waste bytes: 306327457
>>>>> file data blocks allocated: 42200928256
>>>>>  referenced 52868354048
>>>>
>>>>
>>>>
>>>>
>>>> ---
>>>>
>>>> Regards,
>>>> José Luis.
>>>>
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: kernel 5.2 read time tree block corruption

[José Luis]

Thanks for fast response Qu.

I booted into a pendrive live system for the test cause I'm using the
involving fylesystem with kernel 4.19. This time when I mount
>[manjaro@manjaro ~]$ sudo mount /dev/sdb2 /mnt
>mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
and in the dmesg:
[ +30,866472] BTRFS info (device sdb2): disk space caching is enabled
[  +0,017443] BTRFS info (device sdb2): enabling ssd optimizations
[  +0,000637] BTRFS critical (device sdb2): corrupt leaf: root=5
block=32145457152 slot=99, invalid key objectid: has
18446744073709551605 expect 6 or [256, 18446744073709551360] or
18446744073709551604
[  +0,000002] BTRFS error (device sdb2): block=32145457152 read time
tree block corruption detected
[  +0,000012] BTRFS warning (device sdb2): failed to read fs tree: -5
[  +0,061995] BTRFS error (device sdb2): open_ctree failed

So I suppose you need dump output from the block 32145457152 so I pastebin that:
sudo btrfs ins dump-tree -b 32145457152 /dev/sdb2
output --> https://pastebin.com/ssB5HTn7

Please provide the parameter to the grep redirection for: "btrfs ins
dump-tree -t 5 /dev/sdb2 | grep -A 7"

El mar., 15 oct. 2019 a las 14:38, Qu Wenruo
(<quwenruo.btrfs@gmx.com>) escribió:
>
>
>
> On 2019/10/15 下午8:24, Qu Wenruo wrote:
> >
> >
> > On 2019/10/15 下午6:15, José Luis wrote:
> >> Dear devs,
> >>
> >> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
> >> filesystems. I can work as intended on 4.19 which is an LTS version,
> >> previously using 5.1 but Manjaro removed it from their repositories.
> >>
> >> More info:
> >> · dmesg:
> >>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
> >>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
> >>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
> >
> > In fs tree, you are hitting a free space cache inode?
> > That doesn't sound good.
> >
> > Please provide the following dump:
> >
> > # btrfs ins dump-tree -b 30622793728 /dev/sdb2
> >
> > The output may contain filename, feel free to remove filenames.
> >
> >>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
> >>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
> >>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
> >>
> >>
> >>
> >> · sudo mount  /dev/sdb2 /mnt/
> >>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
> >>
> >> (cannot read superblock on /dev...)
> >>
> >> · sudo btrfs rescue super-recover /dev/sdb2
> >>> All supers are valid, no need to recover
> >>
> >>
> >> · sudo btrfs check /dev/sdb2
> >>> Opening filesystem to check...
> >>> Checking filesystem on /dev/sdb2
> >>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
> >>> [1/7] checking root items
> >>> [2/7] checking extents
> >>> [3/7] checking free space cache
> >>> cache and super generation don't match, space cache will be invalidated
> >>> [4/7] checking fs roots
> >>> root 5 inode 431 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 755 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
> >>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
> >>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
> >
> > Check is reporting the same problem of the inode.
> >
> > We need to make sure what's going wrong on that leaf, based on the
> > mentioned dump.
> >
> > For the csum missing error and bad file extent, it should be a big problem.
>
> s/should/should not/
>
> > if you want to make sure what's going wrong, please provide the
> > following dump:
> >
> > # btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7
> >
> > Also feel free the censor the filenames.
> >
> > Thanks,
> > Qu
> >
> >>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
> >>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
> >>> ERROR: errors found in fs roots
> >>> found 37167415296 bytes used, error(s) found
> >>> total csum bytes: 33793568
> >>> total tree bytes: 1676722176
> >>> total fs tree bytes: 1540243456
> >>> total extent tree bytes: 81510400
> >>> btree space waste bytes: 306327457
> >>> file data blocks allocated: 42200928256
> >>>  referenced 52868354048
> >>
> >>
> >>
> >>
> >> ---
> >>
> >> Regards,
> >> José Luis.
> >>
> >
>

Re: kernel 5.2 read time tree block corruption

[José Luis]

I also noticed the craziness of the previous dump. I cannot remember
the kernel running by this date but I use to install the latest stable
kernel on the Manjaro repositories (I'm an early adopter :P).
According the Manjaro forum release news they roll up version 4.19 by
these days so probably I was using kernel 4.19 or 4.18. Diggin on my
memory, maybe I could access that filesystem from a Windows 10 running
on another disk using the windows btrfs driver that could be the
origin of the problem.

I added a \s to the pattern you provided to avoid undesired inode information:
[manjaro@manjaro ~]$ sudo btrfs ins dump-tree -t 5 /dev/sdb2 | grep "(431 " -A 7
output --> https://pastebin.com/y3LzqNx6

Is there any magic command to repair my sdb2 filesystem? Or I have to
backup data and rebuild those filesystems?

Thanks Qu,
José Luis

El mar., 15 oct. 2019 a las 15:25, Qu Wenruo
(<quwenruo.btrfs@gmx.com>) escribió:
>
>
>
> On 2019/10/15 下午11:03, José Luis wrote:
> > Thanks for fast response Qu.
> >
> > I booted into a pendrive live system for the test cause I'm using the
> > involving fylesystem with kernel 4.19. This time when I mount
> >> [manjaro@manjaro ~]$ sudo mount /dev/sdb2 /mnt
> >> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
> > and in the dmesg:
> > [ +30,866472] BTRFS info (device sdb2): disk space caching is enabled
> > [  +0,017443] BTRFS info (device sdb2): enabling ssd optimizations
> > [  +0,000637] BTRFS critical (device sdb2): corrupt leaf: root=5
> > block=32145457152 slot=99, invalid key objectid: has
> > 18446744073709551605 expect 6 or [256, 18446744073709551360] or
> > 18446744073709551604
> > [  +0,000002] BTRFS error (device sdb2): block=32145457152 read time
> > tree block corruption detected
> > [  +0,000012] BTRFS warning (device sdb2): failed to read fs tree: -5
> > [  +0,061995] BTRFS error (device sdb2): open_ctree failed
> >
> > So I suppose you need dump output from the block 32145457152 so I pastebin that:
> > sudo btrfs ins dump-tree -b 32145457152 /dev/sdb2
> > output --> https://pastebin.com/ssB5HTn7
>
> The output is way crazier than I thought...
>
> I was only expecting some strange inode number, but what I got is
> completely ridiculous.
>
> From item 96, we are having completely impossible inodes.
> From FREE_INO to DATA_RELOC, even EXTENT_CSUM.
>
> All of these are impossible to exist in fs tree.
> The most strange thing is, they are all last modified in 2019-2-15.
>
> Anyway, the tree-checker is doing completely valid behavior for this
> case. The data is really ridiculous.
>
> Any history about the kernel used in that time?
> I see something only possible in Windows, any clue?
>
> >
> > Please provide the parameter to the grep redirection for: "btrfs ins
> > dump-tree -t 5 /dev/sdb2 | grep -A 7"
>
> My bad, the parameter is "(431"
>
> It will output all info about inode 431, so we can make sure what's
> going wrong.
>
> Thanks,
> Qu
> >
> > El mar., 15 oct. 2019 a las 14:38, Qu Wenruo
> > (<quwenruo.btrfs@gmx.com>) escribió:
> >>
> >>
> >>
> >> On 2019/10/15 下午8:24, Qu Wenruo wrote:
> >>>
> >>>
> >>> On 2019/10/15 下午6:15, José Luis wrote:
> >>>> Dear devs,
> >>>>
> >>>> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
> >>>> filesystems. I can work as intended on 4.19 which is an LTS version,
> >>>> previously using 5.1 but Manjaro removed it from their repositories.
> >>>>
> >>>> More info:
> >>>> · dmesg:
> >>>>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
> >>>>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
> >>>>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
> >>>
> >>> In fs tree, you are hitting a free space cache inode?
> >>> That doesn't sound good.
> >>>
> >>> Please provide the following dump:
> >>>
> >>> # btrfs ins dump-tree -b 30622793728 /dev/sdb2
> >>>
> >>> The output may contain filename, feel free to remove filenames.
> >>>
> >>>>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
> >>>>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
> >>>>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
> >>>>
> >>>>
> >>>>
> >>>> · sudo mount  /dev/sdb2 /mnt/
> >>>>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
> >>>>
> >>>> (cannot read superblock on /dev...)
> >>>>
> >>>> · sudo btrfs rescue super-recover /dev/sdb2
> >>>>> All supers are valid, no need to recover
> >>>>
> >>>>
> >>>> · sudo btrfs check /dev/sdb2
> >>>>> Opening filesystem to check...
> >>>>> Checking filesystem on /dev/sdb2
> >>>>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
> >>>>> [1/7] checking root items
> >>>>> [2/7] checking extents
> >>>>> [3/7] checking free space cache
> >>>>> cache and super generation don't match, space cache will be invalidated
> >>>>> [4/7] checking fs roots
> >>>>> root 5 inode 431 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 755 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
> >>>>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
> >>>>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
> >>>
> >>> Check is reporting the same problem of the inode.
> >>>
> >>> We need to make sure what's going wrong on that leaf, based on the
> >>> mentioned dump.
> >>>
> >>> For the csum missing error and bad file extent, it should be a big problem.
> >>
> >> s/should/should not/
> >>
> >>> if you want to make sure what's going wrong, please provide the
> >>> following dump:
> >>>
> >>> # btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7
> >>>
> >>> Also feel free the censor the filenames.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
> >>>>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
> >>>>> ERROR: errors found in fs roots
> >>>>> found 37167415296 bytes used, error(s) found
> >>>>> total csum bytes: 33793568
> >>>>> total tree bytes: 1676722176
> >>>>> total fs tree bytes: 1540243456
> >>>>> total extent tree bytes: 81510400
> >>>>> btree space waste bytes: 306327457
> >>>>> file data blocks allocated: 42200928256
> >>>>>  referenced 52868354048
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ---
> >>>>
> >>>> Regards,
> >>>> José Luis.
> >>>>
> >>>
> >>
>

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 9583 bytes --]



On 2019/10/16 上午1:55, José Luis wrote:
> I also noticed the craziness of the previous dump. I cannot remember
> the kernel running by this date but I use to install the latest stable
> kernel on the Manjaro repositories (I'm an early adopter :P).
> According the Manjaro forum release news they roll up version 4.19 by
> these days so probably I was using kernel 4.19 or 4.18. Diggin on my
> memory, maybe I could access that filesystem from a Windows 10 running
> on another disk using the windows btrfs driver that could be the
> origin of the problem.

That explains the problem why there are some strange windows related file.

And that also explains why kernel tree-checker isn't happy about that at
all.
Maybe Windows btrfs driver is using some strange inode number to do its
own work, but definitely not something friendly to upstream btrfs.

You may want to report the bug to windows btrfs developers.

> 
> I added a \s to the pattern you provided to avoid undesired inode information:
> [manjaro@manjaro ~]$ sudo btrfs ins dump-tree -t 5 /dev/sdb2 | grep "(431 " -A 7
> output --> https://pastebin.com/y3LzqNx6

I see no obvious problem. Maybe some compressed data extent doesn't have
csum, then btrfs check reports it as bad file extent.

Original mode doesn't report info as detailed as possible.
But anyway, it shouldn't be a big problem.

If you're not confident about it, you can try to defrag those inodes, it
should rewrite them and populate the csum.

> 
> Is there any magic command to repair my sdb2 filesystem? Or I have to
> backup data and rebuild those filesystems?

In fact it's not that hard to repair, just remove the offending craziness.

btrfs-corrupt-block should provide the ability to delete items.
It a tool included in btrfs-progs, but not provided in btrfs-progs
packages. You may need to compile it from source code.

In your case, you need quite some calls to delete all the bad inodes:

/* FREE_INO INODE_ITEM 0 */
# ./btrfs-corrupt-block -d 18446744073709551604,1,0 /dev/sdb2

/* FREE_SPACE UNTYPED 0 */
# ./btrfs-corrupt-block -d 18446744073709551605,0,0 /dev/sdb2

...

And so on. You need to parse the key output to numeric value and pass it
to btrfs-corrupt-block, until all finished.

If it's too slow, I could add a new hack into btrfs-corrupt-block to
delete them in a batch.

Thanks,
Qu
> 
> Thanks Qu,
> José Luis
> 
> El mar., 15 oct. 2019 a las 15:25, Qu Wenruo
> (<quwenruo.btrfs@gmx.com>) escribió:
>>
>>
>>
>> On 2019/10/15 下午11:03, José Luis wrote:
>>> Thanks for fast response Qu.
>>>
>>> I booted into a pendrive live system for the test cause I'm using the
>>> involving fylesystem with kernel 4.19. This time when I mount
>>>> [manjaro@manjaro ~]$ sudo mount /dev/sdb2 /mnt
>>>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
>>> and in the dmesg:
>>> [ +30,866472] BTRFS info (device sdb2): disk space caching is enabled
>>> [  +0,017443] BTRFS info (device sdb2): enabling ssd optimizations
>>> [  +0,000637] BTRFS critical (device sdb2): corrupt leaf: root=5
>>> block=32145457152 slot=99, invalid key objectid: has
>>> 18446744073709551605 expect 6 or [256, 18446744073709551360] or
>>> 18446744073709551604
>>> [  +0,000002] BTRFS error (device sdb2): block=32145457152 read time
>>> tree block corruption detected
>>> [  +0,000012] BTRFS warning (device sdb2): failed to read fs tree: -5
>>> [  +0,061995] BTRFS error (device sdb2): open_ctree failed
>>>
>>> So I suppose you need dump output from the block 32145457152 so I pastebin that:
>>> sudo btrfs ins dump-tree -b 32145457152 /dev/sdb2
>>> output --> https://pastebin.com/ssB5HTn7
>>
>> The output is way crazier than I thought...
>>
>> I was only expecting some strange inode number, but what I got is
>> completely ridiculous.
>>
>> From item 96, we are having completely impossible inodes.
>> From FREE_INO to DATA_RELOC, even EXTENT_CSUM.
>>
>> All of these are impossible to exist in fs tree.
>> The most strange thing is, they are all last modified in 2019-2-15.
>>
>> Anyway, the tree-checker is doing completely valid behavior for this
>> case. The data is really ridiculous.
>>
>> Any history about the kernel used in that time?
>> I see something only possible in Windows, any clue?
>>
>>>
>>> Please provide the parameter to the grep redirection for: "btrfs ins
>>> dump-tree -t 5 /dev/sdb2 | grep -A 7"
>>
>> My bad, the parameter is "(431"
>>
>> It will output all info about inode 431, so we can make sure what's
>> going wrong.
>>
>> Thanks,
>> Qu
>>>
>>> El mar., 15 oct. 2019 a las 14:38, Qu Wenruo
>>> (<quwenruo.btrfs@gmx.com>) escribió:
>>>>
>>>>
>>>>
>>>> On 2019/10/15 下午8:24, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2019/10/15 下午6:15, José Luis wrote:
>>>>>> Dear devs,
>>>>>>
>>>>>> I cannot use kernel >= 5.2, They cannot mount sdb2 nor sb3 both btrfs
>>>>>> filesystems. I can work as intended on 4.19 which is an LTS version,
>>>>>> previously using 5.1 but Manjaro removed it from their repositories.
>>>>>>
>>>>>> More info:
>>>>>> · dmesg:
>>>>>>> [oct15 13:47] BTRFS info (device sdb2): disk space caching is enabled
>>>>>>> [  +0,009974] BTRFS info (device sdb2): enabling ssd optimizations
>>>>>>> [  +0,000481] BTRFS critical (device sdb2): corrupt leaf: root=5 block=30622793728 slot=115, invalid key objectid: has 18446744073709551605 expect 6 or [256, 18446744073709551360] or 18446744073709551604
>>>>>
>>>>> In fs tree, you are hitting a free space cache inode?
>>>>> That doesn't sound good.
>>>>>
>>>>> Please provide the following dump:
>>>>>
>>>>> # btrfs ins dump-tree -b 30622793728 /dev/sdb2
>>>>>
>>>>> The output may contain filename, feel free to remove filenames.
>>>>>
>>>>>>> [  +0,000002] BTRFS error (device sdb2): block=30622793728 read time tree block corruption detected
>>>>>>> [  +0,000021] BTRFS warning (device sdb2): failed to read fs tree: -5
>>>>>>> [  +0,044643] BTRFS error (device sdb2): open_ctree failed
>>>>>>
>>>>>>
>>>>>>
>>>>>> · sudo mount  /dev/sdb2 /mnt/
>>>>>>> mount: /mnt: no se puede leer el superbloque en /dev/sdb2.
>>>>>>
>>>>>> (cannot read superblock on /dev...)
>>>>>>
>>>>>> · sudo btrfs rescue super-recover /dev/sdb2
>>>>>>> All supers are valid, no need to recover
>>>>>>
>>>>>>
>>>>>> · sudo btrfs check /dev/sdb2
>>>>>>> Opening filesystem to check...
>>>>>>> Checking filesystem on /dev/sdb2
>>>>>>> UUID: ff559c37-bc38-491c-9edc-fa6bb0874942
>>>>>>> [1/7] checking root items
>>>>>>> [2/7] checking extents
>>>>>>> [3/7] checking free space cache
>>>>>>> cache and super generation don't match, space cache will be invalidated
>>>>>>> [4/7] checking fs roots
>>>>>>> root 5 inode 431 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 755 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 2379 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 11721 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 12211 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 15368 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 35329 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 960427 errors 1040, bad file extent, some csum missing
>>>>>>> root 5 inode 18446744073709551605 errors 2001, no inode item, link count wrong
>>>>>>>         unresolved ref dir 256 index 0 namelen 12 name $RECYCLE.BIN filetype 2 errors 6, no dir index, no inode ref
>>>>>
>>>>> Check is reporting the same problem of the inode.
>>>>>
>>>>> We need to make sure what's going wrong on that leaf, based on the
>>>>> mentioned dump.
>>>>>
>>>>> For the csum missing error and bad file extent, it should be a big problem.
>>>>
>>>> s/should/should not/
>>>>
>>>>> if you want to make sure what's going wrong, please provide the
>>>>> following dump:
>>>>>
>>>>> # btrfs ins dump-tree -t 5 /dev/sdb2 | grep -A 7
>>>>>
>>>>> Also feel free the censor the filenames.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>> root 388 inode 1245 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 1288 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 1292 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 1313 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 11870 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 68126 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88051 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88255 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88455 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88588 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88784 errors 1040, bad file extent, some csum missing
>>>>>>> root 388 inode 88916 errors 1040, bad file extent, some csum missing
>>>>>>> ERROR: errors found in fs roots
>>>>>>> found 37167415296 bytes used, error(s) found
>>>>>>> total csum bytes: 33793568
>>>>>>> total tree bytes: 1676722176
>>>>>>> total fs tree bytes: 1540243456
>>>>>>> total extent tree bytes: 81510400
>>>>>>> btree space waste bytes: 306327457
>>>>>>> file data blocks allocated: 42200928256
>>>>>>>  referenced 52868354048
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> Regards,
>>>>>> José Luis.
>>>>>>
>>>>>
>>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: kernel 5.2 read time tree block corruption

[Nikolay Borisov]

On 16.10.19 г. 3:43 ч., Qu Wenruo wrote:
> 
> 
> On 2019/10/16 上午1:55, José Luis wrote:
>> I also noticed the craziness of the previous dump. I cannot remember
>> the kernel running by this date but I use to install the latest stable
>> kernel on the Manjaro repositories (I'm an early adopter :P).
>> According the Manjaro forum release news they roll up version 4.19 by
>> these days so probably I was using kernel 4.19 or 4.18. Diggin on my
>> memory, maybe I could access that filesystem from a Windows 10 running
>> on another disk using the windows btrfs driver that could be the
>> origin of the problem.
> 
> That explains the problem why there are some strange windows related file.
> 
> And that also explains why kernel tree-checker isn't happy about that at
> all.
> Maybe Windows btrfs driver is using some strange inode number to do its
> own work, but definitely not something friendly to upstream btrfs.
> 
> You may want to report the bug to windows btrfs developers.
> 
>>
>> I added a \s to the pattern you provided to avoid undesired inode information:
>> [manjaro@manjaro ~]$ sudo btrfs ins dump-tree -t 5 /dev/sdb2 | grep "(431 " -A 7
>> output --> https://pastebin.com/y3LzqNx6
> 
> I see no obvious problem. Maybe some compressed data extent doesn't have
> csum, then btrfs check reports it as bad file extent.
> 
> Original mode doesn't report info as detailed as possible.
> But anyway, it shouldn't be a big problem.
> 
> If you're not confident about it, you can try to defrag those inodes, it
> should rewrite them and populate the csum.
> 
>>
>> Is there any magic command to repair my sdb2 filesystem? Or I have to
>> backup data and rebuild those filesystems?
> 
> In fact it's not that hard to repair, just remove the offending craziness.
> 
> btrfs-corrupt-block should provide the ability to delete items.
> It a tool included in btrfs-progs, but not provided in btrfs-progs
> packages. You may need to compile it from source code.
> 
> In your case, you need quite some calls to delete all the bad inodes:
> 
> /* FREE_INO INODE_ITEM 0 */
> # ./btrfs-corrupt-block -d 18446744073709551604,1,0 /dev/sdb2
> 
> /* FREE_SPACE UNTYPED 0 */
> # ./btrfs-corrupt-block -d 18446744073709551605,0,0 /dev/sdb2
> 
> ...
> 
> And so on. You need to parse the key output to numeric value and pass it
> to btrfs-corrupt-block, until all finished.
> 
> If it's too slow, I could add a new hack into btrfs-corrupt-block to
> delete them in a batch.

Please don't. btrfs-corrupt-block is supposed to aid development not fix
user problems. If it can fix them, by accident, then OK but it shouldn't
be cluttered with code used in some _very_ specific cases.

You can provide this code on your own accord but let's not merge it into
the upstream btrfs-corrupt-block source code.

<snip>

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

On 2019/10/16 下午3:42, Nikolay Borisov wrote:
>
>
> On 16.10.19 г. 3:43 ч., Qu Wenruo wrote:
>>
>>
>> On 2019/10/16 上午1:55, José Luis wrote:
>>> I also noticed the craziness of the previous dump. I cannot remember
>>> the kernel running by this date but I use to install the latest stable
>>> kernel on the Manjaro repositories (I'm an early adopter :P).
>>> According the Manjaro forum release news they roll up version 4.19 by
>>> these days so probably I was using kernel 4.19 or 4.18. Diggin on my
>>> memory, maybe I could access that filesystem from a Windows 10 running
>>> on another disk using the windows btrfs driver that could be the
>>> origin of the problem.
>>
>> That explains the problem why there are some strange windows related file.
>>
>> And that also explains why kernel tree-checker isn't happy about that at
>> all.
>> Maybe Windows btrfs driver is using some strange inode number to do its
>> own work, but definitely not something friendly to upstream btrfs.
>>
>> You may want to report the bug to windows btrfs developers.
>>
>>>
>>> I added a \s to the pattern you provided to avoid undesired inode information:
>>> [manjaro@manjaro ~]$ sudo btrfs ins dump-tree -t 5 /dev/sdb2 | grep "(431 " -A 7
>>> output --> https://pastebin.com/y3LzqNx6
>>
>> I see no obvious problem. Maybe some compressed data extent doesn't have
>> csum, then btrfs check reports it as bad file extent.
>>
>> Original mode doesn't report info as detailed as possible.
>> But anyway, it shouldn't be a big problem.
>>
>> If you're not confident about it, you can try to defrag those inodes, it
>> should rewrite them and populate the csum.
>>
>>>
>>> Is there any magic command to repair my sdb2 filesystem? Or I have to
>>> backup data and rebuild those filesystems?
>>
>> In fact it's not that hard to repair, just remove the offending craziness.
>>
>> btrfs-corrupt-block should provide the ability to delete items.
>> It a tool included in btrfs-progs, but not provided in btrfs-progs
>> packages. You may need to compile it from source code.
>>
>> In your case, you need quite some calls to delete all the bad inodes:
>>
>> /* FREE_INO INODE_ITEM 0 */
>> # ./btrfs-corrupt-block -d 18446744073709551604,1,0 /dev/sdb2
>>
>> /* FREE_SPACE UNTYPED 0 */
>> # ./btrfs-corrupt-block -d 18446744073709551605,0,0 /dev/sdb2
>>
>> ...
>>
>> And so on. You need to parse the key output to numeric value and pass it
>> to btrfs-corrupt-block, until all finished.
>>
>> If it's too slow, I could add a new hack into btrfs-corrupt-block to
>> delete them in a batch.
>
> Please don't. btrfs-corrupt-block is supposed to aid development not fix
> user problems. If it can fix them, by accident, then OK but it shouldn't
> be cluttered with code used in some _very_ specific cases.
>
> You can provide this code on your own accord but let's not merge it into
> the upstream btrfs-corrupt-block source code.

Of course, just as my usual dirty fixes (a special branch for the user
to do, never intended to upstream).

Thanks,
Qu

>
> <snip>
>

Re: kernel 5.2 read time tree block corruption

[Roman Mamedov]

On Wed, 16 Oct 2019 15:45:54 +0800
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> dirty fixes (a special branch for the user
> to do, never intended to upstream).

Still it would be nice to get a btrfs check mode which would include trying
destructive actions, as in accepting the loss of some part of user data
(outright delete corrupt blocks, etc), just to restore the filesystem itself to
a fully correct and mountable state.

Sometimes there are cases when it is inconvenient to recreate the entire FS
(remember seeing 40TB mentioned recently), just to fix a few "parent transid
failed", not even many transactions apart. And the data itself is often backed
up, so restoring only the missing part from the backup would be much easier.

-- 
With respect,
Roman

Re: kernel 5.2 read time tree block corruption

[Qu Wenruo]

On 2019/10/16 下午4:12, Roman Mamedov wrote:
> On Wed, 16 Oct 2019 15:45:54 +0800
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>> dirty fixes (a special branch for the user
>> to do, never intended to upstream).
>
> Still it would be nice to get a btrfs check mode which would include trying
> destructive actions, as in accepting the loss of some part of user data
> (outright delete corrupt blocks, etc), just to restore the filesystem itself to
> a fully correct and mountable state.

It's already there.

The problem is, that mode only works for fs tree, and it can only handle
cases like full corrupted (not transid, but the child node/leaf can't be
read out completely).


>
> Sometimes there are cases when it is inconvenient to recreate the entire FS
> (remember seeing 40TB mentioned recently), just to fix a few "parent transid
> failed", not even many transactions apart.

That transid problem is the biggest and the most complext problem.
Transid problem can lead to futher corruption, like a tree block is
completely incorrect in the context of parent tree.
E.g. a csum tree block points to a fs tree block.

So we can't easily treat transid by easily ignore them.

But your idea is pretty god, maybe we should tread transid error as bad
csum, and completely ignore the child, do the regular corrupted
leaf/node repair.

Thanks,
Qu

> And the data itself is often backed
> up, so restoring only the missing part from the backup would be much easier.
>