Re: BTRFS error (device dm-0): parent transid verify failed on 1382301696 wanted 262166 found 22

[Qu Wenruo <wqu@suse.com>]

On 2022/3/1 10:14, Christoph Anton Mitterer wrote:
> Hey.
> 
> memtest[0] showed, that in fact memory is damaged in some higher region... as you've guessed, its always a single but flip.

Btrfs is now a pretty good memtest tool too :)

> 
> 
> 
> Am 1. März 2022 01:19:12 MEZ schrieb Qu Wenruo <quwenruo.btrfs@gmx.com>:
>>> It would be interesting to see how much is actually affected,...
>>> shouldn't it be possible to run something like dd_rescue on it? I mean
>>> I'd probably get thousands of csum errors, but in the end it should
>>> show me how much of the file is gone.
>>
>> As said, no real file is damaged.
>> It's just we can get csum.
> 
> Sure. I've had understood that. What I've meant was to find out how much of the file (or, if more were affected, which files) was not guaranteed to be integrity protected, because its csum data is broken.
> 
> 
> 
>>> So would expect that the corruption or bit-flip would need to have
>>> happened at some point after it was first sent/received?
>>
>> I guess the corrupted csum tree block happen at that time.
> 
> It's still a bit strange, though, because I most likely had run a scrub since then,  and no errors were found...
> 
> But in principle, scrub should notice these corruptions in the csum tree, shouldn't it?

In theory, it should, especially for csum tree with skinny metadata feature.

In that case we should do a tree search and locate that tree block.

But there is a catch, if the tree block is still cached in memory, we 
may not do full comprehensive check on it and thus it may be a hole 
allowing it to sneak in.

Anyway, I need more investigate to be sure on how this happened without 
triggering scrub, and find a way to make btrfs a more robust memtester :)

Thanks,
Qu
> 
> 
>> And fortunately that range doesn't get much utilized thus later
>> read/write won't get interrupted by that corrupted tree block.
> 
> That I don't understand. You mean the csum tree isn't read/written in that region (i.e. not unless the associated files are read)... and that's why it went so long unnoticed?
> 
> 
> 
>>> Shouldn't I be able to find out simply by copying away each file (like
>>> what I did during yesterday's backup)?
>>
>> Yep, that's possible.
>>
>>> Or something like tar -cf /dev/null /
>>>
>>> Every file that tar cannot read should give an error, and I'd see which
>>> are affected?
>>
>> That's also a way.
> 
> Ok... if both works to find out files are affected (in the sense that they cannot be verified because the csum is broken... and thus may or may not be valid)... then I guess that's the easiest way for me to recover.
> 
> 
> 
>>>> 1) Which logical bytenr range is in that csum tree block
>>>>
>>>> 2) Which files owns the logical bytenr range.
>>>
>>> Is this possible already with standard tools?
>>
>> We have tools for 2), "btrfs ins logical-resolve" to search for all the
>> files owning a logical bytenr range.
> 
> So in principle,  since my tar yesterday brought no further errors,  the should result in only that one file.
> 
> 
>>> If I'd delete the affected file(s) would btrfs simply clear the broken
>>> csum block?
>>
>> Nope. That generation mismatch would prevent btrfs to do any
>> modification including CoW the tree block to a new location.
> 
> Ah, OK.
> 
> 
> Thanks,
> Chris.
> 
> [0] For those who haven't seen yet,  there's pcmemtest (https://github.com/martinwhitaker/pcmemtest) which is a fork off (or based upon memtest86+)... but with UEFI support, which memtest86+ cannot be used with.
>