Re: read time tree block corruption detected

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2021/5/27 上午7:03, Gervais, Francois wrote:
> I had to re-send this message as I forgot to send it as plain text earlier today. I apologize if some people receive it twice.
>
>>>> - Anything you would recommend as of configuration of the device?
>>>
>>> You can test using dm-logwrites, which really logs every writes and
>>> replay it.
>>> By using dm-logwrites, you can emulate powerloss for each write operation.
>
> I'm in the process of putting together a test scenario using dm-log-writes on our
> device but I have a few questions.
>
> 1. What I'm thinking of doing is do the firmware operation that the device was
> doing when we got the corruption but this time using dm-log-writes and then
> replay the log entry per entry.
>
> Will tree-checker catch to issue just by replaying or do I need to do an additional
> step for every log entry replay? (btrfs check? btrfs scrub?)

Normally we only care about the filesystem consistency at FUA/FLUSH.

Doing extra check at each replay won't hurt, but it would be very slow
though.

And for the tool to catch the problem, I'm afraid you have to use both
kernel tree-checker (aka mount it), and btrfs-check to catch all problems.

Tree-check can check all involved metadata, while btrfs-check doesn't
check log tree.

>
> 2. From what I understand, our corruption of the log-tree can either be software
> which would have requested a corrupted entry to be written to disk or hardware
> which would have corrupted the entry when trying to write it to disk.

I don't believe it's hardware, one point to notice is, all tree block
written back to disk has checksum.

If it's hardware problem leading to some data corruption on-disk, its
csum should not match at all.

Thus I believe this bug is mostly from btrfs.

Filipe is doing a lot of fixes related to log tree, and some of them are
related to item size overflow.
But if you can reproduce the problem reliably, then it would be a great
help to pin down the cause.

>
> Debugging with dm-log-writes would not catch a corruption by the hardware if due
> to a one-off glitch or something right?

Although I don't believe it's hardware problem, but just to be clear,
no, it won't detect any hardware problem.

Dm-log-writes is only to verify the filesystem layer is doing correct work.

Thanks,
Qu

>
>>>>     - Should we run a newer kernel than our current v5.4?
>>>
>>> Definitely. In fact my fuzzy memory points me to some fix, but I can't
>>> remember exactly which fix.
>>>
>>>>     - Any debug you think would be useful to enable or add?
>>>>
>>>
>>> Tree-checker, which is already enabled by default (in fact no way to
>>> disable) in newer kernels.
>>>
>>> Thanks,
>>> Qu
>>
>> Thank you very much,
>>
>> I will report here if/when we get more information on this issue.