Re: Massive filesystem corruption since kernel 5.2 (ARCH)

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2019/8/8 下午5:55, Swâmi Petaramesh wrote:
> Hi Qu,
>
> On 8/8/19 10:46 AM, Qu Wenruo wrote:
>> Follow up questions about the corruption.
>>
>> Is there enough free space (not only unallocated, but allocated bg) for
>> metadata?
>>
>> As further digging into the case, it looks like btrfs is even harder to
>> get corrupted for tree blocks.
>>
>> If we have enough metadata free space, we will try to allocate tree
>> blocks at bytenr sequence, without reusing old bytenr until there is not
>> enough space or hit the end of the block group.
>>
>> This means, even we have something wrong implementing barrier, we still
>> won't write new data to old tree blocks (even several trans ago).
>
>
> It's kind of hard for me to say if the 2 filesystems that got corrupt
> lacked allocated metadata space at any time, and now both filesystems
> have been reformatted, so I cannot tell.
>
> What I can be 100% sure is that I never got any “No space left on
> device” ENOSPC on any of them.

No need to hit ENOSPC, although it needs extra info to get the metadata
bg usage to determine, so I didn't expect it to be easy to get.

>
> *BUT* the SSD on which the machine runs may have run close to full as I
> had copied a bunch of ISOs on it shortly before upgrading packages - and
> kernel.
>
> However the upgrade went seemingly good and I didn't see no ENOSPC at
> any time.
>
>
> On the external HD that went corrupt as well, I'm pretty sure it
> happened as follows :
>
> - I started a full backup onto it in an emergency ;
>
> - I asked myself « Will I have enough space » and checked with “df”.
>
> - There were still several dozens of GBs free but not enough for a full
> system backup. I cannot tell if these had been allocated or not in the past.
>
> - Noticing that I would miss HD space (but far before it actually
> happened) I deleted a high number of snapshots from the HD.
>
> - I thus assume that the deletion of snapshots would have freed a good
> amount of data AND metadata space.
>
> So the situation of the external HD was that a full backup was in
> progress and a vast number of snapshots have been deleted meanwhile.
>
> After that the FS got corrupt at some point.
>
>
> For the internal SSD, it looks like the kernel upgrade went good and the
> machine rebooted OK, then midnight came and with it probably the cron
> task that performs “snapper” timeline snapshots deletion.
>
>
> Then the machine was turned off and rebooted next day, and by that time
> the FS was corrupt.
>
>
> So I strongly suspect the issue has something to do with snapshots
> deletion, but I cannot tell more.

I was also working on that in recent days, hasn't yet got any clue. (In
fact, just find btrfs harder to get corrupted if there is enough
metadata space).
But will definitely continue digging.

>
>
> It may be worth noticing that the machine has been running a lot since I
> reverted back to kernel 5.1 and reformatted the filesystems, and that no
> corruption has occurred since, even though I performed quite a lot of
> backups on the external HD after it has been reformatted.
>
> Everything is in the exact same setup as before, except for the kernel.
>
> So I would definitely exclude an hardware problem on the machine : it's
> now running fine as it ever did.
>
> I plan to retry upgrading to Arch kernel 5.2 in the coming weeks after
> having performed a full disk binary clone in case it happens again.
>
> (However I've seen that Arch has released 3-4 kernel 5.2 package updates
> since, so it won't be the exact same kernel by the time I test again).

No problem, not that many fixes get backport, none of them are really
high priority so I'd say it would not make much difference.

>
> I will be on vacation until August, 20, so I cannot perform this test
> before I'm back.
>
> But I'll be glad to help if I can and thank you very much for your help
> with this issue.

My pleasure, if we could finally pin down the cause, it would be a great
improvement for btrfs.

Thanks,
Qu
>
> Best regards.
>
> ॐ
>