Re: Massive filesystem corruption since kernel 5.2 (ARCH)

["Swâmi Petaramesh" <swami@petaramesh.org>]

Hi Qu,

On 8/8/19 10:46 AM, Qu Wenruo wrote:
> Follow up questions about the corruption.
>
> Is there enough free space (not only unallocated, but allocated bg) for
> metadata?
>
> As further digging into the case, it looks like btrfs is even harder to
> get corrupted for tree blocks.
>
> If we have enough metadata free space, we will try to allocate tree
> blocks at bytenr sequence, without reusing old bytenr until there is not
> enough space or hit the end of the block group.
>
> This means, even we have something wrong implementing barrier, we still
> won't write new data to old tree blocks (even several trans ago).


It's kind of hard for me to say if the 2 filesystems that got corrupt
lacked allocated metadata space at any time, and now both filesystems
have been reformatted, so I cannot tell.

What I can be 100% sure is that I never got any “No space left on
device” ENOSPC on any of them.

*BUT* the SSD on which the machine runs may have run close to full as I
had copied a bunch of ISOs on it shortly before upgrading packages - and
kernel.

However the upgrade went seemingly good and I didn't see no ENOSPC at
any time.


On the external HD that went corrupt as well, I'm pretty sure it
happened as follows :

- I started a full backup onto it in an emergency ;

- I asked myself « Will I have enough space » and checked with “df”.

- There were still several dozens of GBs free but not enough for a full
system backup. I cannot tell if these had been allocated or not in the past.

- Noticing that I would miss HD space (but far before it actually
happened) I deleted a high number of snapshots from the HD.

- I thus assume that the deletion of snapshots would have freed a good
amount of data AND metadata space.

So the situation of the external HD was that a full backup was in
progress and a vast number of snapshots have been deleted meanwhile.

After that the FS got corrupt at some point.


For the internal SSD, it looks like the kernel upgrade went good and the
machine rebooted OK, then midnight came and with it probably the cron
task that performs “snapper” timeline snapshots deletion.


Then the machine was turned off and rebooted next day, and by that time
the FS was corrupt.


So I strongly suspect the issue has something to do with snapshots
deletion, but I cannot tell more.


It may be worth noticing that the machine has been running a lot since I
reverted back to kernel 5.1 and reformatted the filesystems, and that no
corruption has occurred since, even though I performed quite a lot of
backups on the external HD after it has been reformatted.

Everything is in the exact same setup as before, except for the kernel.

So I would definitely exclude an hardware problem on the machine : it's
now running fine as it ever did.

I plan to retry upgrading to Arch kernel 5.2 in the coming weeks after
having performed a full disk binary clone in case it happens again.

(However I've seen that Arch has released 3-4 kernel 5.2 package updates
since, so it won't be the exact same kernel by the time I test again).

I will be on vacation until August, 20, so I cannot perform this test
before I'm back.

But I'll be glad to help if I can and thank you very much for your help
with this issue.

Best regards.

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E