Re: Btrfs filesystem trashed after OOM scenario

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2019-09-25 00:25, Nick Bowler wrote:
> On Tue, Sep 24, 2019, 18:34 Chris Murphy, <lists@colorremedies.com> wrote:
>> On Tue, Sep 24, 2019 at 4:04 PM Nick Bowler <nbowler@draconx.ca> wrote:
>>> - Running Linux 5.2.14, I pushed this system to OOM; the oom killer
>>> ran and killed some userspace tasks.  At this point many of the
>>> remaining tasks were stuck in uninterruptible sleeps.  Not really
>>> worried, I turned the machine off and on again to just get everything
>>> back to normal.  But I guess now that everything had gone horribly
>>> wrong already at this point...
>>
>> Yeah the kernel oomkiller is pretty much only about kernel
>> preservation, not user space preservation.
> 
> Indeed I am not bothered at all by needing to turn it off and on again
> in this situation.  But filesystems being completely trashed is
> another matter...
> 
>>> - Upon reboot, the system boots OK but now btrfs is throwing zillions
>>> of checksum errors.  After some time the filesystem is remounted
>>> readonly and I lose the ability to interact with the system at all, so
>>> it gets powered off.
>>>
>>> - Now the filesystem is unmountable.
>>
>> The transid errors look like they might be caused by the 5.2 regression
>>
>> https://lore.kernel.org/linux-btrfs/20190911145542.1125-1-fdmanana@kernel.org/T/#u
>>
>> Fixed since 5.2.15 and 5.3.0.
> 
> Yikes, so my decision to update the latest kernel two weeks ago
> perhaps was a very bad one.  Should've stuck with 4.19.y I guess.
> 
>> So if you're willing to blow shit up again, you can try to reproduce
>> with one of those.
> 
> Well I could try but it sounds like this might be hard to reproduce...
> 
>> I was also doing oomkiller blow shit up tests a few weeks ago with
>> these same problem kernels and never hit this bug, or any others. I
>> also had to do a LOT of force power offs because the system just
>> became totally wedged in and I had no way of estimating how long it
>> would be for recovery so after 30 minutes I hit the power button. Many
>> times. Zero corruptions. That's with a single Samsung 840 EVO in a
>> laptop relegated to such testing.
> 
> Just a thought... the system was alive but I was able to briefly
> inspect the situation and notice that tasks were blocked and
> unkillable... until my shell hung too and then I was hosed.  But I
> didn't hit the power button but rather rebooted with sysrq+e, sysrq+u,
> sysrq+b.  Not sure if that makes a difference.
Not sure if this mattered, but as a general rule, unless you're dealing 
with an issue with the disk, you should always issue sysrq+s and wait a 
few seconds (or until the message that all filesystems have been synced 
shows up if you're on the console and can see kernel messages) before 
issuing a sysrq+u.  Remounting all filesystems read-only through sysrq+u 
does not reliably flush caches before forcing everything read-only.
> 
>> Might be a different bug. Not sure. But also, this is with
>>
>>> [  347.551595] CPU: 3 PID: 1143 Comm: mount Not tainted 4.19.34-1-lts #1
>>
>> So I don't know how an older kernel will report on the problem caused
>> by the 5.2 bug.
> 
> This is the kernel from systemrescuecd.  I can try taking a disk image
> and mounting on another machine with a newer linux version.
> 
> Thanks,
>    Nick
>