Corruption of newly created files after degraded mount of a raid1 filesystem

Corruption of newly created files after degraded mount of a raid1 filesystem

[Andrei Purdea]

Hi everyone,

I have a 2xSSD raid1 btrfs filesystem, 213GiB, used as a root
filesystem for a debian install.
I had to mount this filesystem in degraded mode because I needed to
temporarily steal a SATA port, and I was planning to rebuild the raid
after I was done.

But all hell broke loose right after the first boot with rootflags=degraded.

Here is a demo to show how the corruption happens with some simple commands:
https://pastebin.com/ev2hd5Lm
The corruption sometimes happens almost immediately, and sometimes it
takes tens of seconds to happen.

Note: /boot is an ext3 filesystem.
If I run "vbindiff /boot/bla ./bla", I see the first 0x1000 bytes of
the file have been zeroed out,
and the remaining bytes remain untouched.
If I run "sync; echo 3 > /proc/sys/vm/drop_caches", and run vbindiff
after that, then the entire file is all zeros. (but it keeps the same
size as the original file)
If I unmount and re-mount the file "bla" becomes a 0-length file.

During all of this there are no btrfs csum errors in dmesg.

I reproduced the problem both by booting the damanged filesystem on
its original machine using kernel 4.19.0-4-amd64 (official kernel in
debian repo), and then I also tried moving the SSD to a different
computer, this one is running kernel "5.1.0-rc5+", compiled fresh from
git about a week ago. The behaviour is exactly the same, and on the
second machine I didn't boot the damaged filesystem as root, so the
only thing accessing this filesystem was me running my cp, diff,
vbindiff commands.

The filesystem was mounted as such:
/dev/sdb2 on / type btrfs
(rw,noatime,nodiratime,degraded,compress=lzo,ssd,discard,space_cache,subvolid=5,subvol=/)

But I also tried without "compress=", and without "discard", and the
behavior remains the same.

Somebody on IRC asked me to run "filefrag -v" on the file, so here's
it's output: https://pastebin.com/4pgRXu4N

And here's the output of "btrfs check --readonly":
https://pastebin.com/dEHJ55ui

Could any of the check errors explain the problems I am seeing? Or are
they just symptoms too?

I am fairly confident that the problem was caused by me mounting the
filesystem degraded. Before the first mount the system was working
properly, and after the first degraded mount I couldn't do basic
things: I couldn't run apt-get update (it is complaining about
checksum errors, non-btrfs checksum errors), and google-chrome is
complaining that it can't access it's cache. The system is effectively
unusable. The only suggestion that there may have been anything wrong
from before is that this is the only btrfs filesystem that I have that
reported non-zero values for "btrfs device stats": (the below values
were non-zero even before the degraded mount)

[/dev/sda2].write_io_errs 161
[/dev/sda2].read_io_errs 0
[/dev/sda2].flush_io_errs 0
[/dev/sda2].corruption_errs 1863
[/dev/sda2].generation_errs 0
[/dev/sdb2].write_io_errs 2230
[/dev/sdb2].read_io_errs 51
[/dev/sdb2].flush_io_errs 0
[/dev/sdb2].corruption_errs 37
[/dev/sdb2].generation_errs 0

I chucked this up to the fact that these are SSDs, and sometimes
sectors go bad on SSDs. The couple btrfs scrub operations I ran over
the lifetime of the array never turned up with any errors. I still
don't think this is related to the corruption, but I am showing this
in the interest of full disclosure.

Here are some more outputs of device usage, filesystem usage,
filesystem show, and filesystem df:
https://pastebin.com/hD3Me4qS

The only error message that I'm ever getting is if I try to copy the
same file over itself like this:
https://pastebin.com/tRic42Dx
and no errors when I create new files.

I realize that I have 0 unallocated space, but I have 9GiB free on
data and 1.7GiB free on metadata, I don't think I should be getting
these errors...

Anyway, any ideas on how I can continue to debug and explain this?

Thanks,
Andrei

Re: Corruption of newly created files after degraded mount of a raid1 filesystem

[Andrei Purdea]

> [/dev/sda2].write_io_errs 161
> [/dev/sda2].read_io_errs 0
> [/dev/sda2].flush_io_errs 0
> [/dev/sda2].corruption_errs 1863
> [/dev/sda2].generation_errs 0
> [/dev/sdb2].write_io_errs 2230
> [/dev/sdb2].read_io_errs 51
> [/dev/sdb2].flush_io_errs 0
> [/dev/sdb2].corruption_errs 37
> [/dev/sdb2].generation_errs 0

I just found some of my older notes. These numbers have not increased
since 2017.
I think they were caused by a bad sata cable when I first built the machine.

Thanks,
Andrei