Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Chris Murphy <lists@colorremedies.com>
To: Richard Weinberger <richard@nod.at>
Cc: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>,
	linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: Decoding "unable to fixup (regular)" errors
Date: Wed, 13 Nov 2019 18:17:25 +0000
Message-ID: <CAJCQCtTeqxxO=xAw5ogLORoNCvx7E7n0NQ4uF725bYX6vab25Q@mail.gmail.com> (raw)
In-Reply-To: <1389491920.79042.1573293642654.JavaMail.zimbra@nod.at>

On Sat, Nov 9, 2019 at 10:00 AM Richard Weinberger <richard@nod.at> wrote:
>
> ----- Ursprüngliche Mail -----
> > Von: "Zygo Blaxell" <ce3g8jdj@umail.furryterror.org>
> > An: "richard" <richard@nod.at>
> > CC: "linux-btrfs" <linux-btrfs@vger.kernel.org>
> > Gesendet: Samstag, 9. November 2019 00:39:33
> > Betreff: Re: Decoding "unable to fixup (regular)" errors
>
> > On Fri, Nov 08, 2019 at 11:31:22PM +0100, Richard Weinberger wrote:
> >> ----- Ursprüngliche Mail -----
> >> > Von: "Zygo Blaxell" <ce3g8jdj@umail.furryterror.org>
> >> > An: "richard" <richard@nod.at>
> >> > CC: "linux-btrfs" <linux-btrfs@vger.kernel.org>
> >> > Gesendet: Freitag, 8. November 2019 23:25:57
> >> > Betreff: Re: Decoding "unable to fixup (regular)" errors
> >>
> >> > On Fri, Nov 08, 2019 at 11:21:56PM +0100, Richard Weinberger wrote:
> >> >> ----- Ursprüngliche Mail -----
> >> >> > btrfs found corrupted data on md1.  You appear to be using btrfs
> >> >> > -dsingle on a single mdadm raid1 device, so no recovery is possible
> >> >> > ("unable to fixup").
> >> >> >
> >> >> >> The system has ECC memory with md1 being a RAID1 which passes all health checks.
> >> >> >
> >> >> > mdadm doesn't have any way to repair data corruption--it can find
> >> >> > differences, but it cannot identify which version of the data is correct.
> >> >> > If one of your drives is corrupting data without reporting IO errors,
> >> >> > mdadm will simply copy the corruption to the other drive.  If one
> >> >> > drive is failing by intermittently injecting corrupted bits into reads
> >> >> > (e.g. because of a failure in the RAM on the drive control board),
> >> >> > this behavior may not show up in mdadm health checks.
> >> >>
> >> >> Well, this is not cheap hardware...
> >> >> Possible, but not very likely IMHO
> >> >
> >> > Even the disks?  We see RAM failures in disk drive embedded boards from
> >> > time to time.
> >>
> >> Yes. Enterprise-Storage RAID-Edition disks (sorry for the marketing buzzwords).
> >
> > Can you share the model numbers and firmware revisions?  There are a
> > lot of enterprise RE disks.  Not all of them work.
>
> Yes, will answer in a second mail. I found two more systems with such
> errors.

The original kernel used 4.12, should be sane. I've got Btrfs file
systems older than that in continuous use and haven't ever run into
corruptions that I didn't myself inject on purpose. The single data
point provided that makes me suspicious of a bug somewhere, possibly
not even in Btrfs or the drive, is that you have multiple systems
exhibiting this problem. That they are enterprise drives doesn't
itself significantly increase the likelihood it's a Btrfs bug rather
than a hardware bug, just because I've seen too many hardware bugs.
And Zygo has seen a whole lot more than I have.

I think Zygo's point of trying to isolate the cause of the problem is
what's important here. Assuming it's a Btrfs bug that somehow no one
else is experiencing, is going to take a huge amount of resources to
discover. Whereas assuming it's hardware related, or otherwise related
to the storage stack arragement, is a lot easier to isolate and test
against - process of elimination is faster. And that's the goal. Find
out exactly what's causing the corruption, because yeah, if it's a
Btrfs bug, it's an oh sh*t moment because it can affect quite a lot of
other instances than just the enterprise drives you're using.


> I didn't claim that my setup is perfect. What strikes me a little is that
> the only possible explanation from your side are super corner cases like
> silent data corruption within an enterprise disk, followed by silent failure
> of my RAID1, etc..
>
> I fully agree that such things *can* happen but it is not the most likely
> kind of failure.
> All devices are being checked by SMART. Sure, SMART could also be lying to me, but...

I don't think Zygo is saying it's definitely not a Btrfs bug. I think
he's saying there is a much easier way to isolate where this bug is
coming from, than to assume it's Btrfs and therefore start digging
through a hundreds of millions of lines of code to find it, when no
one else has a reproducer and no idea where to even start looking for
it.

And the long standing Btrfs developers from the very beginning can
tell you about their experiences with very high end hardware being
caught in the act of silent data corruption that Btrfs exposed. This
is the same sort of things the ZFS developers also discovered ages ago
and keep on encountering often enough that even now ext4 and XFS do
metadata checksumming because it's such a known issue.

-- 
Chris Murphy

  parent reply index

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-05 22:03 Richard Weinberger
2019-11-08 22:06 ` Richard Weinberger
2019-11-08 22:16   ` Zygo Blaxell
2019-11-08 22:09 ` Zygo Blaxell
2019-11-08 22:21   ` Richard Weinberger
2019-11-08 22:25     ` Zygo Blaxell
2019-11-08 22:31       ` Richard Weinberger
2019-11-08 23:39         ` Zygo Blaxell
2019-11-09  9:58           ` checksum errors in orphaned blocks on multiple systems (Was: Re: Decoding "unable to fixup (regular)" errors) Richard Weinberger
2019-11-13  3:34             ` Zygo Blaxell
2019-11-09 10:00           ` Decoding "unable to fixup (regular)" errors Richard Weinberger
2019-11-13  3:31             ` Zygo Blaxell
2019-11-13 18:17             ` Chris Murphy [this message]
2019-11-13 18:24               ` Chris Murphy
2019-11-16  6:16               ` Zygo Blaxell

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJCQCtTeqxxO=xAw5ogLORoNCvx7E7n0NQ4uF725bYX6vab25Q@mail.gmail.com' \
    --to=lists@colorremedies.com \
    --cc=ce3g8jdj@umail.furryterror.org \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=richard@nod.at \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git