From: Dave Chinner <david@fromorbit.com>
To: Chris Mason <chris.mason@oracle.com>
Cc: Jon Nelson <jnelson@jamponi.net>,
Mike Snitzer <snitzer@redhat.com>, Matt <jackdachef@gmail.com>,
Milan Broz <mbroz@redhat.com>, Andi Kleen <andi@firstfloor.org>,
linux-btrfs <linux-btrfs@vger.kernel.org>,
dm-devel <dm-devel@redhat.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
htd <htd@fancy-poultry.org>, htejun <htejun@gmail.com>,
linux-ext4 <linux-ext4@vger.kernel.org>
Subject: Re: hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective)
Date: Thu, 16 Dec 2010 14:37:18 +1100 [thread overview]
Message-ID: <20101216033718.GM9925@dastard> (raw)
In-Reply-To: <1291810586-sup-8211@think>
On Wed, Dec 08, 2010 at 07:20:24AM -0500, Chris Mason wrote:
> Excerpts from Jon Nelson's message of 2010-12-07 22:29:26 -0500:
> > On Tue, Dec 7, 2010 at 3:02 PM, Chris Mason <chris.mason@oracle.com> wrote:
> > > Excerpts from Jon Nelson's message of 2010-12-07 15:48:58 -0500:
> > >> On Tue, Dec 7, 2010 at 2:41 PM, Chris Mason <chris.mason@oracle.com> wrote:
> > >> > Excerpts from Jon Nelson's message of 2010-12-07 15:25:47 -0500:
> > >> >> On Tue, Dec 7, 2010 at 2:02 PM, Chris Mason <chris.mason@oracle.com> wrote:
> > >> >> > Excerpts from Jon Nelson's message of 2010-12-07 14:34:40 -0500:
> > >> >> >> On Tue, Dec 7, 2010 at 12:52 PM, Chris Mason <chris.mason@oracle.com> wrote:
> > >> >> >> >> postgresql errors. Typically, header corruption but from the limited
> > >> >> >> >> visibility I've had into this via strace, what I see is zeroed pages
> > >> >> >> >> where there shouldn't be.
> > >> >> >> >
> > >> >> >> > This sounds a lot like a bug higher up than dm-crypt. Zeros tend to
> > >> >> >> > come from some piece of code explicitly filling a page with zeros, and
> > >> >> >> > that often happens in the corner cases for O_DIRECT and a few other
> > >> >> >> > places in the filesystem.
> > >> >> >> >
> > >> >> >> > Have you tried triggering this with a regular block device?
> > >> >> >>
> > >> >> >> I just tried the whole set of tests, but with /dev/sdb directly (as
> > >> >> >> ext4) without any crypt-y bits.
> > >> >> >> It takes more iterations but out of 6 tests I had one failure: same
> > >> >> >> type of thing, 'invalid page header in block ....'.
> > >> >> >>
> > >> >> >> I can't guarantee that it is a full-page of zeroes, just what I saw
> > >> >> >> from the (limited) stracing I did.
> > >> >> >
> > >> >> > Fantastic. Now for our usual suspects:
> >
> > Maybe not so fantastic. I kept testing and had no more failures. At
> > all. After 40+ iterations I gave up.
> > I went back to trying ext4 on a LUKS volume. The 'hit' ratio went to
> > something like 1 in 3, or better.
> >
> > I will continue to do testing with and without LUKS. I did /not/
> > reboot between tests, but I do start with a fresh postgres database.
> >
>
> Once we trigger once without dm-crypt, dm-crypt is off the hook. Just
> to verify, when you say without luks, you mean without any crypto bits
> in use at all on the filesystems postgres uses?
>
> Usually the trick to reproducing filesystem corruptions is adding memory
> pressure. The corruption is probably a bad interaction between reads
> and writes, and we need to make sure the reads actually happen.
>
> http://oss.oracle.com/~mason/pin_ram.c
>
> gcc -Wall -o pin_ram pin_ram.c
>
> pin_ram -m 80%-of-your-ram-in-mb
Implemented in xfstests about 10 years ago:
http://git.kernel.org/?p=fs/xfs/xfstests-dev.git;a=blob;f=src/usemem.c;h=b8794a6b209cebf8dbf312a8ef131e2e54b18d29;hb=HEAD
:P
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2010-12-16 3:37 UTC|newest]
Thread overview: 100+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-06 22:16 [PATCH] DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ? Matt
2010-11-07 14:30 ` Milan Broz
2010-11-07 17:49 ` Matt
2010-11-07 19:32 ` Matt
2010-11-07 19:45 ` Andi Kleen
2010-11-07 21:39 ` Milan Broz
2010-11-07 23:05 ` Andi Kleen
2010-11-08 14:16 ` [dm-devel] " Alasdair G Kergon
2010-11-08 14:58 ` Mike Snitzer
2010-11-08 17:59 ` Chris Mason
2010-11-14 20:59 ` dm-crypt barrier support is effective (was: Re: DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ?) Mike Snitzer
2010-11-14 21:49 ` Matt
2010-11-14 21:54 ` dm-crypt barrier support is effective Milan Broz
2010-11-14 23:24 ` Matt
2010-12-01 16:05 ` Matt
2010-12-01 16:52 ` Mike Snitzer
2010-12-01 17:35 ` Matt
2010-12-01 18:24 ` Milan Broz
2010-12-01 19:34 ` Jon Nelson
2010-12-01 20:45 ` Milan Broz
2010-12-01 21:23 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Mike Snitzer
2010-12-02 21:30 ` Matt
2010-12-04 19:18 ` Matt
2010-12-04 19:38 ` Mike Snitzer
2010-12-04 23:47 ` Matt
2010-12-07 14:21 ` Chris Mason
2010-12-07 18:10 ` Jon Nelson
2010-12-07 18:15 ` Chris Mason
2010-12-07 18:22 ` Mike Snitzer
2010-12-07 18:45 ` Jon Nelson
2010-12-07 18:52 ` Chris Mason
2010-12-07 19:34 ` Jon Nelson
2010-12-07 20:02 ` Chris Mason
2010-12-07 20:25 ` Jon Nelson
2010-12-07 20:33 ` Chris Mason
2010-12-07 20:36 ` Jon Nelson
2010-12-07 20:41 ` Chris Mason
2010-12-07 20:48 ` Jon Nelson
2010-12-07 21:02 ` Chris Mason
2010-12-08 3:29 ` Jon Nelson
2010-12-08 8:03 ` hunt for 2.6.37 dm-crypt+ext4 corruption? Milan Broz
2010-12-08 12:20 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Chris Mason
2010-12-16 3:37 ` Dave Chinner [this message]
2010-12-16 12:29 ` Chris Mason
2010-12-08 3:55 ` Jon Nelson
2010-12-07 19:35 ` Ted Ts'o
2010-12-07 21:01 ` Jon Nelson
2010-12-08 3:37 ` Jon Nelson
2010-12-08 15:26 ` Jon Nelson
2010-12-09 18:01 ` Ted Ts'o
2010-12-09 18:10 ` Jon Nelson
2010-12-09 20:13 ` Ted Ts'o
2010-12-09 20:38 ` Jon Nelson
2010-12-09 23:16 ` Andi Kleen
2010-12-10 1:38 ` Chris Mason
2010-12-10 1:53 ` Matt
2010-12-10 2:38 ` Ted Ts'o
2010-12-10 6:52 ` Jon Nelson
2010-12-10 14:58 ` Jon Nelson
2010-12-10 16:54 ` Jon Nelson
2010-12-11 2:14 ` Jon Nelson
2010-12-12 1:40 ` Ted Ts'o
2010-12-12 2:34 ` Ted Ts'o
2010-12-12 3:16 ` Jon Nelson
2010-12-12 10:18 ` Jon Nelson
2010-12-12 12:43 ` Ted Ts'o
2010-12-12 13:11 ` Jon Nelson
2010-12-13 2:06 ` Ted Ts'o
2010-12-13 18:56 ` Jon Nelson
2010-12-15 19:15 ` Matt
2010-12-15 19:16 ` Andi Kleen
2010-12-15 19:25 ` Matt
2010-12-15 19:28 ` Matt
2010-12-10 1:58 ` Mike Fedyk
2010-12-10 2:00 ` Chris Mason
2010-12-10 2:05 ` Jon Nelson
2010-12-04 23:52 ` Matt
2010-12-05 10:09 ` Heinz Diehl
2010-12-05 10:21 ` hunt for 2.6.37 dm-crypt+ext4 corruption? Milan Broz
2010-12-05 12:49 ` Heinz Diehl
2010-12-05 13:24 ` [dm-devel] " Theodore Tso
2010-12-05 13:44 ` Matt
2010-12-05 14:02 ` Ted Ts'o
2010-12-05 14:33 ` Heinz Diehl
2010-12-05 20:17 ` Daniel J Blueman
2010-12-06 7:08 ` Heinz Diehl
2010-12-05 20:28 ` Andi Kleen
2010-12-05 21:15 ` Mike Snitzer
2010-12-05 21:42 ` [dm-devel] " Milan Broz
2010-12-06 2:37 ` Valdis.Kletnieks
2011-01-06 15:56 ` Heinz Diehl
2011-01-07 16:45 ` Matt
2010-12-05 13:30 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Matt
2010-12-05 0:57 ` Matt
2010-12-04 20:51 ` Heinz Diehl
2010-12-01 19:59 ` dm-crypt barrier support is effective Heinz Diehl
2010-11-15 7:25 ` Heinz Diehl
2010-11-15 8:41 ` Milan Broz
2010-11-07 20:36 ` [PATCH] DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ? Heinz Diehl
2010-11-07 16:03 ` Heinz Diehl
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101216033718.GM9925@dastard \
--to=david@fromorbit.com \
--cc=andi@firstfloor.org \
--cc=chris.mason@oracle.com \
--cc=dm-devel@redhat.com \
--cc=htd@fancy-poultry.org \
--cc=htejun@gmail.com \
--cc=jackdachef@gmail.com \
--cc=jnelson@jamponi.net \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mbroz@redhat.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).