From: Matt <jackdachef@gmail.com>
To: Mike Snitzer <snitzer@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>,
linux-btrfs <linux-btrfs@vger.kernel.org>,
dm-devel <dm-devel@redhat.com>, Milan Broz <mbroz@redhat.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
htd <htd@fancy-poultry.org>, Chris Mason <chris.mason@oracle.com>
Subject: Re: dm-crypt barrier support is effective (was: Re: DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ?)
Date: Sun, 14 Nov 2010 22:49:12 +0100 [thread overview]
Message-ID: <AANLkTi=rVyozmvqCJD1kGqnV3uEn-0ak6B12-O_9XOKJ@mail.gmail.com> (raw)
In-Reply-To: <20101114205925.GA20451@redhat.com>
On Sun, Nov 14, 2010 at 9:59 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> On Mon, Nov 08 2010 at 12:59pm -0500,
> Chris Mason <chris.mason@oracle.com> wrote:
>
>> Excerpts from Mike Snitzer's message of 2010-11-08 09:58:09 -0500:
>> > On Sun, Nov 07 2010 at 6:05pm -0500,
>> > Andi Kleen <andi@firstfloor.org> wrote:
>> >
>> > > On Sun, Nov 07, 2010 at 10:39:23PM +0100, Milan Broz wrote:
>> > > > On 11/07/2010 08:45 PM, Andi Kleen wrote:
>> > > > >> I read about barrier-problems and data getting to the partition when
>> > > > >> using dm-crypt and several layers so I don't know if that could be
>> > > > >> related
>> > > > >
>> > > > > Barriers seem to be totally broken on dm-crypt currently.
>> > > >
>> > > > Can you explain it?
>> > >
>> > > e.g. the btrfs mailing list is full of corruption reports
>> > > on dm-crypt and most of the symptoms point to broken barriers.
>> >
>> > [cc'ing linux-btrfs, hopefully in the future dm-devel will get cc'd when
>> > concerns about DM come up on linux-btrfs (or other lists)]
>> >
>> > I spoke with Josef Bacik and these corruption reports are apparently
>> > against older kernels (e.g. <= 2.6.33). I say <= 2.6.33 because:
>>
>> We've consistently seen reports about corruptions on power hits with
>> dm-crypt. The logs didn't have any messages about barriers failing, but
>> the corruptions were still there. The most likely cause is that
>> barriers just aren't getting through somehow.
>
> Can't blame anyone for assuming as much (although it does create FUD)
> but in practice (testing dm-crypt with ext4 using your barrier-test
> script) I have not been able to see any evidence that dm-crypt's barrier
> support is ineffective.
>
> Could be that the barrier-test script isn't able to reproduce the unique
> failure case that btrfs does (on power failure)?
>
>> > > > Barriers/flush change should work, if it is broken, it is not only dm-crypt.
>> > > > (dm-crypt simply relies on dm-core implementation, when barrier/flush
>> > > > request come to dmcrypt, all previous IO must be already finished).
>> > >
>> > > Possibly, at least it doesn't seem to work.
>> >
>> > Can you please be more specific? What test(s)? What kernel(s)?
>> >
>> > Any pointers to previous (and preferably: recent) reports would be
>> > appreciated.
>
> I still think we need specific bug reports that detail workloads and if
> possible reproducers.
>
>> > The DM barrier code has seen considerable change recently (via flush+fua
>> > changes in 2.6.37). Those changes have been tested quite a bit
>> > (including ext4 consistency after a crash).
>> >
>> > But even prior to those flush+fua changes DM's support for barriers
>> > (Linux >= 2.6.31) was held to be robust. No known (at least no
>> > reported) issues with DM's barrier support.
>>
>> I think it would be best to move forward with just hammering on the
>> dm-crypt barriers:
>>
>> http://oss.oracle.com/~mason/barrier-test
>>
>> This script is the best I've found so far to reliably trigger
>> corruptions with barriers off. I'd start with ext3 + barriers off just
>> to prove it corrupts things, then move to ext3 + barriers on.
>
> I started with ext4 + barrier=0,journal_async_commit and could reliably
> cause directory corruption (~75% of the time). I then switched to
> barrier=1 and could not cause corruption.
>
> I then added dm-crypt and got the same results: with barrier=1 I could
> not cause directory corruption. barrier=0 resulted in directory
> corruption (again ~75% of the time), no corruption occurred with
> barrier=1.
>
> Both 2.6.36 (original barrier code) and latest 2.6.37-rc1+ (new
> flush+fua code) were tested. 6 iterations of barrier=0 and 10
> iterations of barrier=1.
>
> So my hope is we can now put this general dm-crypt barrier doubt to one
> side and work together on identifying the cause of corruption when
> dm-crypt is paired with btrfs.
>
> Thanks,
> Mike
>
Hi Mike,
I'm pretty sure that dm-crypt is rockstable :)
My report wasn't meant to be / cause FUD sorry if it got picked up that way:
with the vanilla dm-crypt implementation I saw *NO* corruption at all
in the small testing amount of time I ran it
however as soon as I applied the
[dm-devel] [PATCH] DM-CRYPT: Scale to multiple CPUs v3
and
[PATCH] Fix double free and use generic private pointer in per-cpu
patches, recompiled the kernel and rebooted into that new environment
it seemingly caused corruptions right from the start (the mentioned
corruption /etc/env.d/02opengl to be the most obvious candidate and
probably even more)
with those corruptions being anticipated over longer uptime and heavy
use-patterns (such as re-compiling the whole system).
I don't know if the new multi-cpu scaling patch for dm-crypt makes a
change (since I can't test it right now due to a busy schedule)
[PATCH v5] dm crypt: scale to multiple CPUs
I however have a request:
could you guys please take this patch through a "battery of heavy
tests" until it's included in mainline ?
so that you can spot any issues (races, BUGs, etc.) which might be
inherent/triggered in current dm-crypt so that my reported corruptions
might be prevented in the future ?
Again:
the vanilla kernel and dm-crypt are perfectly stable !
only with the dm-crypt scaling patch I could observe the data-corruption
Thanks !
Matt
next prev parent reply other threads:[~2010-11-14 21:49 UTC|newest]
Thread overview: 100+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-06 22:16 [PATCH] DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ? Matt
2010-11-07 14:30 ` Milan Broz
2010-11-07 17:49 ` Matt
2010-11-07 19:32 ` Matt
2010-11-07 19:45 ` Andi Kleen
2010-11-07 21:39 ` Milan Broz
2010-11-07 23:05 ` Andi Kleen
2010-11-08 14:16 ` [dm-devel] " Alasdair G Kergon
2010-11-08 14:58 ` Mike Snitzer
2010-11-08 17:59 ` Chris Mason
2010-11-14 20:59 ` dm-crypt barrier support is effective (was: Re: DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ?) Mike Snitzer
2010-11-14 21:49 ` Matt [this message]
2010-11-14 21:54 ` dm-crypt barrier support is effective Milan Broz
2010-11-14 23:24 ` Matt
2010-12-01 16:05 ` Matt
2010-12-01 16:52 ` Mike Snitzer
2010-12-01 17:35 ` Matt
2010-12-01 18:24 ` Milan Broz
2010-12-01 19:34 ` Jon Nelson
2010-12-01 20:45 ` Milan Broz
2010-12-01 21:23 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Mike Snitzer
2010-12-02 21:30 ` Matt
2010-12-04 19:18 ` Matt
2010-12-04 19:38 ` Mike Snitzer
2010-12-04 23:47 ` Matt
2010-12-07 14:21 ` Chris Mason
2010-12-07 18:10 ` Jon Nelson
2010-12-07 18:15 ` Chris Mason
2010-12-07 18:22 ` Mike Snitzer
2010-12-07 18:45 ` Jon Nelson
2010-12-07 18:52 ` Chris Mason
2010-12-07 19:34 ` Jon Nelson
2010-12-07 20:02 ` Chris Mason
2010-12-07 20:25 ` Jon Nelson
2010-12-07 20:33 ` Chris Mason
2010-12-07 20:36 ` Jon Nelson
2010-12-07 20:41 ` Chris Mason
2010-12-07 20:48 ` Jon Nelson
2010-12-07 21:02 ` Chris Mason
2010-12-08 3:29 ` Jon Nelson
2010-12-08 8:03 ` hunt for 2.6.37 dm-crypt+ext4 corruption? Milan Broz
2010-12-08 12:20 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Chris Mason
2010-12-16 3:37 ` Dave Chinner
2010-12-16 12:29 ` Chris Mason
2010-12-08 3:55 ` Jon Nelson
2010-12-07 19:35 ` Ted Ts'o
2010-12-07 21:01 ` Jon Nelson
2010-12-08 3:37 ` Jon Nelson
2010-12-08 15:26 ` Jon Nelson
2010-12-09 18:01 ` Ted Ts'o
2010-12-09 18:10 ` Jon Nelson
2010-12-09 20:13 ` Ted Ts'o
2010-12-09 20:38 ` Jon Nelson
2010-12-09 23:16 ` Andi Kleen
2010-12-10 1:38 ` Chris Mason
2010-12-10 1:53 ` Matt
2010-12-10 2:38 ` Ted Ts'o
2010-12-10 6:52 ` Jon Nelson
2010-12-10 14:58 ` Jon Nelson
2010-12-10 16:54 ` Jon Nelson
2010-12-11 2:14 ` Jon Nelson
2010-12-12 1:40 ` Ted Ts'o
2010-12-12 2:34 ` Ted Ts'o
2010-12-12 3:16 ` Jon Nelson
2010-12-12 10:18 ` Jon Nelson
2010-12-12 12:43 ` Ted Ts'o
2010-12-12 13:11 ` Jon Nelson
2010-12-13 2:06 ` Ted Ts'o
2010-12-13 18:56 ` Jon Nelson
2010-12-15 19:15 ` Matt
2010-12-15 19:16 ` Andi Kleen
2010-12-15 19:25 ` Matt
2010-12-15 19:28 ` Matt
2010-12-10 1:58 ` Mike Fedyk
2010-12-10 2:00 ` Chris Mason
2010-12-10 2:05 ` Jon Nelson
2010-12-04 23:52 ` Matt
2010-12-05 10:09 ` Heinz Diehl
2010-12-05 10:21 ` hunt for 2.6.37 dm-crypt+ext4 corruption? Milan Broz
2010-12-05 12:49 ` Heinz Diehl
2010-12-05 13:24 ` [dm-devel] " Theodore Tso
2010-12-05 13:44 ` Matt
2010-12-05 14:02 ` Ted Ts'o
2010-12-05 14:33 ` Heinz Diehl
2010-12-05 20:17 ` Daniel J Blueman
2010-12-06 7:08 ` Heinz Diehl
2010-12-05 20:28 ` Andi Kleen
2010-12-05 21:15 ` Mike Snitzer
2010-12-05 21:42 ` [dm-devel] " Milan Broz
2010-12-06 2:37 ` Valdis.Kletnieks
2011-01-06 15:56 ` Heinz Diehl
2011-01-07 16:45 ` Matt
2010-12-05 13:30 ` hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective) Matt
2010-12-05 0:57 ` Matt
2010-12-04 20:51 ` Heinz Diehl
2010-12-01 19:59 ` dm-crypt barrier support is effective Heinz Diehl
2010-11-15 7:25 ` Heinz Diehl
2010-11-15 8:41 ` Milan Broz
2010-11-07 20:36 ` [PATCH] DM-CRYPT: Scale to multiple CPUs v3 on 2.6.37-rc* ? Heinz Diehl
2010-11-07 16:03 ` Heinz Diehl
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='AANLkTi=rVyozmvqCJD1kGqnV3uEn-0ak6B12-O_9XOKJ@mail.gmail.com' \
--to=jackdachef@gmail.com \
--cc=andi@firstfloor.org \
--cc=chris.mason@oracle.com \
--cc=dm-devel@redhat.com \
--cc=htd@fancy-poultry.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mbroz@redhat.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).