Re: dramatic I/O slowdown after upgrading 2.6.32->3.0

From: Dave Chinner <david@fromorbit.com>
To: Michael Tokarev <mjt@tls.msk.ru>
Cc: Jan Kara <jack@suse.cz>,
	Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: dramatic I/O slowdown after upgrading 2.6.32->3.0
Date: Tue, 10 Apr 2012 12:26:28 +1000	[thread overview]
Message-ID: <20120410022628.GN18323@dastard> (raw)
In-Reply-To: <4F7E74F4.90604@msgid.tls.msk.ru>

On Fri, Apr 06, 2012 at 08:45:40AM +0400, Michael Tokarev wrote:
> On 06.04.2012 03:29, Jan Kara wrote:
> 
> > On Fri 30-03-12 20:50:54, Michael Tokarev wrote:
> >> I'm observing a dramatic slowdown of several hosts after upgrading
> >> from 2.6.32.y to 3.0.x i686 kernels (in both cases from kernel.org,
> >> on both cases the last version is relatively latest).
> []
> >> What's the way to debug this issue?
> >   Identifying a particular kernel where things regresses might help as Jon
> > wrote. Just from top of my head, 3.0 had a bug in device plugging so
> > readahead was broken. I think it was addressed in -stable series so you
> 
> That's definitely not readahead, it since writes are painfully slow
> too.  I found one more example -- extlinux --once="test kernel" with
> 3.0 takes about 20 seconds to complete on an idle system.
> 
> > might want to check out latest 3.0-stable.
> 
> I did mention this in my initial email (that part quoted above) --
> both 2.6.32 and 3.0 are relatively latest from each series,
> right now it is 3.0.27.
> 
> Yesterday I tried to do some bisection, but ended up in an unbootable
> system (it is remote production server), so now I'm waiting for remote
> hands to repair it (I don't yet know what went wrong, we'll figure it
> out).  I've some time during nights when I can do anything with that
> machine, but I have to keep it reachable/working on each reboot.
> 
> Apparently I was wrong saying that there's another machine which
> suffers from the same issue -- nope, the other machine had an unrelated
> issue which I fixed.  So it turns out that from about 200 different
> machines, I've just one machine which does not run 3.0 kernel properly.
> I especially tried 3.0 on a few more - different - machines last
> weekend, in order to see what other machines has this problem, but
> found nothing.
> 
> So I'll try to continue (or actually _start_) the bisection on this
> very server, the way it will be possible having in mind the difficult
> conditions.
> 
> I just thoght I'd ask first, maybe someone knows offhand what may be
> the problem.. ;)

Barriers. Turn them off, and see if that fixes your problem.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com