Re: ext4: journal has aborted

From: Eric Whitney <enwlinux@gmail.com>
To: Theodore Ts'o <tytso@mit.edu>
Cc: David Jander <david@protonic.nl>,
	Dmitry Monakhov <dmonakhov@openvz.org>,
	Matteo Croce <technoboy85@gmail.com>,
	"Darrick J. Wong" <darrick.wong@oracle.com>,
	linux-ext4@vger.kernel.org, Eric Whitney <enwlinux@gmail.com>
Subject: Re: ext4: journal has aborted
Date: Thu, 10 Jul 2014 14:57:48 -0400	[thread overview]
Message-ID: <20140710185748.GA26636@wallace> (raw)
In-Reply-To: <20140707225619.GD8254@thunk.org>

* Theodore Ts'o <tytso@mit.edu>:
> On Mon, Jul 07, 2014 at 11:53:10AM -0400, Theodore Ts'o wrote:
> > An update from today's ext4 concall.  Eric Whitney can fairly reliably
> > reproduce this on his Panda board with 3.15, and definitely not on
> > 3.14.  So at this point there seems to be at least some kind of 3.15
> > regression going on here, regardless of whether it's in the eMMC
> > driver or the ext4 code.  (It also means that the bug fix I found is
> > irrelevant for the purposes of working this issue, since that's a much
> > harder to hit, and that bug has been around long before 3.14.)
> > 
> > The problem in terms of narrowing it down any further is that the
> > Pandaboard is running into RCU bugs which makes it hard to test the
> > early 3.15-rcX kernels.....
> 
> In the hopes of making it easy to bisect, I've created a kernel branch
> which starts with 3.14, and then adds on all of the ext4-related
> commits since then.   You can find it at:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git test-mb_generate_buddy-failure
> 
> https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git/log/?h=test-mb_generate_buddy-failure
> 
> Eric, can you see if you can repro the failure on your Panda Board?
> If you can, try doing a bisection search on these series:
> 
> git bisect start
> git bisect good v3.14
> git bisect bad test-mb_generate_buddy-failure
> 
> Hopefully if it is caused by one of the commits in this series, we'll
> be able to pin point it this way.

First, the good news (with luck):

My testing currently suggests that the patch causing this regression was
pulled into 3.15-rc3 -

007649375f6af242d5b1df2c15996949714303ba
ext4: initialize multi-block allocator before checking block descriptors

Bisection by selectively reverting ext4 commits in -rc3 identified this patch
while running on the Pandaboard.  I'm still using generic/068 as my reproducer.
It occasionally yields a false negative, but it has passed 10 consecutive
trials on my revert/bisect kernel derived from 3.15-rc3.  Given the frequency
of false negatives I've seen, I'm reasonably confident in that result.  I'm
going to run another series with just that patch reverted on 3.16-rc3.

Looking at the patch, the call to ext4_mb_init() was hoisted above the code
performing journal recovery in ext4_fill_super().  The regression occurs only
after journal recovery on the root filesystem.

Secondly:

Thanks for that git tree!  However, I discovered that the same "RCU bug" I
thought I was seeing on the Panda was also visible on the x86_64 KVM, and
it was actually just RCU noticing stalls.  These also occurred when using
your git tree as well as on mainline 3.15-rc1 and 3.15-rc2 and during
bisection attempts on 3.15-rc3 within the ext4 patches, and had the effect of
masking the regression on the root filesystem.  The test system would lock up
completely - no console response - and made it impossible to force the reboot
which was required to set up the failure.  Hence the reversion approach, since
RCU does not report stalls in 3.15-rc3 (final).

Eric

> 
> Thanks!!
> 
> 						- Ted