From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1756262Ab1D0Q0i (ORCPT <rfc822;w@1wt.eu>);
	Wed, 27 Apr 2011 12:26:38 -0400
Received: from legolas.restena.lu ([158.64.1.34]:34859 "EHLO
	legolas.restena.lu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1750988Ab1D0Q0h convert rfc822-to-8bit (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Wed, 27 Apr 2011 12:26:37 -0400
Date: Wed, 27 Apr 2011 18:26:22 +0200
From: Bruno =?UTF-8?B?UHLDqW1vbnQ=?= <bonbons@linux-vserver.org>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs-masters@oss.sgi.com, xfs@oss.sgi.com,
        Christoph Hellwig <hch@infradead.org>, Alex Elder <aelder@sgi.com>,
        Dave Chinner <dchinner@redhat.com>, linux-kernel@vger.kernel.org
Subject: Re: 2.6.39-rc3, 2.6.39-rc4: XFS lockup - regression since 2.6.38
Message-ID: <20110427182622.05a068a2@neptune.home>
In-Reply-To: <20110427050850.GG12436@dastard>
References: <20110423224403.5fd1136a@neptune.home>
	<20110427050850.GG12436@dastard>
X-Mailer: Claws Mail 3.7.8 (GTK+ 2.22.1; i686-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8BIT
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, 27 April 2011 Dave Chinner <david@fromorbit.com> wrote:
> On Sat, Apr 23, 2011 at 10:44:03PM +0200, Bruno Prémont wrote:
> > Running 2.6.39-rc3+ and now again on 2.6.39-rc4+ (I've not tested -rc1
> > or -rc2) I've hit a "dying machine" where processes writing to disk end
> > up in D state.
> > From occurrence with -rc3+ I don't have logs as those never hit the disk,
> > for -rc4+ I have the following (sysrq+t was too big, what I have of it
> > misses a dozen of kernel tasks - if needed, please ask):
> > 
> > The -rc4 kernel is at commit 584f79046780e10cb24367a691f8c28398a00e84
> > (+ 1 patch of mine to stop disk on reboot),
> > full dmesg available if needed; kernel config attached (only selected
> > options). In case there is something I should do at next occurrence
> > please tell. Unfortunately I have no trigger for it and it does not
> > happen very often.
> > 
> > [    0.000000] Linux version 2.6.39-rc4-00120-g73b5b55 (kbuild@neptune) (gcc version 4.4.5 (Gentoo 4.4.5 p1.2, pie-0.4.5) ) #12 Thu Apr 21 19:28:45 CEST 2011
> > 
> > 
> > [32040.120055] INFO: task flush-8:0:1665 blocked for more than 120 seconds.
> > [32040.120068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [32040.120077] flush-8:0       D 00000000  4908  1665      2 0x00000000
> > [32040.120099]  f55efb5c 00000046 00000000 00000000 00000000 00000001 e0382924 00000000
> > [32040.120118]  f55efb0c f55efb5c 00000004 f629ba70 572f01a2 00001cfe f629ba70 ffffffc0
> > [32040.120135]  f55efc68 f55efb30 f889d7f8 f55efb20 00000000 f55efc68 e0382900 f55efc94
> > [32040.120153] Call Trace:
> > [32040.120220]  [<f889d7f8>] ? xfs_bmap_search_multi_extents+0x88/0xe0 [xfs]
> > [32040.120239]  [<c109ce1d>] ? kmem_cache_alloc+0x2d/0x110
> > [32040.120294]  [<f88c88ca>] ? xlog_space_left+0x2a/0xc0 [xfs]
> > [32040.120346]  [<f88c85cb>] xlog_wait+0x4b/0x70 [xfs]
> > [32040.120359]  [<c102ca00>] ? try_to_wake_up+0xc0/0xc0
> > [32040.120411]  [<f88c948b>] xlog_grant_log_space+0x8b/0x240 [xfs]
> > [32040.120464]  [<f88c936e>] ? xlog_grant_push_ail+0xbe/0xf0 [xfs]
> > [32040.120516]  [<f88c99db>] xfs_log_reserve+0xab/0xb0 [xfs]
> > [32040.120571]  [<f88d6dc8>] xfs_trans_reserve+0x78/0x1f0 [xfs]
> 
> Hmmmmm. That may be caused by the conversion of the xfsaild to a
> work queue. Can you post the output of "xfs_info <mntpt>" and the
> mount options (/proc/mounts) used on you system?

Here it comes (including all XFS mount-points - with affected kernel
but after fresh boot):

* /proc/mountinfo *
/dev/sda6 /mnt/.SRC xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda7 /home xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda6 /var/cache/edb xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda6 /usr/src xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda6 /var/tmp xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda6 /var/log xfs rw,noatime,attr2,delaylog,noquota 0 0
/dev/sda6 /var/lib/portage/packages xfs rw,noatime,attr2,delaylog,noquota 0 0

* xfs_info *
meta-data=/dev/sda7              isize=256    agcount=4, agsize=987996 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=3951982, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0


meta-data=/dev/sda6              isize=256    agcount=4, agsize=655149 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2620595, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0


Bruno

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <xfs-bounces@oss.sgi.com>
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id
	p3RGN4YT221861 for <xfs@oss.sgi.com>; Wed, 27 Apr 2011 11:23:04 -0500
Date: Wed, 27 Apr 2011 18:26:22 +0200
From: Bruno =?UTF-8?B?UHLDqW1vbnQ=?= <bonbons@linux-vserver.org>
Subject: Re: 2.6.39-rc3, 2.6.39-rc4: XFS lockup - regression since 2.6.38
Message-ID: <20110427182622.05a068a2@neptune.home>
In-Reply-To: <20110427050850.GG12436@dastard>
References: <20110423224403.5fd1136a@neptune.home>
	<20110427050850.GG12436@dastard>
Mime-Version: 1.0
List-Id: XFS Filesystem from SGI <xfs.oss.sgi.com>
List-Unsubscribe: <http://oss.sgi.com/mailman/options/xfs>,
	<mailto:xfs-request@oss.sgi.com?subject=unsubscribe>
List-Archive: <http://oss.sgi.com/pipermail/xfs>
List-Post: <mailto:xfs@oss.sgi.com>
List-Help: <mailto:xfs-request@oss.sgi.com?subject=help>
List-Subscribe: <http://oss.sgi.com/mailman/listinfo/xfs>,
	<mailto:xfs-request@oss.sgi.com?subject=subscribe>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Sender: xfs-bounces@oss.sgi.com
Errors-To: xfs-bounces@oss.sgi.com
To: Dave Chinner <david@fromorbit.com>
Cc: Dave Chinner <dchinner@redhat.com>, linux-kernel@vger.kernel.org, xfs@oss.sgi.com, Christoph Hellwig <hch@infradead.org>, xfs-masters@oss.sgi.com, Alex Elder <aelder@sgi.com>

T24gV2VkLCAyNyBBcHJpbCAyMDExIERhdmUgQ2hpbm5lciA8ZGF2aWRAZnJvbW9yYml0LmNvbT4g
d3JvdGU6Cj4gT24gU2F0LCBBcHIgMjMsIDIwMTEgYXQgMTA6NDQ6MDNQTSArMDIwMCwgQnJ1bm8g
UHLDqW1vbnQgd3JvdGU6Cj4gPiBSdW5uaW5nIDIuNi4zOS1yYzMrIGFuZCBub3cgYWdhaW4gb24g
Mi42LjM5LXJjNCsgKEkndmUgbm90IHRlc3RlZCAtcmMxCj4gPiBvciAtcmMyKSBJJ3ZlIGhpdCBh
ICJkeWluZyBtYWNoaW5lIiB3aGVyZSBwcm9jZXNzZXMgd3JpdGluZyB0byBkaXNrIGVuZAo+ID4g
dXAgaW4gRCBzdGF0ZS4KPiA+IEZyb20gb2NjdXJyZW5jZSB3aXRoIC1yYzMrIEkgZG9uJ3QgaGF2
ZSBsb2dzIGFzIHRob3NlIG5ldmVyIGhpdCB0aGUgZGlzaywKPiA+IGZvciAtcmM0KyBJIGhhdmUg
dGhlIGZvbGxvd2luZyAoc3lzcnErdCB3YXMgdG9vIGJpZywgd2hhdCBJIGhhdmUgb2YgaXQKPiA+
IG1pc3NlcyBhIGRvemVuIG9mIGtlcm5lbCB0YXNrcyAtIGlmIG5lZWRlZCwgcGxlYXNlIGFzayk6
Cj4gPiAKPiA+IFRoZSAtcmM0IGtlcm5lbCBpcyBhdCBjb21taXQgNTg0Zjc5MDQ2NzgwZTEwY2Iy
NDM2N2E2OTFmOGMyODM5OGEwMGU4NAo+ID4gKCsgMSBwYXRjaCBvZiBtaW5lIHRvIHN0b3AgZGlz
ayBvbiByZWJvb3QpLAo+ID4gZnVsbCBkbWVzZyBhdmFpbGFibGUgaWYgbmVlZGVkOyBrZXJuZWwg
Y29uZmlnIGF0dGFjaGVkIChvbmx5IHNlbGVjdGVkCj4gPiBvcHRpb25zKS4gSW4gY2FzZSB0aGVy
ZSBpcyBzb21ldGhpbmcgSSBzaG91bGQgZG8gYXQgbmV4dCBvY2N1cnJlbmNlCj4gPiBwbGVhc2Ug
dGVsbC4gVW5mb3J0dW5hdGVseSBJIGhhdmUgbm8gdHJpZ2dlciBmb3IgaXQgYW5kIGl0IGRvZXMg
bm90Cj4gPiBoYXBwZW4gdmVyeSBvZnRlbi4KPiA+IAo+ID4gWyAgICAwLjAwMDAwMF0gTGludXgg
dmVyc2lvbiAyLjYuMzktcmM0LTAwMTIwLWc3M2I1YjU1IChrYnVpbGRAbmVwdHVuZSkgKGdjYyB2
ZXJzaW9uIDQuNC41IChHZW50b28gNC40LjUgcDEuMiwgcGllLTAuNC41KSApICMxMiBUaHUgQXBy
IDIxIDE5OjI4OjQ1IENFU1QgMjAxMQo+ID4gCj4gPiAKPiA+IFszMjA0MC4xMjAwNTVdIElORk86
IHRhc2sgZmx1c2gtODowOjE2NjUgYmxvY2tlZCBmb3IgbW9yZSB0aGFuIDEyMCBzZWNvbmRzLgo+
ID4gWzMyMDQwLjEyMDA2OF0gImVjaG8gMCA+IC9wcm9jL3N5cy9rZXJuZWwvaHVuZ190YXNrX3Rp
bWVvdXRfc2VjcyIgZGlzYWJsZXMgdGhpcyBtZXNzYWdlLgo+ID4gWzMyMDQwLjEyMDA3N10gZmx1
c2gtODowICAgICAgIEQgMDAwMDAwMDAgIDQ5MDggIDE2NjUgICAgICAyIDB4MDAwMDAwMDAKPiA+
IFszMjA0MC4xMjAwOTldICBmNTVlZmI1YyAwMDAwMDA0NiAwMDAwMDAwMCAwMDAwMDAwMCAwMDAw
MDAwMCAwMDAwMDAwMSBlMDM4MjkyNCAwMDAwMDAwMAo+ID4gWzMyMDQwLjEyMDExOF0gIGY1NWVm
YjBjIGY1NWVmYjVjIDAwMDAwMDA0IGY2MjliYTcwIDU3MmYwMWEyIDAwMDAxY2ZlIGY2MjliYTcw
IGZmZmZmZmMwCj4gPiBbMzIwNDAuMTIwMTM1XSAgZjU1ZWZjNjggZjU1ZWZiMzAgZjg4OWQ3Zjgg
ZjU1ZWZiMjAgMDAwMDAwMDAgZjU1ZWZjNjggZTAzODI5MDAgZjU1ZWZjOTQKPiA+IFszMjA0MC4x
MjAxNTNdIENhbGwgVHJhY2U6Cj4gPiBbMzIwNDAuMTIwMjIwXSAgWzxmODg5ZDdmOD5dID8geGZz
X2JtYXBfc2VhcmNoX211bHRpX2V4dGVudHMrMHg4OC8weGUwIFt4ZnNdCj4gPiBbMzIwNDAuMTIw
MjM5XSAgWzxjMTA5Y2UxZD5dID8ga21lbV9jYWNoZV9hbGxvYysweDJkLzB4MTEwCj4gPiBbMzIw
NDAuMTIwMjk0XSAgWzxmODhjODhjYT5dID8geGxvZ19zcGFjZV9sZWZ0KzB4MmEvMHhjMCBbeGZz
XQo+ID4gWzMyMDQwLjEyMDM0Nl0gIFs8Zjg4Yzg1Y2I+XSB4bG9nX3dhaXQrMHg0Yi8weDcwIFt4
ZnNdCj4gPiBbMzIwNDAuMTIwMzU5XSAgWzxjMTAyY2EwMD5dID8gdHJ5X3RvX3dha2VfdXArMHhj
MC8weGMwCj4gPiBbMzIwNDAuMTIwNDExXSAgWzxmODhjOTQ4Yj5dIHhsb2dfZ3JhbnRfbG9nX3Nw
YWNlKzB4OGIvMHgyNDAgW3hmc10KPiA+IFszMjA0MC4xMjA0NjRdICBbPGY4OGM5MzZlPl0gPyB4
bG9nX2dyYW50X3B1c2hfYWlsKzB4YmUvMHhmMCBbeGZzXQo+ID4gWzMyMDQwLjEyMDUxNl0gIFs8
Zjg4Yzk5ZGI+XSB4ZnNfbG9nX3Jlc2VydmUrMHhhYi8weGIwIFt4ZnNdCj4gPiBbMzIwNDAuMTIw
NTcxXSAgWzxmODhkNmRjOD5dIHhmc190cmFuc19yZXNlcnZlKzB4NzgvMHgxZjAgW3hmc10KPiAK
PiBIbW1tbW0uIFRoYXQgbWF5IGJlIGNhdXNlZCBieSB0aGUgY29udmVyc2lvbiBvZiB0aGUgeGZz
YWlsZCB0byBhCj4gd29yayBxdWV1ZS4gQ2FuIHlvdSBwb3N0IHRoZSBvdXRwdXQgb2YgInhmc19p
bmZvIDxtbnRwdD4iIGFuZCB0aGUKPiBtb3VudCBvcHRpb25zICgvcHJvYy9tb3VudHMpIHVzZWQg
b24geW91IHN5c3RlbT8KCkhlcmUgaXQgY29tZXMgKGluY2x1ZGluZyBhbGwgWEZTIG1vdW50LXBv
aW50cyAtIHdpdGggYWZmZWN0ZWQga2VybmVsCmJ1dCBhZnRlciBmcmVzaCBib290KToKCiogL3By
b2MvbW91bnRpbmZvICoKL2Rldi9zZGE2IC9tbnQvLlNSQyB4ZnMgcncsbm9hdGltZSxhdHRyMixk
ZWxheWxvZyxub3F1b3RhIDAgMAovZGV2L3NkYTcgL2hvbWUgeGZzIHJ3LG5vYXRpbWUsYXR0cjIs
ZGVsYXlsb2csbm9xdW90YSAwIDAKL2Rldi9zZGE2IC92YXIvY2FjaGUvZWRiIHhmcyBydyxub2F0
aW1lLGF0dHIyLGRlbGF5bG9nLG5vcXVvdGEgMCAwCi9kZXYvc2RhNiAvdXNyL3NyYyB4ZnMgcncs
bm9hdGltZSxhdHRyMixkZWxheWxvZyxub3F1b3RhIDAgMAovZGV2L3NkYTYgL3Zhci90bXAgeGZz
IHJ3LG5vYXRpbWUsYXR0cjIsZGVsYXlsb2csbm9xdW90YSAwIDAKL2Rldi9zZGE2IC92YXIvbG9n
IHhmcyBydyxub2F0aW1lLGF0dHIyLGRlbGF5bG9nLG5vcXVvdGEgMCAwCi9kZXYvc2RhNiAvdmFy
L2xpYi9wb3J0YWdlL3BhY2thZ2VzIHhmcyBydyxub2F0aW1lLGF0dHIyLGRlbGF5bG9nLG5vcXVv
dGEgMCAwCgoqIHhmc19pbmZvICoKbWV0YS1kYXRhPS9kZXYvc2RhNyAgICAgICAgICAgICAgaXNp
emU9MjU2ICAgIGFnY291bnQ9NCwgYWdzaXplPTk4Nzk5NiBibGtzCiAgICAgICAgID0gICAgICAg
ICAgICAgICAgICAgICAgIHNlY3Rzej01MTIgICBhdHRyPTIKZGF0YSAgICAgPSAgICAgICAgICAg
ICAgICAgICAgICAgYnNpemU9NDA5NiAgIGJsb2Nrcz0zOTUxOTgyLCBpbWF4cGN0PTI1CiAgICAg
ICAgID0gICAgICAgICAgICAgICAgICAgICAgIHN1bml0PTAgICAgICBzd2lkdGg9MCBibGtzCm5h
bWluZyAgID12ZXJzaW9uIDIgICAgICAgICAgICAgIGJzaXplPTQwOTYgICBhc2NpaS1jaT0wCmxv
ZyAgICAgID1pbnRlcm5hbCAgICAgICAgICAgICAgIGJzaXplPTQwOTYgICBibG9ja3M9MjU2MCwg
dmVyc2lvbj0yCiAgICAgICAgID0gICAgICAgICAgICAgICAgICAgICAgIHNlY3Rzej01MTIgICBz
dW5pdD0wIGJsa3MsIGxhenktY291bnQ9MApyZWFsdGltZSA9bm9uZSAgICAgICAgICAgICAgICAg
ICBleHRzej00MDk2ICAgYmxvY2tzPTAsIHJ0ZXh0ZW50cz0wCgoKbWV0YS1kYXRhPS9kZXYvc2Rh
NiAgICAgICAgICAgICAgaXNpemU9MjU2ICAgIGFnY291bnQ9NCwgYWdzaXplPTY1NTE0OSBibGtz
CiAgICAgICAgID0gICAgICAgICAgICAgICAgICAgICAgIHNlY3Rzej01MTIgICBhdHRyPTIKZGF0
YSAgICAgPSAgICAgICAgICAgICAgICAgICAgICAgYnNpemU9NDA5NiAgIGJsb2Nrcz0yNjIwNTk1
LCBpbWF4cGN0PTI1CiAgICAgICAgID0gICAgICAgICAgICAgICAgICAgICAgIHN1bml0PTAgICAg
ICBzd2lkdGg9MCBibGtzCm5hbWluZyAgID12ZXJzaW9uIDIgICAgICAgICAgICAgIGJzaXplPTQw
OTYgICBhc2NpaS1jaT0wCmxvZyAgICAgID1pbnRlcm5hbCAgICAgICAgICAgICAgIGJzaXplPTQw
OTYgICBibG9ja3M9MjU2MCwgdmVyc2lvbj0yCiAgICAgICAgID0gICAgICAgICAgICAgICAgICAg
ICAgIHNlY3Rzej01MTIgICBzdW5pdD0wIGJsa3MsIGxhenktY291bnQ9MApyZWFsdGltZSA9bm9u
ZSAgICAgICAgICAgICAgICAgICBleHRzej00MDk2ICAgYmxvY2tzPTAsIHJ0ZXh0ZW50cz0wCgoK
QnJ1bm8KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCnhm
cyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5zZ2kuY29tCmh0dHA6Ly9vc3Muc2dpLmNvbS9tYWlsbWFu
L2xpc3RpbmZvL3hmcwo=