3.5.0-rc5: inconsistent lock state

3.5.0-rc5: inconsistent lock state

[Christian Kujau]

Hi,

this powerpc machine is running 3.5.0-rc5 for 2 days now and this happened 
today:

=================================
[ INFO: inconsistent lock state ]
3.5.0-rc5-00003-gca24a14-dirty #1 Not tainted
---------------------------------
inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
kswapd0/237 [HC0[0]:SC0[0]:HE1:SE1] takes:
 (xfs_iolock_reclaimable){+.+.?.}, at: [<c01d0a24>] xfs_ilock+0x84/0xc0
{RECLAIM_FS-ON-W} state was registered at:
  [<c00766e8>] lockdep_trace_alloc+0xac/0xe4
  [<c00d33e0>] kmem_cache_alloc+0x3c/0x134
  [<c01e25dc>] kmem_zone_alloc+0xa4/0x114
  [<c01e2668>] kmem_zone_zalloc+0x1c/0x58
  [<c0221ca4>] _xfs_trans_alloc+0x40/0x84
  [<c0222a0c>] xfs_trans_alloc+0xa0/0xbc
  [<c01e7148>] xfs_attr_inactive+0x5c/0x1a4
  [<c01e19d8>] xfs_inactive_attrs+0x58/0x10c
  [<c01e1c74>] xfs_inactive+0x1e8/0x440
  [<c01da5e4>] xfs_fs_evict_inode+0x98/0xac
  [<c00f5a90>] evict+0xc8/0x1b0
  [<c00f20d0>] d_delete+0x1cc/0x1f8
  [<c00e8100>] vfs_rmdir+0x138/0x13c
  [<c00e8220>] do_rmdir+0x11c/0x12c
  [<c00117e8>] ret_from_syscall+0x0/0x38
irq event stamp: 183359685
hardirqs last  enabled at (183359685): [<c0539f20>] _raw_spin_unlock_irqrestore+0x40/0x9c
hardirqs last disabled at (183359684): [<c053952c>] _raw_spin_lock_irqsave+0x30/0x84
softirqs last  enabled at (183358622): [<c000f3e8>] call_do_softirq+0x14/0x24
softirqs last disabled at (183358603): [<c000f3e8>] call_do_softirq+0x14/0x24


My mount options:

# grep xfs /proc/mounts 
/dev/mapper/wdc1 /mnt/media xfs rw,nosuid,nodev,noexec,relatime,attr2,noquota 0 0
/dev/mapper/sam0 /mnt/vault0 xfs ro,nosuid,nodev,noexec,relatime,attr2,noquota 0 0
/dev/mapper/sam1 /mnt/vault1 xfs ro,nosuid,nodev,noexec,relatime,attr2,noquota 0 0

The whole dmesg & .config is here: 

  http://nerdbynature.de/bits/3.5.0-rc5/soft_lockup/xfs/

Note: the -dirty flag in the kernel version comes from a single patch 
applied so that the system would boot: https://lkml.org/lkml/2012/7/2/32

Thanks,
Christian.
-- 
BOFH excuse #339:

manager in the cable duct

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Christian Kujau]

On Wed, 4 Jul 2012 at 12:13, Christian Kujau wrote:
> The whole dmesg & .config is here: 
>   http://nerdbynature.de/bits/3.5.0-rc5/soft_lockup/xfs/

Sorry, this should read: 

  http://nerdbynature.de/bits/3.5.0-rc5/xfs/

C.
-- 
BOFH excuse #339:

manager in the cable duct

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Christoph Hellwig]

On Wed, Jul 04, 2012 at 12:13:32PM -0700, Christian Kujau wrote:
> Hi,
> 
> this powerpc machine is running 3.5.0-rc5 for 2 days now and this happened 
> today:

See my "do not take the iolock in inode reclaim context" series from
yesterday, which should take care of this.

Btw, if you hit this it's a sign you have out of the inode attributes,
so if this isn't just a single inode with them you might be better off
using larger inodes.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Christian Kujau]

On Thu, 5 Jul 2012 at 03:43, Christoph Hellwig wrote:
> See my "do not take the iolock in inode reclaim context" series from
> yesterday, which should take care of this.

OK, good. I'll wait until this hits mainline then.

> Btw, if you hit this it's a sign you have out of the inode attributes,
> so if this isn't just a single inode with them you might be better off
> using larger inodes.

..."(run) out of inode attributes" - not really sure what that means and 
why that would be the case.

And "using larger inodes" sounds like a 
reformat with a larger "-i size=" option. Is there a way to find out my 
current inode size? "mkfs.xfs -N" does not print its default values on a 
mounted filesystem and I'm pretty sure I've used the default values when I 
created the filesystem a while ago (i.e. just plain "mkfs.xfs" w/o any 
options)

Thanks,
Christian.
-- 
BOFH excuse #109:

The electricity substation in the car park blew up.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Dave Chinner]

On Thu, Jul 05, 2012 at 05:44:50AM -0700, Christian Kujau wrote:
> On Thu, 5 Jul 2012 at 03:43, Christoph Hellwig wrote:
> > See my "do not take the iolock in inode reclaim context" series from
> > yesterday, which should take care of this.
> 
> OK, good. I'll wait until this hits mainline then.
> 
> > Btw, if you hit this it's a sign you have out of the inode attributes,
> > so if this isn't just a single inode with them you might be better off
> > using larger inodes.
> 
> ..."(run) out of inode attributes" - not really sure what that means and 
> why that would be the case.

It means that you have enough attributes that they don't fit in the
inode, so every time they are read or written you have to do an
extra IO on top of reading/writing the inode. Performance can easily
drop by an order of magnitude when the attributes are moved out of
the inode....

Typically there is 50-70 bytes of attribute space available in 256
byte inodes, larger attributes or lots of them will push them out of
the inode....

> And "using larger inodes" sounds like a 
> reformat with a larger "-i size=" option.

Yes, inode size is defined at mkfs time.

> Is there a way to find out my 
> current inode size? "mkfs.xfs -N" does not print its default values on a 
> mounted filesystem and I'm pretty sure I've used the default values when I 
> created the filesystem a while ago (i.e. just plain "mkfs.xfs" w/o any 
> options)

xfs_info <mntpt>

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Christian Kujau]

On Fri, 6 Jul 2012 at 07:59, Dave Chinner wrote:
> It means that you have enough attributes that they don't fit in the
> inode, so every time they are read or written you have to do an
> extra IO on top of reading/writing the inode. Performance can easily
> drop by an order of magnitude when the attributes are moved out of
> the inode....

xfs_info shows isize=256 - but I'm not sure how I would have exceeded that 
limit? I'm not using SELinux or anhy other security frameworks on that 
machine, only plain unix permissions. Just check again, no ACLs, no EAs, 
no file attributes are set on these filesystems.

The filesystems make heavy use of hardlinks, but files usually have no
more than ~12 hardlinks, so that counter should not exceed the inode
size either.

> Typically there is 50-70 bytes of attribute space available in 256
> byte inodes, larger attributes or lots of them will push them out of
> the inode....

50 bytes sounds more than enought for holding only unix permissions. Does 
it matter that the filesystem is somewhat larger? Not too large though, 
all xfs filesystems are < 1TB in size.

Thanks,
Christian.
-- 
BOFH excuse #211:

Lightning strikes.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Dave Chinner]

On Thu, Jul 05, 2012 at 04:39:21PM -0700, Christian Kujau wrote:
> On Fri, 6 Jul 2012 at 07:59, Dave Chinner wrote:
> > It means that you have enough attributes that they don't fit in the
> > inode, so every time they are read or written you have to do an
> > extra IO on top of reading/writing the inode. Performance can easily
> > drop by an order of magnitude when the attributes are moved out of
> > the inode....
> 
> xfs_info shows isize=256 - but I'm not sure how I would have exceeded that 
> limit? I'm not using SELinux or anhy other security frameworks on that 
> machine, only plain unix permissions. Just check again, no ACLs, no EAs, 
> no file attributes are set on these filesystems.

Applications can use attributes without you being aware of them.
e.g. Samba, desktop search/indexing, etc might be using attributes
even though you aren't explicitly using them....

> The filesystems make heavy use of hardlinks, but files usually have no
> more than ~12 hardlinks, so that counter should not exceed the inode
> size either.

Hardlinks are not attributes, and the counter is in the inode core
so this won't have any impact on attributes being places out of
line.

> > Typically there is 50-70 bytes of attribute space available in 256
> > byte inodes, larger attributes or lots of them will push them out of
> > the inode....
> 
> 50 bytes sounds more than enought for holding only unix permissions.

Unix permissions are held in the inode core, not in the attribute
space.

And i did say "typically". if you have a file that has 6-7 extents,
then there won't be any space for attributes and it will put new
attributes out of line immediately....

> Does 
> it matter that the filesystem is somewhat larger? Not too large though, 
> all xfs filesystems are < 1TB in size.

No.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3.5.0-rc5: inconsistent lock state

[Christian Kujau]

On Fri, 6 Jul 2012 at 11:01, Dave Chinner wrote:
> Applications can use attributes without you being aware of them.
> e.g. Samba, desktop search/indexing, etc might be using attributes
> even though you aren't explicitly using them....

Hm, it's some kind of server, samba is serving those mounts, but only as 
readonly shares. When running "getfattr -R" or "getfacl -R" on these xfs 
filesystems, only very few (<100) file have attribtues / ACLs set.

> > 50 bytes sounds more than enought for holding only unix permissions.
> Unix permissions are held in the inode core, not in the attribute
> space.

Even better, so I'm really at a loss what could use up this attribute 
space.


> And i did say "typically". if you have a file that has 6-7 extents,
> then there won't be any space for attributes and it will put new
> attributes out of line immediately....

$ xfs_bmap foo | wc -l
933

And there are lots of files held by lots of extents, so this might be it.

Thanks for this guide through this attribute business. There's currently 
no problem with any of the filesystems but I was tipped off by Christoph's 
remark about "running out of inode attributes" which got me scared a 
little. I understand that it might be a performance issue having to do 
an extra lookup for attributes, but I still fail to see how (and where) 
they sneaked into the fs.

Christian.
-- 
BOFH excuse #284:

Electrons on a bender

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs