Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

["Darrick J. Wong" <darrick.wong@oracle.com>]

On Fri, Nov 22, 2019 at 10:17:33AM -0800, Blake Golliher wrote:
> Where would those core dumps be?  Are they automatically dumped or do we
> have to set a flag, then trigger the condition?

ulimit -c 9999999999, then run whatever it was you were running that
invokes xfs_db.

--D

> On Fri, Nov 22, 2019 at 8:12 AM Darrick J. Wong <darrick.wong@oracle.com>
> wrote:
> 
> >
> >
> > CAUTION: External Email
> >
> >
> >
> >
> > On Fri, Nov 22, 2019 at 09:08:26AM -0500, Andrew Carr wrote:
> > > Hi Dave  / Others,
> > >
> > > It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
> > > at least no deadlocks are occurring now.
> > >
> > > There are segfaults in xfs_db appearing now though.  I am attempting
> > > to get the full syslog, here is an example.... thoughts?
> > >
> > > [Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
> > > 0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
> > > [Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
> > > ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
> > > 24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
> > > 08 c1 e0 18 41 c1
> >
> > Actual coredumps of the crashed xfs_db would help.
> >
> > --D
> >
> > > Thanks so much in advance!
> > > Andrew
> > >
> > > On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com>
> > wrote:
> > > >
> > > > Genius Dave, Thanks so much!
> > > >
> > > > On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com>
> > wrote:
> > > > >
> > > > > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > > > > Dave / Eric / Others,
> > > > > >
> > > > > > Syslog:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_QYQYpPFY&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=cFo_9R18qcbqlKAa2jfsMB02h74aHd4m04zbNPYS1-I&e=
> > > > > >
> > > > > > Dmesg:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_MdBCPmp9&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=E9ryV4GnH02exSAsoFGbq1arjRLkyffNjka_kZ4MV60&e=
> > > > >
> > > > > which shows no stack traces, again.
> > > > >
> > > > >
> > > > >
> > > > > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > > > > is the likely cause of the issue: the filesystem is using 64k
> > > > > directory blocks - the allocation size is larger than 64kB:
> > > > >
> > > > > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory
> > allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> > > > >
> > > > > Upstream fixed this some time ago:
> > > > >
> > > > > $ ▶ gl -n 1 -p cb0a8d23024e
> > > > > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > > > > Author: Dave Chinner <dchinner@redhat.com>
> > > > > Date:   Tue Mar 6 17:03:28 2018 -0800
> > > > >
> > > > >     xfs: fall back to vmalloc when allocation log vector buffers
> > > > >
> > > > >     When using large directory blocks, we regularly see memory
> > > > >     allocations of >64k being made for the shadow log vector buffer.
> > > > >     When we are under memory pressure, kmalloc() may not be able to
> > find
> > > > >     contiguous memory chunks large enough to satisfy these
> > allocations
> > > > >     easily, and if memory is fragmented we can potentially stall
> > here.
> > > > >
> > > > >     TO avoid this problem, switch the log vector buffer allocation to
> > > > >     use kmem_alloc_large(). This will allow failed allocations to
> > fall
> > > > >     back to vmalloc and so remove the dependency on large contiguous
> > > > >     regions of memory being available. This should prevent slowdowns
> > > > >     and potential stalls when memory is low and/or fragmented.
> > > > >
> > > > >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> > > > >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Dave.
> > > > > --
> > > > > Dave Chinner
> > > > > david@fromorbit.com
> > > >
> > > >
> > > >
> > > > --
> > > > With Regards,
> > > > Andrew Carr
> > > >
> > > > e. andrewlanecarr@gmail.com
> > > > w. andrew.carr@openlogic.com
> > > > c. 4239489206
> > > > a. P.O. Box 1231, Greeneville, TN, 37744
> > >
> > >
> > >
> > > --
> > > With Regards,
> > > Andrew Carr
> > >
> > > e. andrewlanecarr@gmail.com
> > > w. andrew.carr@openlogic.com
> > > c. 4239489206
> > > a. P.O. Box 1231, Greeneville, TN, 37744
> >