Linux-XFS Archive on lore.kernel.org
 help / color / Atom feed
* Fwd: XFS Memory allocation deadlock in kmem_alloc
       [not found] <CAKQeeLMxJR-ToX5HG9Q-z0-AL9vZG-OMjHyM+rnEEBP6k6nxHw@mail.gmail.com>
@ 2019-11-15 19:11 ` Andrew Carr
  2019-11-15 19:52   ` Eric Sandeen
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Carr @ 2019-11-15 19:11 UTC (permalink / raw)
  To: linux-xfs

Hello,

This list has recommended enabling stack traces to determine the root
cause of issues with XFS deadlocks occurring in Centos 7.7
(3.10.0-1062).

Based on what was recommended by Eric Sandeen, we have tried updating
the following files to generate XFS stack traces:

# echo 11 > /proc/sys/fs/xfs/error_level

And

# echo 3 > /proc/sys/fs/xfs/error_level

But no stack traces are printed to dmesg.  I was thinking of
re-compiling the kernel with debug flags enabled.  Do you think this
is necessary?

Thanks so much for your time and keep up the good work!

Sincerely,
--
Andrew Carr | Enterprise Architect
Rogue Wave Software, Inc.
Innovate with Confidence
P 720.295.8044
www.roguewave.com | andrew.carr@roguewave.com


--
With Regards,
Andrew Carr

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-15 19:11 ` Fwd: XFS Memory allocation deadlock in kmem_alloc Andrew Carr
@ 2019-11-15 19:52   ` Eric Sandeen
  2019-11-15 23:43     ` Dave Chinner
  0 siblings, 1 reply; 10+ messages in thread
From: Eric Sandeen @ 2019-11-15 19:52 UTC (permalink / raw)
  To: Andrew Carr, linux-xfs

On 11/15/19 1:11 PM, Andrew Carr wrote:
> Hello,
> 
> This list has recommended enabling stack traces to determine the root
> cause of issues with XFS deadlocks occurring in Centos 7.7
> (3.10.0-1062).
> 
> Based on what was recommended by Eric Sandeen, we have tried updating
> the following files to generate XFS stack traces:
> 
> # echo 11 > /proc/sys/fs/xfs/error_level
> 
> And
> 
> # echo 3 > /proc/sys/fs/xfs/error_level
> 
> But no stack traces are printed to dmesg.  I was thinking of
> re-compiling the kernel with debug flags enabled.  Do you think this
> is necessary?
> 
> Thanks so much for your time and keep up the good work!

I've looked over the way xfs_err() gets defined, and I cannot see how
we can call xfs_err with error_level == 11 and not get a stack trace.

Maybe other eyes can spot something...

-Eric

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-15 19:52   ` Eric Sandeen
@ 2019-11-15 23:43     ` Dave Chinner
  2019-11-16 16:19       ` Andrew Carr
  0 siblings, 1 reply; 10+ messages in thread
From: Dave Chinner @ 2019-11-15 23:43 UTC (permalink / raw)
  To: Eric Sandeen; +Cc: Andrew Carr, linux-xfs

On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> On 11/15/19 1:11 PM, Andrew Carr wrote:
> > Hello,
> > 
> > This list has recommended enabling stack traces to determine the root
> > cause of issues with XFS deadlocks occurring in Centos 7.7
> > (3.10.0-1062).
> > 
> > Based on what was recommended by Eric Sandeen, we have tried updating
> > the following files to generate XFS stack traces:
> > 
> > # echo 11 > /proc/sys/fs/xfs/error_level
> > 
> > And
> > 
> > # echo 3 > /proc/sys/fs/xfs/error_level
> > 
> > But no stack traces are printed to dmesg.  I was thinking of
> > re-compiling the kernel with debug flags enabled.  Do you think this
> > is necessary?

dmesg -n 7 will remove all filters on the console/dmesg output - if
you've utrned this down in the past you may not be seeing messages
of the error level XFS is using...

Did you check syslog - that should have all the unfiltered messages
in it...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-15 23:43     ` Dave Chinner
@ 2019-11-16 16:19       ` Andrew Carr
  2019-11-19 15:49         ` Andrew Carr
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Carr @ 2019-11-16 16:19 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, linux-xfs

Thanks Dave,
Checking now.

On Fri, Nov 15, 2019 at 6:43 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> > On 11/15/19 1:11 PM, Andrew Carr wrote:
> > > Hello,
> > >
> > > This list has recommended enabling stack traces to determine the root
> > > cause of issues with XFS deadlocks occurring in Centos 7.7
> > > (3.10.0-1062).
> > >
> > > Based on what was recommended by Eric Sandeen, we have tried updating
> > > the following files to generate XFS stack traces:
> > >
> > > # echo 11 > /proc/sys/fs/xfs/error_level
> > >
> > > And
> > >
> > > # echo 3 > /proc/sys/fs/xfs/error_level
> > >
> > > But no stack traces are printed to dmesg.  I was thinking of
> > > re-compiling the kernel with debug flags enabled.  Do you think this
> > > is necessary?
>
> dmesg -n 7 will remove all filters on the console/dmesg output - if
> you've utrned this down in the past you may not be seeing messages
> of the error level XFS is using...
>
> Did you check syslog - that should have all the unfiltered messages
> in it...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-16 16:19       ` Andrew Carr
@ 2019-11-19 15:49         ` Andrew Carr
  2019-11-19 20:20           ` Dave Chinner
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Carr @ 2019-11-19 15:49 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, linux-xfs

Dave / Eric / Others,

Syslog: https://pastebin.com/QYQYpPFY

Dmesg: https://pastebin.com/MdBCPmp9

-Andrew Carr


On Sat, Nov 16, 2019 at 11:19 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
>
> Thanks Dave,
> Checking now.
>
> On Fri, Nov 15, 2019 at 6:43 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> > > On 11/15/19 1:11 PM, Andrew Carr wrote:
> > > > Hello,
> > > >
> > > > This list has recommended enabling stack traces to determine the root
> > > > cause of issues with XFS deadlocks occurring in Centos 7.7
> > > > (3.10.0-1062).
> > > >
> > > > Based on what was recommended by Eric Sandeen, we have tried updating
> > > > the following files to generate XFS stack traces:
> > > >
> > > > # echo 11 > /proc/sys/fs/xfs/error_level
> > > >
> > > > And
> > > >
> > > > # echo 3 > /proc/sys/fs/xfs/error_level
> > > >
> > > > But no stack traces are printed to dmesg.  I was thinking of
> > > > re-compiling the kernel with debug flags enabled.  Do you think this
> > > > is necessary?
> >
> > dmesg -n 7 will remove all filters on the console/dmesg output - if
> > you've utrned this down in the past you may not be seeing messages
> > of the error level XFS is using...
> >
> > Did you check syslog - that should have all the unfiltered messages
> > in it...
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
>
>
>
> --
> With Regards,
> Andrew Carr
>
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-19 15:49         ` Andrew Carr
@ 2019-11-19 20:20           ` Dave Chinner
  2019-11-20 15:43             ` Andrew Carr
  0 siblings, 1 reply; 10+ messages in thread
From: Dave Chinner @ 2019-11-19 20:20 UTC (permalink / raw)
  To: Andrew Carr; +Cc: Eric Sandeen, linux-xfs

On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> Dave / Eric / Others,
> 
> Syslog: https://pastebin.com/QYQYpPFY
> 
> Dmesg: https://pastebin.com/MdBCPmp9

which shows no stack traces, again.



Anyway, you've twiddled mkfs knobs on these filesystems, and that
is the likely cause of the issue: the filesystem is using 64k
directory blocks - the allocation size is larger than 64kB:

[Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)

Upstream fixed this some time ago:

$ ▶ gl -n 1 -p cb0a8d23024e
commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
Author: Dave Chinner <dchinner@redhat.com>
Date:   Tue Mar 6 17:03:28 2018 -0800

    xfs: fall back to vmalloc when allocation log vector buffers
    
    When using large directory blocks, we regularly see memory
    allocations of >64k being made for the shadow log vector buffer.
    When we are under memory pressure, kmalloc() may not be able to find
    contiguous memory chunks large enough to satisfy these allocations
    easily, and if memory is fragmented we can potentially stall here.
    
    TO avoid this problem, switch the log vector buffer allocation to
    use kmem_alloc_large(). This will allow failed allocations to fall
    back to vmalloc and so remove the dependency on large contiguous
    regions of memory being available. This should prevent slowdowns
    and potential stalls when memory is low and/or fragmented.
    
    Signed-Off-By: Dave Chinner <dchinner@redhat.com>
    Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>


Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-19 20:20           ` Dave Chinner
@ 2019-11-20 15:43             ` Andrew Carr
  2019-11-22 14:08               ` Andrew Carr
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Carr @ 2019-11-20 15:43 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, linux-xfs

Genius Dave, Thanks so much!

On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > Dave / Eric / Others,
> >
> > Syslog: https://pastebin.com/QYQYpPFY
> >
> > Dmesg: https://pastebin.com/MdBCPmp9
>
> which shows no stack traces, again.
>
>
>
> Anyway, you've twiddled mkfs knobs on these filesystems, and that
> is the likely cause of the issue: the filesystem is using 64k
> directory blocks - the allocation size is larger than 64kB:
>
> [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
>
> Upstream fixed this some time ago:
>
> $ ▶ gl -n 1 -p cb0a8d23024e
> commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> Author: Dave Chinner <dchinner@redhat.com>
> Date:   Tue Mar 6 17:03:28 2018 -0800
>
>     xfs: fall back to vmalloc when allocation log vector buffers
>
>     When using large directory blocks, we regularly see memory
>     allocations of >64k being made for the shadow log vector buffer.
>     When we are under memory pressure, kmalloc() may not be able to find
>     contiguous memory chunks large enough to satisfy these allocations
>     easily, and if memory is fragmented we can potentially stall here.
>
>     TO avoid this problem, switch the log vector buffer allocation to
>     use kmem_alloc_large(). This will allow failed allocations to fall
>     back to vmalloc and so remove the dependency on large contiguous
>     regions of memory being available. This should prevent slowdowns
>     and potential stalls when memory is low and/or fragmented.
>
>     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
>     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-20 15:43             ` Andrew Carr
@ 2019-11-22 14:08               ` Andrew Carr
  2019-11-22 16:12                 ` Darrick J. Wong
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Carr @ 2019-11-22 14:08 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, linux-xfs

Hi Dave  / Others,

It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
at least no deadlocks are occurring now.

There are segfaults in xfs_db appearing now though.  I am attempting
to get the full syslog, here is an example.... thoughts?

[Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
[Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
08 c1 e0 18 41 c1

Thanks so much in advance!
Andrew

On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
>
> Genius Dave, Thanks so much!
>
> On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > Dave / Eric / Others,
> > >
> > > Syslog: https://pastebin.com/QYQYpPFY
> > >
> > > Dmesg: https://pastebin.com/MdBCPmp9
> >
> > which shows no stack traces, again.
> >
> >
> >
> > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > is the likely cause of the issue: the filesystem is using 64k
> > directory blocks - the allocation size is larger than 64kB:
> >
> > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> >
> > Upstream fixed this some time ago:
> >
> > $ ▶ gl -n 1 -p cb0a8d23024e
> > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > Author: Dave Chinner <dchinner@redhat.com>
> > Date:   Tue Mar 6 17:03:28 2018 -0800
> >
> >     xfs: fall back to vmalloc when allocation log vector buffers
> >
> >     When using large directory blocks, we regularly see memory
> >     allocations of >64k being made for the shadow log vector buffer.
> >     When we are under memory pressure, kmalloc() may not be able to find
> >     contiguous memory chunks large enough to satisfy these allocations
> >     easily, and if memory is fragmented we can potentially stall here.
> >
> >     TO avoid this problem, switch the log vector buffer allocation to
> >     use kmem_alloc_large(). This will allow failed allocations to fall
> >     back to vmalloc and so remove the dependency on large contiguous
> >     regions of memory being available. This should prevent slowdowns
> >     and potential stalls when memory is low and/or fragmented.
> >
> >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> >
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
>
>
>
> --
> With Regards,
> Andrew Carr
>
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
  2019-11-22 14:08               ` Andrew Carr
@ 2019-11-22 16:12                 ` Darrick J. Wong
       [not found]                   ` <CAC752A=7x+gh9Jr8-koQtuZDvMzrs6qRc+saj=TMC3js9EdHbg@mail.gmail.com>
  0 siblings, 1 reply; 10+ messages in thread
From: Darrick J. Wong @ 2019-11-22 16:12 UTC (permalink / raw)
  To: Andrew Carr; +Cc: Dave Chinner, Eric Sandeen, linux-xfs

On Fri, Nov 22, 2019 at 09:08:26AM -0500, Andrew Carr wrote:
> Hi Dave  / Others,
> 
> It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
> at least no deadlocks are occurring now.
> 
> There are segfaults in xfs_db appearing now though.  I am attempting
> to get the full syslog, here is an example.... thoughts?
> 
> [Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
> 0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
> [Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
> ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
> 24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
> 08 c1 e0 18 41 c1

Actual coredumps of the crashed xfs_db would help.

--D

> Thanks so much in advance!
> Andrew
> 
> On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
> >
> > Genius Dave, Thanks so much!
> >
> > On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > > Dave / Eric / Others,
> > > >
> > > > Syslog: https://pastebin.com/QYQYpPFY
> > > >
> > > > Dmesg: https://pastebin.com/MdBCPmp9
> > >
> > > which shows no stack traces, again.
> > >
> > >
> > >
> > > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > > is the likely cause of the issue: the filesystem is using 64k
> > > directory blocks - the allocation size is larger than 64kB:
> > >
> > > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> > >
> > > Upstream fixed this some time ago:
> > >
> > > $ ▶ gl -n 1 -p cb0a8d23024e
> > > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > > Author: Dave Chinner <dchinner@redhat.com>
> > > Date:   Tue Mar 6 17:03:28 2018 -0800
> > >
> > >     xfs: fall back to vmalloc when allocation log vector buffers
> > >
> > >     When using large directory blocks, we regularly see memory
> > >     allocations of >64k being made for the shadow log vector buffer.
> > >     When we are under memory pressure, kmalloc() may not be able to find
> > >     contiguous memory chunks large enough to satisfy these allocations
> > >     easily, and if memory is fragmented we can potentially stall here.
> > >
> > >     TO avoid this problem, switch the log vector buffer allocation to
> > >     use kmem_alloc_large(). This will allow failed allocations to fall
> > >     back to vmalloc and so remove the dependency on large contiguous
> > >     regions of memory being available. This should prevent slowdowns
> > >     and potential stalls when memory is low and/or fragmented.
> > >
> > >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> > >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> > >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > >
> > >
> > > Cheers,
> > >
> > > Dave.
> > > --
> > > Dave Chinner
> > > david@fromorbit.com
> >
> >
> >
> > --
> > With Regards,
> > Andrew Carr
> >
> > e. andrewlanecarr@gmail.com
> > w. andrew.carr@openlogic.com
> > c. 4239489206
> > a. P.O. Box 1231, Greeneville, TN, 37744
> 
> 
> 
> -- 
> With Regards,
> Andrew Carr
> 
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Fwd: XFS Memory allocation deadlock in kmem_alloc
       [not found]                   ` <CAC752A=7x+gh9Jr8-koQtuZDvMzrs6qRc+saj=TMC3js9EdHbg@mail.gmail.com>
@ 2019-11-22 18:49                     ` Darrick J. Wong
  0 siblings, 0 replies; 10+ messages in thread
From: Darrick J. Wong @ 2019-11-22 18:49 UTC (permalink / raw)
  To: Blake Golliher; +Cc: Andrew Carr, Dave Chinner, Eric Sandeen, linux-xfs

On Fri, Nov 22, 2019 at 10:17:33AM -0800, Blake Golliher wrote:
> Where would those core dumps be?  Are they automatically dumped or do we
> have to set a flag, then trigger the condition?

ulimit -c 9999999999, then run whatever it was you were running that
invokes xfs_db.

--D

> On Fri, Nov 22, 2019 at 8:12 AM Darrick J. Wong <darrick.wong@oracle.com>
> wrote:
> 
> >
> >
> > CAUTION: External Email
> >
> >
> >
> >
> > On Fri, Nov 22, 2019 at 09:08:26AM -0500, Andrew Carr wrote:
> > > Hi Dave  / Others,
> > >
> > > It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
> > > at least no deadlocks are occurring now.
> > >
> > > There are segfaults in xfs_db appearing now though.  I am attempting
> > > to get the full syslog, here is an example.... thoughts?
> > >
> > > [Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
> > > 0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
> > > [Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
> > > ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
> > > 24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
> > > 08 c1 e0 18 41 c1
> >
> > Actual coredumps of the crashed xfs_db would help.
> >
> > --D
> >
> > > Thanks so much in advance!
> > > Andrew
> > >
> > > On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com>
> > wrote:
> > > >
> > > > Genius Dave, Thanks so much!
> > > >
> > > > On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com>
> > wrote:
> > > > >
> > > > > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > > > > Dave / Eric / Others,
> > > > > >
> > > > > > Syslog:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_QYQYpPFY&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=cFo_9R18qcbqlKAa2jfsMB02h74aHd4m04zbNPYS1-I&e=
> > > > > >
> > > > > > Dmesg:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_MdBCPmp9&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=E9ryV4GnH02exSAsoFGbq1arjRLkyffNjka_kZ4MV60&e=
> > > > >
> > > > > which shows no stack traces, again.
> > > > >
> > > > >
> > > > >
> > > > > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > > > > is the likely cause of the issue: the filesystem is using 64k
> > > > > directory blocks - the allocation size is larger than 64kB:
> > > > >
> > > > > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory
> > allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> > > > >
> > > > > Upstream fixed this some time ago:
> > > > >
> > > > > $ ▶ gl -n 1 -p cb0a8d23024e
> > > > > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > > > > Author: Dave Chinner <dchinner@redhat.com>
> > > > > Date:   Tue Mar 6 17:03:28 2018 -0800
> > > > >
> > > > >     xfs: fall back to vmalloc when allocation log vector buffers
> > > > >
> > > > >     When using large directory blocks, we regularly see memory
> > > > >     allocations of >64k being made for the shadow log vector buffer.
> > > > >     When we are under memory pressure, kmalloc() may not be able to
> > find
> > > > >     contiguous memory chunks large enough to satisfy these
> > allocations
> > > > >     easily, and if memory is fragmented we can potentially stall
> > here.
> > > > >
> > > > >     TO avoid this problem, switch the log vector buffer allocation to
> > > > >     use kmem_alloc_large(). This will allow failed allocations to
> > fall
> > > > >     back to vmalloc and so remove the dependency on large contiguous
> > > > >     regions of memory being available. This should prevent slowdowns
> > > > >     and potential stalls when memory is low and/or fragmented.
> > > > >
> > > > >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> > > > >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Dave.
> > > > > --
> > > > > Dave Chinner
> > > > > david@fromorbit.com
> > > >
> > > >
> > > >
> > > > --
> > > > With Regards,
> > > > Andrew Carr
> > > >
> > > > e. andrewlanecarr@gmail.com
> > > > w. andrew.carr@openlogic.com
> > > > c. 4239489206
> > > > a. P.O. Box 1231, Greeneville, TN, 37744
> > >
> > >
> > >
> > > --
> > > With Regards,
> > > Andrew Carr
> > >
> > > e. andrewlanecarr@gmail.com
> > > w. andrew.carr@openlogic.com
> > > c. 4239489206
> > > a. P.O. Box 1231, Greeneville, TN, 37744
> >

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CAKQeeLMxJR-ToX5HG9Q-z0-AL9vZG-OMjHyM+rnEEBP6k6nxHw@mail.gmail.com>
2019-11-15 19:11 ` Fwd: XFS Memory allocation deadlock in kmem_alloc Andrew Carr
2019-11-15 19:52   ` Eric Sandeen
2019-11-15 23:43     ` Dave Chinner
2019-11-16 16:19       ` Andrew Carr
2019-11-19 15:49         ` Andrew Carr
2019-11-19 20:20           ` Dave Chinner
2019-11-20 15:43             ` Andrew Carr
2019-11-22 14:08               ` Andrew Carr
2019-11-22 16:12                 ` Darrick J. Wong
     [not found]                   ` <CAC752A=7x+gh9Jr8-koQtuZDvMzrs6qRc+saj=TMC3js9EdHbg@mail.gmail.com>
2019-11-22 18:49                     ` Darrick J. Wong

Linux-XFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-xfs/0 linux-xfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-xfs linux-xfs/ https://lore.kernel.org/linux-xfs \
		linux-xfs@vger.kernel.org
	public-inbox-index linux-xfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-xfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git