All of lore.kernel.org
 help / color / mirror / Atom feed
From: Michal Hocko <mhocko@kernel.org>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: linux-xfs@vger.kernel.org, linux-mm@kvack.org
Subject: Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?
Date: Fri, 3 Mar 2017 14:39:51 +0100	[thread overview]
Message-ID: <20170303133950.GD31582@dhcp22.suse.cz> (raw)
In-Reply-To: <201703031948.CHJ81278.VOHSFFFOOLJQMt@I-love.SAKURA.ne.jp>

On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> 
> While I was testing a patch which avoids infinite too_many_isolated() loop in
> shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> waiting for memory allocation. I guess that we overlooked a basic thing about
> WQ_MEM_RECLAIM.
> 
>   WQ_MEM_RECLAIM helps only when the cause of failing to complete
>   a work item is lack of "struct task_struct" to run that work item, for
>   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
>   will not be blocked waiting for memory allocation for "struct task_struct".
> 
>   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
>   item is blocked waiting for memory allocation (or is indirectly blocked
>   on a lock where the owner of that lock is blocked waiting for memory
>   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
>   if memory allocation (including indirect memory allocation via
>   locks/completions) is needed.
> 
> In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> 
> What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> workqueues (which are RESCUER) got stuck waiting for memory allocation.

If those workers are really required for a further progress of the
memory reclaim then they shouldn't block on allocation at all and either
use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
that only very limited amount of memory is allocated from that context
and there will be at least the same amount of memory freed as a result
in a reasonable time.

This is something for xfs people to answer though. Please note that I
didn't really have time to look through the below traces so the above
note is rather generic. It would be really helpful if you could provide
a high level dependency chains to see why those rescuers are necessary
for the forward progress because it is really easy to get lost in so
many traces.
-- 
Michal Hocko
SUSE Labs

WARNING: multiple messages have this Message-ID
From: Michal Hocko <mhocko@kernel.org>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: linux-xfs@vger.kernel.org, linux-mm@kvack.org
Subject: Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?
Date: Fri, 3 Mar 2017 14:39:51 +0100	[thread overview]
Message-ID: <20170303133950.GD31582@dhcp22.suse.cz> (raw)
In-Reply-To: <201703031948.CHJ81278.VOHSFFFOOLJQMt@I-love.SAKURA.ne.jp>

On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> 
> While I was testing a patch which avoids infinite too_many_isolated() loop in
> shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> waiting for memory allocation. I guess that we overlooked a basic thing about
> WQ_MEM_RECLAIM.
> 
>   WQ_MEM_RECLAIM helps only when the cause of failing to complete
>   a work item is lack of "struct task_struct" to run that work item, for
>   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
>   will not be blocked waiting for memory allocation for "struct task_struct".
> 
>   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
>   item is blocked waiting for memory allocation (or is indirectly blocked
>   on a lock where the owner of that lock is blocked waiting for memory
>   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
>   if memory allocation (including indirect memory allocation via
>   locks/completions) is needed.
> 
> In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> 
> What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> workqueues (which are RESCUER) got stuck waiting for memory allocation.

If those workers are really required for a further progress of the
memory reclaim then they shouldn't block on allocation at all and either
use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
that only very limited amount of memory is allocated from that context
and there will be at least the same amount of memory freed as a result
in a reasonable time.

This is something for xfs people to answer though. Please note that I
didn't really have time to look through the below traces so the above
note is rather generic. It would be really helpful if you could provide
a high level dependency chains to see why those rescuers are necessary
for the forward progress because it is really easy to get lost in so
many traces.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  reply	other threads:[~2017-03-03 13:39 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-03-03 10:48 How to favor memory allocations for WQ_MEM_RECLAIM threads? Tetsuo Handa
2017-03-03 10:48 ` Tetsuo Handa
2017-03-03 13:39 ` Michal Hocko [this message]
2017-03-03 13:39   ` Michal Hocko
2017-03-03 15:37   ` Brian Foster
2017-03-03 15:37     ` Brian Foster
2017-03-03 15:52     ` Michal Hocko
2017-03-03 15:52       ` Michal Hocko
2017-03-03 17:29       ` Brian Foster
2017-03-03 17:29         ` Brian Foster
2017-03-04 14:54         ` Tetsuo Handa
2017-03-04 14:54           ` Tetsuo Handa
2017-03-06 13:25           ` Brian Foster
2017-03-06 13:25             ` Brian Foster
2017-03-06 16:08             ` Tetsuo Handa
2017-03-06 16:08               ` Tetsuo Handa
2017-03-06 16:17               ` Brian Foster
2017-03-06 16:17                 ` Brian Foster
2017-03-03 23:25   ` Dave Chinner
2017-03-03 23:25     ` Dave Chinner
2017-03-07 12:15     ` Michal Hocko
2017-03-07 12:15       ` Michal Hocko
2017-03-07 19:36       ` Tejun Heo
2017-03-07 19:36         ` Tejun Heo
2017-03-07 21:21         ` Dave Chinner
2017-03-07 21:21           ` Dave Chinner
2017-03-07 21:48           ` Tejun Heo
2017-03-07 21:48             ` Tejun Heo
2017-03-08 23:03             ` Tejun Heo
2017-03-08 23:03               ` Tejun Heo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170303133950.GD31582@dhcp22.suse.cz \
    --to=mhocko@kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-xfs@vger.kernel.org \
    --cc=penguin-kernel@I-love.SAKURA.ne.jp \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.