RE: Regarding WQ_MEM_RECLAIM

RE: Regarding WQ_MEM_RECLAIM

[Xue, Zhan]

Convert to plain text..

From: Xue, Zhan 
Sent: Wednesday, August 24, 2022 1:47 PM
To: tj@kernel.org
Cc: florian@mickler.org; LKML <linux-kernel@vger.kernel.org>
Subject: RE: Regarding WQ_MEM_RECLAIM

Hi Tejun,
          Please refer my inline comments infra. Thx.


BR
XueZhan

-----Original Message-----
From: Tejun Heo <mailto:htejun@gmail.com> On Behalf Of mailto:tj@kernel.org
Sent: Wednesday, August 24, 2022 2:27 AM
To: Xue, Zhan <mailto:zhan.xue@intel.com>
Cc: mailto:florian@mickler.org
Subject: Re: Regarding WQ_MEM_RECLAIM

Hello,

On Tue, Aug 23, 2022 at 09:29:07AM +0000, Xue, Zhan wrote:
> Hi Tejun Heo, Florian Mickler,
>        I see a few patches and discussions regarding WQ_MEM_RECLAIM.  Could you pls confirm the explicit rules?   e.g.  Is  devm_kzalloc with GFP_KERNEL allowed in the work queued to wq marked with WQ_MEM_RECLAIM?  Thx

Yeah, for sure. The only thing WQ_MEM_RECLAIM does is guaranteeing forward progress to work items on the workqueue under memory shortage. The workqueue can be used for whatever. The thing to be careful about is that the forward progress guarantee is good for one concurrent excecution instance for a workqueue.

[XZ]:  We could also say WQ_MEM_RECLAIM can benefit memory pressure mitigation for the whole system (not only to the current wq or other work items on it)  if some work items are with memory reclaim?    

Let's say you have two work items queued on the workqueue and the system is under severe memory pressure, so the rescuer thread is running the first work item. If that first work item does a GFP_KERNEL allocation and gets stuck and the second work item won't be able to run until somebody releases the memory pressure. If the second work item is actually needed to make forward progress in memory reclaim, the situation isn't great. The second work item probably shouldn't have shared the workqueue with the first work item.

[XZ]:  if the rescuer thread could distinguish the work items for memory reclaim and choose to execute them first, would it be possible to mitigate such situation?  It needs to  add specific tag to indicate if the work item is with memory reclaim or not ( or other mechanism to indicate the dependency relationship between work items on that wq for execution sequence/priority).

[XZ]:  Also would like to confirm the understanding of check_flush_dependency (https://docs.kernel.org/core-api/workqueue.html).    If the current wq is without WQ_MEM_RECLAIM ( memory reclaim),  there will be no WQ_MEM_RECLAIM requirement for the target wq and it's up to target wq to set WQ_MEM_RECLAIM or not per its need ?

One use case is that the current wq may be with the WQ_MEM_RECLAIM  for memory reclaim (e.g. the work item on current wq may free quite a few pages)  but the target wq (work items on it)  is irrelevant to memory reclaim.   There will be a warning " workqueue: WQ_MEM_RECLAIM (current wq) is flushing !WQ_MEM_RECLAIM events:(target wq)"  due to the requirement of flush dependency sanity.  To address it there may be several options as below.  It seems opt1 is preferred?  

Opt1:   The target wq can be marked with WQ_MEM_RECLAIM as well even it isn't really in any memory reclaim path.   It's to ensure the work item on target wq can get flushed (e.g. by its rescuer thread)  then the work item on current wq can continue the memory reclaim. 
Opt2:   Having neither current wq nor target wq marked with WQ_MEM_RECLAIM can also pass the check_flush_dependency.  

Actually in practice the scenario may be more complicated, e.g. the work item on current wq may free a big chunk of memory, but before that it may need to  devm_kzalloc a small piece of memory with GFP_KERNEL for specific purpose, etc.   Sometimes it seems more like a trade-off and needs to measure the Pros/Cons case by case.


> If there are indeed some principles it's good to update them into doc like https://www.kernel.org/doc/html/v4.10/core-api/workqueue.html .

Ah yeah, that'd be great. I don't have the bw right now but if anyone wanna take a stab at it, please be my guest.

[XZ]: I feel it's good to have use cases(e.g. if above case mentioned is appropriate)  listed for guidance.

Thanks.

--
tejun

Re: Regarding WQ_MEM_RECLAIM

[tj]

On Wed, Aug 24, 2022 at 06:10:56AM +0000, Xue, Zhan wrote:
> Convert to plain text..

The email's formatting is too painful to reply directly.

Here are two brief points:

* Just don't share the same workqueue between work items which need forward
  progress guarantee and ones which don't.

* If something can block memory reclaim, it is in the memory reclaim path by
  definition.

Thanks.

-- 
tejun

RE: Regarding WQ_MEM_RECLAIM

[Xue, Zhan]

Hi Tejun,
      The brief point seems the answer to one concern about " workqueue: WQ_MEM_RECLAIM (current wq) is flushing !WQ_MEM_RECLAIM events:(target wq)".   If current wq is for memory reclaim , the target wq should be marked with WQ_MEM_RECLAIM as well in case that the flushing target wq  (its work items) in the context of current wq is  inevitable.


BR
Xuezhan
-----Original Message-----
From: Tejun Heo <htejun@gmail.com> On Behalf Of tj@kernel.org
Sent: Friday, August 26, 2022 2:03 AM
To: Xue, Zhan <zhan.xue@intel.com>
Cc: florian@mickler.org; LKML <linux-kernel@vger.kernel.org>
Subject: Re: Regarding WQ_MEM_RECLAIM

On Wed, Aug 24, 2022 at 06:10:56AM +0000, Xue, Zhan wrote:
> Convert to plain text..

The email's formatting is too painful to reply directly.

Here are two brief points:

* Just don't share the same workqueue between work items which need forward
  progress guarantee and ones which don't.

* If something can block memory reclaim, it is in the memory reclaim path by
  definition.

Thanks.

-- 
tejun