[RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[Tejun Heo]

Hello,

Years ago, workqueue got reimplemented to use common worker pools
across different workqueues and a new set of more expressive workqueue
creation APIs, alloc_*workqueue() were introduced.  The old
create_*workqueue() became simple wrappers around alloc_*workqueue()
with the most conservative parameters.  The plan has always been to
examine each usage and convert to the new interface with parameters
actually required for the use case.

One important flag to decide upon is WQ_MEM_RECLAIM, which declares
that the workqueue may be depended upon during memory reclaim and thus
must be able to make forward-progress even when further memory can't
be allocated without reclaiming some.  Of the network drivers which
already use alloc_*workqueue() interface, some specify this flag and
I'm wondering what the guidelines should be here.

* Are network devices expected to be able to serve as a part of
  storage stack which is depended upon for memory reclamation?

* If so, are all the pieces in place for that to work for all (or at
  least most) network devices?  If it's only for a subset of NICs, how
  can one tell whether a given driver needs forward progress guarantee
  or not?

* I assume that wireless drivers aren't and can't be used in this
  fashion.  Is that a correction assumption?

Thanks.

-- 
tejun

Re: [RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[Jeff Layton]

On Thu, 17 Mar 2016 09:45:46 -0700
Tejun Heo <tj@kernel.org> wrote:

> Hello,
> 
> Years ago, workqueue got reimplemented to use common worker pools
> across different workqueues and a new set of more expressive workqueue
> creation APIs, alloc_*workqueue() were introduced.  The old
> create_*workqueue() became simple wrappers around alloc_*workqueue()
> with the most conservative parameters.  The plan has always been to
> examine each usage and convert to the new interface with parameters
> actually required for the use case.
> 
> One important flag to decide upon is WQ_MEM_RECLAIM, which declares
> that the workqueue may be depended upon during memory reclaim and thus
> must be able to make forward-progress even when further memory can't
> be allocated without reclaiming some.  Of the network drivers which
> already use alloc_*workqueue() interface, some specify this flag and
> I'm wondering what the guidelines should be here.
> 
> * Are network devices expected to be able to serve as a part of
>   storage stack which is depended upon for memory reclamation?
> 

I think they should be. Cached NFS pages can consume a lot of memory,
and flushing them generally takes network device access.

> * If so, are all the pieces in place for that to work for all (or at
>   least most) network devices?  If it's only for a subset of NICs, how
>   can one tell whether a given driver needs forward progress guarantee
>   or not?
> 
> * I assume that wireless drivers aren't and can't be used in this
>   fashion.  Is that a correction assumption?
> 

People do mount NFS over wireless interfaces. It's not terribly common
though, in my experience.

-- 
Jeff Layton <jlayton@poochiereds.net>

Re: [RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[Tejun Heo]

Hello, Jeff.

On Thu, Mar 17, 2016 at 09:32:16PM -0400, Jeff Layton wrote:
> > * Are network devices expected to be able to serve as a part of
> >   storage stack which is depended upon for memory reclamation?
> 
> I think they should be. Cached NFS pages can consume a lot of memory,
> and flushing them generally takes network device access.

But does that actually work?  It's pointless to add WQ_MEM_RECLAIM to
workqueues unless all other things are also guaranteed to make forward
progress regardless of memory pressure.

> > * If so, are all the pieces in place for that to work for all (or at
> >   least most) network devices?  If it's only for a subset of NICs, how
> >   can one tell whether a given driver needs forward progress guarantee
> >   or not?
> > 
> > * I assume that wireless drivers aren't and can't be used in this
> >   fashion.  Is that a correction assumption?
> > 
> 
> People do mount NFS over wireless interfaces. It's not terribly common
> though, in my experience.

Ditto, I'm very skeptical that this actually works in practice and
people expect and depend on it.  I don't follow wireless development
closely but haven't heard anyone talking about reserving memory pools
or people complaining about wireless being the cause of OOM.

So, I really want to avoid spraying WQ_MEM_RECLAIM if it doesn't serve
actual purposes.  It's wasteful, sets bad precedences and confuses
future readers.

Thanks.

-- 
tejun

Re: [RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[J. Bruce Fields]

On Fri, Mar 18, 2016 at 04:46:23PM -0400, Tejun Heo wrote:
> Hello, Jeff.
> 
> On Thu, Mar 17, 2016 at 09:32:16PM -0400, Jeff Layton wrote:
> > > * Are network devices expected to be able to serve as a part of
> > >   storage stack which is depended upon for memory reclamation?
> > 
> > I think they should be. Cached NFS pages can consume a lot of memory,
> > and flushing them generally takes network device access.
> 
> But does that actually work?  It's pointless to add WQ_MEM_RECLAIM to
> workqueues unless all other things are also guaranteed to make forward
> progress regardless of memory pressure.

It's supposed to work.

Also note there was a bunch of work done to allow swap on NFS: see
a564b8f0 "nfs: enable swap on NFS".

> 
> > > * If so, are all the pieces in place for that to work for all (or at
> > >   least most) network devices?  If it's only for a subset of NICs, how
> > >   can one tell whether a given driver needs forward progress guarantee
> > >   or not?
> > > 
> > > * I assume that wireless drivers aren't and can't be used in this
> > >   fashion.  Is that a correction assumption?
> > > 
> > 
> > People do mount NFS over wireless interfaces. It's not terribly common
> > though, in my experience.
> 
> Ditto, I'm very skeptical that this actually works in practice and
> people expect and depend on it.  I don't follow wireless development
> closely but haven't heard anyone talking about reserving memory pools
> or people complaining about wireless being the cause of OOM.
> 
> So, I really want to avoid spraying WQ_MEM_RECLAIM if it doesn't serve
> actual purposes.  It's wasteful, sets bad precedences and confuses
> future readers.

I use NFS mounts over wifi at home.  I may just be odd.  I seem to
recall some bug reports about suspend vs. NFS--were those also on
laptops using NFS?

I wonder if home media centers might do writes over wifi to network
storage?

Googling for "nfs wifi" turns up lots of individuals doing this.

My first impulse is to say that it's probably not perfect but that we
shouldn't make it worse.

But, I don't claim to understand the WQ_MEM_RECLAIM-proliferation issue
you're seeing....

--b.

Re: [RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[Tejun Heo]

Hello,

On Fri, Mar 18, 2016 at 05:24:05PM -0400, J. Bruce Fields wrote:
> > But does that actually work?  It's pointless to add WQ_MEM_RECLAIM to
> > workqueues unless all other things are also guaranteed to make forward
> > progress regardless of memory pressure.
> 
> It's supposed to work.
> 
> Also note there was a bunch of work done to allow swap on NFS: see
> a564b8f0 "nfs: enable swap on NFS".

Alright, WQ_MEM_RECLAIM for ethernet devices, then.

> I use NFS mounts over wifi at home.  I may just be odd.  I seem to
> recall some bug reports about suspend vs. NFS--were those also on
> laptops using NFS?
> 
> I wonder if home media centers might do writes over wifi to network
> storage?
> 
> Googling for "nfs wifi" turns up lots of individuals doing this.
> 
> My first impulse is to say that it's probably not perfect but that we
> shouldn't make it worse.

If everything else is working, I'd be happy to throw in WQ_MEM_RECLAIM
but I really don't want to add it if it doesn't actually achieve the
goal.  Can a wireless person chime in here?

Thanks.

-- 
tejun

Re: [RFD] workqueue: WQ_MEM_RECLAIM usage in network drivers

[Johannes Berg]

On Sun, 2016-03-20 at 14:55 -0400, Tejun Heo wrote:
> If everything else is working, I'd be happy to throw in
> WQ_MEM_RECLAIM but I really don't want to add it if it doesn't
> actually achieve the goal.  Can a wireless person chime in here?
> 

I think for many wireless devices the workqueue, like for iwldvm that
was just changed, isn't in the packet path and thus less relevant.

For some, like SDIO based chips, it's more likely to be in the packet
path, so it would make sense to keep it.

Of course there's always a possibility of getting disconnected and then
not being able to re-establish the connection in memory pressure, but
that's not something we can control/predict (the disconnection). Can of
course also happen with wired if somebody pulls out the cable, but it's
less reliant on memory allocations to get back to working there, I
guess :)

johannes