RFD: multithreading in padata

RFD: multithreading in padata

[Daniel Jordan]

[resending in modified form since this didn't seem to reach the lists]

Hi,

padata has been undergoing some surgery lately[0] and now seems ready for
another enhancement: splitting up and multithreading CPU-intensive kernel work.

I'm planning to send an RFC for this, but I wanted to post some thoughts on the
design and a work-in-progress branch first to see if the direction looks ok.

Quoting from an earlier series[1], this is the problem I'm trying to solve:

  A single CPU can spend an excessive amount of time in the kernel operating
  on large amounts of data.  Often these situations arise during initialization-
  and destruction-related tasks, where the data involved scales with system
  size.  These long-running jobs can slow startup and shutdown of applications
  and the system itself while extra CPUs sit idle.
      
There are several paths where this problem exists, but here are three to start:

 - struct page initialization (at boot-time, during memory hotplug, and for
   persistent memory)
 - VFIO page pinning (kvm guest initialization)
 - fallocating a HugeTLB page (database initialization)

padata is a general mechanism for parallel work and so seems natural for this
functionality[2], but now it can only manage a series of small, ordered jobs.

The coming RFC will bring enhancements to split up a large job among a set of
helper threads according to the user's wishes, load balance the work between
them, set concurrency limits to control the overall number of helpers in the
system and per NUMA node, and run extra helper threads beyond the first at a
low priority level so as not to disturb other activity on the system for the
sake of an optimization.  (While extra helpers are run at low priority for most
of the job, their priority is raised one by one at job end to match the
caller's to avoid starvation on a busy system.)

The existing padata interfaces and features will remain, but serialization
becomes optional because these sorts of jobs don't need it.

The advantage to enhancing padata rather than having the multithreading stand
alone is that there would be one central place in the kernel to manage the
number of helper threads that run at any given time.  A machine's idle CPU
resources can be harnessed yet controlled (the low priority idea) to provide
the right amount of multithreading for the system.

Here's a work-in-progress branch with some of this already done in the last
five patches.

    git://oss.oracle.com/git/linux-dmjordan.git padata-mt-wip
    https://oss.oracle.com/git/gitweb.cgi?p=linux-dmjordan.git;a=shortlog;h=refs/heads/padata-mt-wip

Thoughts?  Questions?

Thanks.

Daniel

[0] https://lore.kernel.org/linux-crypto/?q=s%3Apadata+d%3A20190101..
[1] https://lore.kernel.org/lkml/20181105165558.11698-1-daniel.m.jordan@oracle.com/
[2] https://lore.kernel.org/lkml/20181107103554.GL9781@hirez.programming.kicks-ass.net/

Re: RFD: multithreading in padata

[Steffen Klassert]

On Tue, Dec 03, 2019 at 10:58:41AM -0500, Daniel Jordan wrote:
> [resending in modified form since this didn't seem to reach the lists]
> 
> Hi,
> 
> padata has been undergoing some surgery lately[0] and now seems ready for
> another enhancement: splitting up and multithreading CPU-intensive kernel work.
> 
> I'm planning to send an RFC for this, but I wanted to post some thoughts on the
> design and a work-in-progress branch first to see if the direction looks ok.
> 
> Quoting from an earlier series[1], this is the problem I'm trying to solve:
> 
>   A single CPU can spend an excessive amount of time in the kernel operating
>   on large amounts of data.  Often these situations arise during initialization-
>   and destruction-related tasks, where the data involved scales with system
>   size.  These long-running jobs can slow startup and shutdown of applications
>   and the system itself while extra CPUs sit idle.
>       
> There are several paths where this problem exists, but here are three to start:
> 
>  - struct page initialization (at boot-time, during memory hotplug, and for
>    persistent memory)
>  - VFIO page pinning (kvm guest initialization)
>  - fallocating a HugeTLB page (database initialization)
> 
> padata is a general mechanism for parallel work and so seems natural for this
> functionality[2], but now it can only manage a series of small, ordered jobs.
> 
> The coming RFC will bring enhancements to split up a large job among a set of
> helper threads according to the user's wishes, load balance the work between
> them, set concurrency limits to control the overall number of helpers in the
> system and per NUMA node, and run extra helper threads beyond the first at a
> low priority level so as not to disturb other activity on the system for the
> sake of an optimization.  (While extra helpers are run at low priority for most
> of the job, their priority is raised one by one at job end to match the
> caller's to avoid starvation on a busy system.)
> 
> The existing padata interfaces and features will remain, but serialization
> becomes optional because these sorts of jobs don't need it.
> 
> The advantage to enhancing padata rather than having the multithreading stand
> alone is that there would be one central place in the kernel to manage the
> number of helper threads that run at any given time.  A machine's idle CPU
> resources can be harnessed yet controlled (the low priority idea) to provide
> the right amount of multithreading for the system.
> 
> Here's a work-in-progress branch with some of this already done in the last
> five patches.
> 
>     git://oss.oracle.com/git/linux-dmjordan.git padata-mt-wip
>     https://oss.oracle.com/git/gitweb.cgi?p=linux-dmjordan.git;a=shortlog;h=refs/heads/padata-mt-wip
> 
> Thoughts?  Questions?

I'm ok with this. Please consider to add yourself as
a Co-Maintainer, I guess you know the code in between
much better than I do :)

Re: RFD: multithreading in padata

[Daniel Jordan]

On Mon, Dec 09, 2019 at 01:39:29PM +0100, Steffen Klassert wrote:
> On Tue, Dec 03, 2019 at 10:58:41AM -0500, Daniel Jordan wrote:
> > Thoughts?  Questions?
> 
> I'm ok with this.

Glad to hear it, will continue in this direction.

> Please consider to add yourself as
> a Co-Maintainer, I guess you know the code in between
> much better than I do :)

Sure, I'll consider that :)  Thanks for looking, Steffen.