linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: daniel.m.jordan@oracle.com
To: linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: [RFC PATCH v1 0/6] ktask: multithread cpu-intensive kernel work
Date: Fri, 14 Jul 2017 15:16:07 -0700	[thread overview]
Message-ID: <1500070573-3948-1-git-send-email-daniel.m.jordan@oracle.com> (raw)

ktask is a generic framework for parallelizing cpu-intensive work in the
kernel.  The intended use is for big machines that can use their cpu power to
speed up large tasks that can't otherwise be multithreaded in userland.  The
API is generic enough to add concurrency to many different kinds of tasks--for
example, zeroing a range of pages or evicting a list of inodes--and aims to
save its clients the trouble of splitting up the work, choosing the number of
threads to use, starting these threads, and load balancing the work between
them.

Why do we need ktask when the kernel has other APIs for managing concurrency?
After all, kthread_workers and workqueues already provide ways to start
threads, and the kernel can handle large tasks with a single thread by
periodically yielding the cpu with cond_resched or doing the work in fixed size
batches.

Of the existing concurrency facilities, kthread_worker isn't suited for
providing parallelism because each comes with only a single thread.  Workqueues
are a better fit for this, and in fact ktask is built on an unbound workqueue,
but workqueues aren't designed for splitting up a large task.  ktask instead
uses unbound workqueue threads to run "chunks" of a task.

More background is available in the documentation commit (first commit of the
series).

There are two ktask consumers included, with more to come later.  Other
consumers that are in the works include:

  - Page table walking and/or mmu_gather struct page freeing to optimize exit(2)
    and munmap(2) for large processes.  This is inspired by Aaron Lu's work:
        http://marc.info/?l=linux-mm&m=148793643210514&w=2

  - struct page initialization in early boot (use more threads than the current
    pgdatinit threads to reduce boot time).

The core ktask code is based on work by Pavel Tatashin, Steve Sistare, and
Jonathan Adams.

This series is based on 4.12.

Daniel Jordan (6):
  ktask: add documentation
  ktask: multithread cpu-intensive kernel work
  ktask: add /proc/sys/debug/ktask_max_threads
  mm: enlarge type of offset argument in mem_map_offset and
    mem_map_next
  mm: parallelize clear_gigantic_page
  hugetlbfs: parallelize hugetlbfs_fallocate with ktask

 Documentation/core-api/index.rst |    1 +
 Documentation/core-api/ktask.rst |  104 ++++++++++
 fs/hugetlbfs/inode.c             |  122 ++++++++++---
 include/linux/ktask.h            |  228 ++++++++++++++++++++++
 include/linux/ktask_internal.h   |   19 ++
 include/linux/mm.h               |    5 +
 init/Kconfig                     |    7 +
 kernel/Makefile                  |    2 +-
 kernel/ktask.c                   |  389 ++++++++++++++++++++++++++++++++++++++
 kernel/sysctl.c                  |   10 +
 mm/internal.h                    |    7 +-
 mm/memory.c                      |   35 +++-
 12 files changed, 895 insertions(+), 34 deletions(-)
 create mode 100644 Documentation/core-api/ktask.rst
 create mode 100644 include/linux/ktask.h
 create mode 100644 include/linux/ktask_internal.h
 create mode 100644 kernel/ktask.c

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

             reply	other threads:[~2017-07-14 22:16 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-07-14 22:16 daniel.m.jordan [this message]
2017-07-14 22:16 ` [RFC PATCH v1 1/6] ktask: add documentation daniel.m.jordan
2017-07-14 22:16 ` [RFC PATCH v1 2/6] ktask: multithread cpu-intensive kernel work daniel.m.jordan
2017-07-14 22:16 ` [RFC PATCH v1 3/6] ktask: add /proc/sys/debug/ktask_max_threads daniel.m.jordan
2017-07-14 22:16 ` [RFC PATCH v1 4/6] mm: enlarge type of offset argument in mem_map_offset and mem_map_next daniel.m.jordan
2017-07-14 22:16 ` [RFC PATCH v1 5/6] mm: parallelize clear_gigantic_page daniel.m.jordan
2017-07-17 16:02   ` Dave Hansen
2017-07-18  1:49     ` Daniel Jordan
2017-07-14 22:16 ` [RFC PATCH v1 6/6] hugetlbfs: parallelize hugetlbfs_fallocate with ktask daniel.m.jordan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1500070573-3948-1-git-send-email-daniel.m.jordan@oracle.com \
    --to=daniel.m.jordan@oracle.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).