linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Dave Hansen <dave.hansen@intel.com>
To: Andrew Morton <akpm@linux-foundation.org>,
	Feng Tang <feng.tang@intel.com>
Cc: linux-mm@kvack.org, Michal Hocko <mhocko@kernel.org>,
	David Rientjes <rientjes@google.com>,
	Ben Widawsky <ben.widawsky@intel.com>,
	linux-kernel@vger.kernel.org, linux-api@vger.kernel.org,
	Andrea Arcangeli <aarcange@redhat.com>,
	Mel Gorman <mgorman@techsingularity.net>,
	Mike Kravetz <mike.kravetz@oracle.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	Vlastimil Babka <vbabka@suse.cz>, Andi Kleen <ak@linux.intel.com>,
	Dan Williams <dan.j.williams@intel.com>,
	ying.huang@intel.com
Subject: Re: [PATCH v6 0/6] Introduce multi-preference mempolicy
Date: Thu, 15 Jul 2021 11:49:01 -0700	[thread overview]
Message-ID: <fb4a6789-d461-e83e-c718-baff88026a88@intel.com> (raw)
In-Reply-To: <20210714171540.7cb9e221d683b531928b71f5@linux-foundation.org>

On 7/14/21 5:15 PM, Andrew Morton wrote:
> On Mon, 12 Jul 2021 16:09:28 +0800 Feng Tang <feng.tang@intel.com> wrote:
>> This patch series introduces the concept of the MPOL_PREFERRED_MANY mempolicy.
>> This mempolicy mode can be used with either the set_mempolicy(2) or mbind(2)
>> interfaces. Like the MPOL_PREFERRED interface, it allows an application to set a
>> preference for nodes which will fulfil memory allocation requests. Unlike the
>> MPOL_PREFERRED mode, it takes a set of nodes. Like the MPOL_BIND interface, it
>> works over a set of nodes. Unlike MPOL_BIND, it will not cause a SIGSEGV or
>> invoke the OOM killer if those preferred nodes are not available.
> Do we have any real-world testing which demonstrates the benefits of
> all of this?

Yes, it's actually been quite useful in practice already.

If we take persistent memory media (PMEM) and hot-add/online it with the
DAX kmem driver, we get NUMA nodes with lots of capacity (~6TB is
typical) but weird performance; PMEM has good read speed, but low write
speed.

That low write speed is *so* low that it dominates the performance more
than the distance from the CPUs.  Folks who want PMEM really don't care
about locality.  The discussions with the testers usually go something
like this:

Tester: How do I make my test use PMEM on nodes 2 and 3?
Kernel Guys: use 'numactl --membind=2-3'
Tester: I tried that, but I'm getting allocation failures once I fill up
        PMEM.  Shouldn't it fall back to DRAM?
Kernel Guys: Fine, use 'numactl --preferred=2-3'
Tester: That worked, but it started using DRAM after it exhausted node 2
Kernel Guys:  Dang it.  I forgot --preferred ignores everything after
              the first node.  Fine, we'll patch the kernel.

This has happened more than once.  End users want to be able to specify
a specific physical media, but don't want to have to deal with the sharp
edges of strict binding.

This has happened both with slow media like PMEM and "faster" media like
High-Bandwidth Memory.

      parent reply	other threads:[~2021-07-15 18:55 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-12  8:09 [PATCH v6 0/6] Introduce multi-preference mempolicy Feng Tang
2021-07-12  8:09 ` [PATCH v6 1/6] mm/mempolicy: Add MPOL_PREFERRED_MANY for multiple preferred nodes Feng Tang
2021-07-28 12:31   ` Michal Hocko
2021-07-28 14:11     ` Feng Tang
2021-07-28 16:12       ` Michal Hocko
2021-07-29  7:09         ` Feng Tang
2021-07-29 13:38           ` Michal Hocko
2021-07-29 15:12             ` Feng Tang
2021-07-29 16:21               ` Michal Hocko
2021-07-30  3:05                 ` Feng Tang
2021-07-30  6:36                   ` Michal Hocko
2021-07-30  7:18                     ` Feng Tang
2021-07-30  7:38                       ` Michal Hocko
2021-08-02  8:11                       ` Feng Tang
2021-08-02 11:14                         ` Michal Hocko
2021-08-02 11:33                           ` Feng Tang
2021-08-02 11:47                             ` Michal Hocko
2021-07-12  8:09 ` [PATCH v6 2/6] mm/memplicy: add page allocation function for MPOL_PREFERRED_MANY policy Feng Tang
2021-07-28 12:42   ` Michal Hocko
2021-07-28 15:18     ` Feng Tang
2021-07-28 15:25       ` Feng Tang
2021-07-28 16:15         ` Michal Hocko
2021-07-28 16:14       ` Michal Hocko
2021-07-12  8:09 ` [PATCH v6 3/6] mm/mempolicy: enable page allocation for MPOL_PREFERRED_MANY for general cases Feng Tang
2021-07-12  8:09 ` [PATCH v6 4/6] mm/hugetlb: add support for mempolicy MPOL_PREFERRED_MANY Feng Tang
2021-07-21 20:49   ` Mike Kravetz
2021-07-22  8:11     ` Feng Tang
2021-07-22  9:42     ` Michal Hocko
2021-07-22 16:21       ` Mike Kravetz
2021-07-12  8:09 ` [PATCH v6 5/6] mm/mempolicy: Advertise new MPOL_PREFERRED_MANY Feng Tang
2021-07-28 12:47   ` Michal Hocko
2021-07-28 13:41     ` Feng Tang
2021-07-12  8:09 ` [PATCH v6 6/6] mm/mempolicy: unify the create() func for bind/interleave/prefer-many policies Feng Tang
2021-07-28 12:51   ` Michal Hocko
2021-07-28 13:50     ` Feng Tang
2021-07-15  0:15 ` [PATCH v6 0/6] Introduce multi-preference mempolicy Andrew Morton
2021-07-15  2:13   ` Feng Tang
2021-07-15 18:49   ` Dave Hansen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=fb4a6789-d461-e83e-c718-baff88026a88@intel.com \
    --to=dave.hansen@intel.com \
    --cc=aarcange@redhat.com \
    --cc=ak@linux.intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=ben.widawsky@intel.com \
    --cc=dan.j.williams@intel.com \
    --cc=feng.tang@intel.com \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@techsingularity.net \
    --cc=mhocko@kernel.org \
    --cc=mike.kravetz@oracle.com \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=vbabka@suse.cz \
    --cc=ying.huang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).