From: Michal Hocko <mhocko@kernel.org>
To: kemi <kemi.wang@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@suse.cz>,
Mel Gorman <mgorman@techsingularity.net>,
Johannes Weiner <hannes@cmpxchg.org>,
Christopher Lameter <cl@linux.com>,
YASUAKI ISHIMATSU <yasu.isimatu@gmail.com>,
Andrey Ryabinin <aryabinin@virtuozzo.com>,
Nikolay Borisov <nborisov@suse.com>,
Pavel Tatashin <pasha.tatashin@oracle.com>,
David Rientjes <rientjes@google.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Dave <dave.hansen@linux.intel.com>,
Andi Kleen <andi.kleen@intel.com>,
Tim Chen <tim.c.chen@intel.com>,
Jesper Dangaard Brouer <brouer@redhat.com>,
Ying Huang <ying.huang@intel.com>, Aaron Lu <aaron.lu@intel.com>,
Aubrey Li <aubrey.li@intel.com>, Linux MM <linux-mm@kvack.org>,
Linux Kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 3/5] mm: enlarge NUMA counters threshold size
Date: Thu, 21 Dec 2017 09:17:06 +0100 [thread overview]
Message-ID: <20171221081706.GA4831@dhcp22.suse.cz> (raw)
In-Reply-To: <268b1b6e-ff7a-8f1a-f97c-f94e14591975@intel.com>
On Thu 21-12-17 16:06:50, kemi wrote:
>
>
> On 2017年12月20日 18:12, Michal Hocko wrote:
> > On Wed 20-12-17 13:52:14, kemi wrote:
> >>
> >>
> >> On 2017年12月19日 20:40, Michal Hocko wrote:
> >>> On Tue 19-12-17 14:39:24, Kemi Wang wrote:
> >>>> We have seen significant overhead in cache bouncing caused by NUMA counters
> >>>> update in multi-threaded page allocation. See 'commit 1d90ca897cb0 ("mm:
> >>>> update NUMA counter threshold size")' for more details.
> >>>>
> >>>> This patch updates NUMA counters to a fixed size of (MAX_S16 - 2) and deals
> >>>> with global counter update using different threshold size for node page
> >>>> stats.
> >>>
> >>> Again, no numbers.
> >>
> >> Compare to vanilla kernel, I don't think it has performance improvement, so
> >> I didn't post performance data here.
> >> But, if you would like to see performance gain from enlarging threshold size
> >> for NUMA stats (compare to the first patch), I will do that later.
> >
> > Please do. I would also like to hear _why_ all counters cannot simply
> > behave same. In other words why we cannot simply increase
> > stat_threshold? Maybe calculate_normal_threshold needs a better scaling
> > for larger machines.
> >
>
> I will add this performance data to changelog in V3 patch series.
>
> Test machine: 2-sockets skylake platform (112 CPUs, 62G RAM)
> Benchmark: page_bench03
> Description: 112 threads do single page allocation/deallocation in parallel.
> before after
> (enlarge threshold size)
> CPU cycles 722 379(-47.5%)
Please describe the numbers some more. Is this an average? What is the
std? Can you see any difference with a more generic workload?
> Some thinking about that:
> a) the overhead due to cache bouncing caused by NUMA counter update in fast path
> severely increase with more and more CPUs cores
What is an effect on a smaller system with fewer CPUs?
> b) AFAIK, the typical usage scenario (similar at least)for which this optimization can
> benefit is 10/40G NIC used in high-speed data center network of cloud service providers.
I would expect those would disable the numa accounting altogether.
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2017-12-21 8:17 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-19 6:39 [PATCH v2 0/5] mm: NUMA stats code cleanup and enhancement Kemi Wang
2017-12-19 6:39 ` [PATCH v2 1/5] mm: migrate NUMA stats from per-zone to per-node Kemi Wang
2017-12-19 12:28 ` Michal Hocko
2017-12-20 5:32 ` kemi
2017-12-19 6:39 ` [PATCH v2 2/5] mm: Extends local cpu counter vm_diff_nodestat from s8 to s16 Kemi Wang
2017-12-19 12:38 ` Michal Hocko
2017-12-20 3:05 ` kemi
2017-12-19 16:05 ` Christopher Lameter
2017-12-19 16:20 ` Michal Hocko
2017-12-19 17:21 ` Christopher Lameter
2017-12-20 6:45 ` kemi
2017-12-19 6:39 ` [PATCH v2 3/5] mm: enlarge NUMA counters threshold size Kemi Wang
2017-12-19 12:40 ` Michal Hocko
2017-12-20 5:52 ` kemi
2017-12-20 10:12 ` Michal Hocko
2017-12-20 10:21 ` kemi
2017-12-21 8:06 ` kemi
2017-12-21 8:17 ` Michal Hocko [this message]
2017-12-21 8:23 ` kemi
2017-12-21 8:59 ` Michal Hocko
2017-12-21 10:31 ` kemi
2017-12-22 12:31 ` Michal Hocko
2017-12-21 17:10 ` Christopher Lameter
2017-12-22 2:06 ` kemi
2017-12-26 19:05 ` Christopher Lameter
2017-12-19 6:39 ` [PATCH v2 4/5] mm: use node_page_state_snapshot to avoid deviation Kemi Wang
2017-12-19 12:43 ` Michal Hocko
2017-12-20 6:07 ` kemi
2017-12-20 10:06 ` Michal Hocko
2017-12-20 10:24 ` kemi
2017-12-20 15:58 ` Christopher Lameter
2017-12-21 1:39 ` kemi
2017-12-19 6:39 ` [PATCH v2 5/5] mm: Rename zone_statistics() to numa_statistics() Kemi Wang
2017-12-19 12:44 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171221081706.GA4831@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=aaron.lu@intel.com \
--cc=akpm@linux-foundation.org \
--cc=andi.kleen@intel.com \
--cc=aryabinin@virtuozzo.com \
--cc=aubrey.li@intel.com \
--cc=bigeasy@linutronix.de \
--cc=brouer@redhat.com \
--cc=cl@linux.com \
--cc=dave.hansen@linux.intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@cmpxchg.org \
--cc=kemi.wang@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=nborisov@suse.com \
--cc=pasha.tatashin@oracle.com \
--cc=rientjes@google.com \
--cc=tim.c.chen@intel.com \
--cc=vbabka@suse.cz \
--cc=yasu.isimatu@gmail.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).