From: kemi <kemi.wang@intel.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@suse.cz>,
Mel Gorman <mgorman@techsingularity.net>,
Johannes Weiner <hannes@cmpxchg.org>,
Christopher Lameter <cl@linux.com>,
YASUAKI ISHIMATSU <yasu.isimatu@gmail.com>,
Andrey Ryabinin <aryabinin@virtuozzo.com>,
Nikolay Borisov <nborisov@suse.com>,
Pavel Tatashin <pasha.tatashin@oracle.com>,
David Rientjes <rientjes@google.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Dave <dave.hansen@linux.intel.com>,
Andi Kleen <andi.kleen@intel.com>,
Tim Chen <tim.c.chen@intel.com>,
Jesper Dangaard Brouer <brouer@redhat.com>,
Ying Huang <ying.huang@intel.com>, Aaron Lu <aaron.lu@intel.com>,
Aubrey Li <aubrey.li@intel.com>, Linux MM <linux-mm@kvack.org>,
Linux Kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 3/5] mm: enlarge NUMA counters threshold size
Date: Thu, 21 Dec 2017 16:23:23 +0800 [thread overview]
Message-ID: <1fb66dfd-b64c-f705-ea27-a9f2e11729a4@intel.com> (raw)
In-Reply-To: <20171221081706.GA4831@dhcp22.suse.cz>
On 2017年12月21日 16:17, Michal Hocko wrote:
> On Thu 21-12-17 16:06:50, kemi wrote:
>>
>>
>> On 2017年12月20日 18:12, Michal Hocko wrote:
>>> On Wed 20-12-17 13:52:14, kemi wrote:
>>>>
>>>>
>>>> On 2017年12月19日 20:40, Michal Hocko wrote:
>>>>> On Tue 19-12-17 14:39:24, Kemi Wang wrote:
>>>>>> We have seen significant overhead in cache bouncing caused by NUMA counters
>>>>>> update in multi-threaded page allocation. See 'commit 1d90ca897cb0 ("mm:
>>>>>> update NUMA counter threshold size")' for more details.
>>>>>>
>>>>>> This patch updates NUMA counters to a fixed size of (MAX_S16 - 2) and deals
>>>>>> with global counter update using different threshold size for node page
>>>>>> stats.
>>>>>
>>>>> Again, no numbers.
>>>>
>>>> Compare to vanilla kernel, I don't think it has performance improvement, so
>>>> I didn't post performance data here.
>>>> But, if you would like to see performance gain from enlarging threshold size
>>>> for NUMA stats (compare to the first patch), I will do that later.
>>>
>>> Please do. I would also like to hear _why_ all counters cannot simply
>>> behave same. In other words why we cannot simply increase
>>> stat_threshold? Maybe calculate_normal_threshold needs a better scaling
>>> for larger machines.
>>>
>>
>> I will add this performance data to changelog in V3 patch series.
>>
>> Test machine: 2-sockets skylake platform (112 CPUs, 62G RAM)
>> Benchmark: page_bench03
>> Description: 112 threads do single page allocation/deallocation in parallel.
>> before after
>> (enlarge threshold size)
>> CPU cycles 722 379(-47.5%)
>
> Please describe the numbers some more. Is this an average?
Yes
> What is the std?
I increase the loop times to 10m, so the std is quite slow (repeat 3 times)
> Can you see any difference with a more generic workload?
>
I didn't see obvious improvement for will-it-scale.page_fault1
Two reasons for that:
1) too long code path
2) server zone lock and lru lock contention (access to buddy system frequently)
>> Some thinking about that:
>> a) the overhead due to cache bouncing caused by NUMA counter update in fast path
>> severely increase with more and more CPUs cores
>
> What is an effect on a smaller system with fewer CPUs?
>
Several CPU cycles can be saved using single thread for that.
>> b) AFAIK, the typical usage scenario (similar at least)for which this optimization can
>> benefit is 10/40G NIC used in high-speed data center network of cloud service providers.
>
> I would expect those would disable the numa accounting altogether.
>
Yes, but it is still worthy to do some optimization, isn't?
next prev parent reply other threads:[~2017-12-21 8:25 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-19 6:39 [PATCH v2 0/5] mm: NUMA stats code cleanup and enhancement Kemi Wang
2017-12-19 6:39 ` [PATCH v2 1/5] mm: migrate NUMA stats from per-zone to per-node Kemi Wang
2017-12-19 12:28 ` Michal Hocko
2017-12-20 5:32 ` kemi
2017-12-19 6:39 ` [PATCH v2 2/5] mm: Extends local cpu counter vm_diff_nodestat from s8 to s16 Kemi Wang
2017-12-19 12:38 ` Michal Hocko
2017-12-20 3:05 ` kemi
2017-12-19 16:05 ` Christopher Lameter
2017-12-19 16:20 ` Michal Hocko
2017-12-19 17:21 ` Christopher Lameter
2017-12-20 6:45 ` kemi
2017-12-19 6:39 ` [PATCH v2 3/5] mm: enlarge NUMA counters threshold size Kemi Wang
2017-12-19 12:40 ` Michal Hocko
2017-12-20 5:52 ` kemi
2017-12-20 10:12 ` Michal Hocko
2017-12-20 10:21 ` kemi
2017-12-21 8:06 ` kemi
2017-12-21 8:17 ` Michal Hocko
2017-12-21 8:23 ` kemi [this message]
2017-12-21 8:59 ` Michal Hocko
2017-12-21 10:31 ` kemi
2017-12-22 12:31 ` Michal Hocko
2017-12-21 17:10 ` Christopher Lameter
2017-12-22 2:06 ` kemi
2017-12-26 19:05 ` Christopher Lameter
2017-12-19 6:39 ` [PATCH v2 4/5] mm: use node_page_state_snapshot to avoid deviation Kemi Wang
2017-12-19 12:43 ` Michal Hocko
2017-12-20 6:07 ` kemi
2017-12-20 10:06 ` Michal Hocko
2017-12-20 10:24 ` kemi
2017-12-20 15:58 ` Christopher Lameter
2017-12-21 1:39 ` kemi
2017-12-19 6:39 ` [PATCH v2 5/5] mm: Rename zone_statistics() to numa_statistics() Kemi Wang
2017-12-19 12:44 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1fb66dfd-b64c-f705-ea27-a9f2e11729a4@intel.com \
--to=kemi.wang@intel.com \
--cc=aaron.lu@intel.com \
--cc=akpm@linux-foundation.org \
--cc=andi.kleen@intel.com \
--cc=aryabinin@virtuozzo.com \
--cc=aubrey.li@intel.com \
--cc=bigeasy@linutronix.de \
--cc=brouer@redhat.com \
--cc=cl@linux.com \
--cc=dave.hansen@linux.intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=nborisov@suse.com \
--cc=pasha.tatashin@oracle.com \
--cc=rientjes@google.com \
--cc=tim.c.chen@intel.com \
--cc=vbabka@suse.cz \
--cc=yasu.isimatu@gmail.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).