From: Aaron Lu <email@example.com>
To: "firstname.lastname@example.org" <email@example.com>
Cc: Mel Gorman <firstname.lastname@example.org>,
kernel test robot <email@example.com>,
Linus Torvalds <firstname.lastname@example.org>,
Vlastimil Babka <email@example.com>,
Dave Hansen <firstname.lastname@example.org>,
Jesper Dangaard Brouer <email@example.com>,
Michal Hocko <firstname.lastname@example.org>,
Andrew Morton <email@example.com>,
LKML <firstname.lastname@example.org>, <email@example.com>,
Subject: Re: [mm/page_alloc] f26b3fa046: netperf.Throughput_Mbps -18.0% regression
Date: Wed, 11 May 2022 15:53:34 +0800 [thread overview]
Message-ID: <firstname.lastname@example.org> (raw)
On 5/11/2022 3:32 PM, email@example.com wrote:
> On Wed, 2022-05-11 at 11:40 +0800, Aaron Lu wrote:
>> On Tue, May 10, 2022 at 02:23:28PM +0800, firstname.lastname@example.org wrote:
>>> On Tue, 2022-05-10 at 11:43 +0800, Aaron Lu wrote:
>>>> On 5/7/2022 3:44 PM, email@example.com wrote:
>>>>> On Sat, 2022-05-07 at 15:31 +0800, Aaron Lu wrote:
>>>> ... ...
>>>>>> I thought the overhead of changing the cache line from "shared" to
>>>>>> "own"/"modify" is pretty cheap.
>>>>> This is the read/write pattern of cache ping-pong. Although it should
>>>>> be cheaper than the write/write pattern of cache ping-pong in theory, we
>>>>> have gotten sevious regression for that before.
>>>> Can you point me to the regression report? I would like to take a look,
>>>>>> Also, this is the same case as the Skylake desktop machine, why it is a
>>>>>> gain there but a loss here?
>>>>> I guess the reason is the private cache size. The size of the private
>>>>> L2 cache of SKL server is much larger than that of SKL client (1MB vs.
>>>>> 256KB). So there's much more core-2-core traffic on SKL server.
>>>> It could be. The 256KiB L2 in Skylake desktop can only store 8 order-3
>>>> pages and that means the allocator side may have a higher chance of
>>>> reusing a page that is evicted from the free cpu's L2 cache than the
>>>> server machine, whose L2 can store 40 order-3 pages.
>>>> I can do more tests using different high for the two machines:
>>>> 1) high=0, this is the case when page reuse is the extreme. core-2-core
>>>> transfer should be the most. This is the behavior of this bisected commit.
>>>> 2) high=L2_size, this is the case when page reuse is fewer compared to
>>>> the above case, core-2-core should still be the majority.
>>>> 3) high=2 times of L2_size and smaller than llc size, this is the case
>>>> when cache reuse is further reduced, and when the page is indeed reused,
>>>> it shouldn't cause core-2-core transfer but can benefit from llc.
>>>> 4) high>llc_size, this is the case when page reuse is the least and when
>>>> page is indeed reused, it is likely not in the entire cache hierarchy.
>>>> This is the behavior of this bisected commit's parent commit for the
>>>> Skylake desktop machine.
>>>> I expect case 3) should give us the best performance and 1) or 4) is the
>>>> worst for this testcase.
>>>> case 4) is difficult to test on the server machine due to the cap of
>>>> pcp->high which is affected by the low watermark of the zone. The server
>>>> machine has 128 cpus but only 128G memory, which makes the pcp->high
>>>> capped at 421, while llc size is 40MiB and that translates to a page
>>>> number of 12288.
>>> Sounds good to me.
>> I've run the tests on a 2 sockets Icelake server and a Skylake desktop.
>> On this 2 sockets Icelake server(1.25MiB L2 = 320 pages, 48MiB LLC =
>> 12288 pages):
>> pcp->high score
>> 0 100662 (bypass PCP, most page resue, most core-2-core transfer)
>> 320(L2) 117252
>> 640 133149
>> 6144(1/2 llc) 134674
>> 12416(>llc) 103193 (least page reuse)
>> Setting pcp->high to 640(2 times L2 size) gives very good result, only
>> slightly lower than 6144(1/2 llc size). Bypassing PCP to get the most
>> cache reuse didn't deliver good performance, so I think Ying is right:
>> core-2-core really hurts.
>> On this 4core/8cpu Skylake desktop(256KiB L2 = 64 pages, 8MiB LLC = 2048
>> 0 86780 (bypass PCP, most page reuse, most core-2-core transfer)
>> 64(L2) 85813
>> 128 85521
>> 1024(1/2 llc) 85557
>> 2176(> llc) 74458 (least page reuse)
>> Things are different on this small machine. Bypassing PCP gives the best
>> performance. I find it hard to explain this. Maybe the 256KiB is too
>> small that even bypassing PCP, the page still ends up being evicted from
>> L2 when allocator side reuses it? Or maybe core-2-core transfer is
>> fast on this small machine?
> 86780 / 85813 = 1.011
> So, there's almost no measurable difference among the configurations
> except the last one. I would rather say the test isn't sensitive to L2
> size, but sensitive to LLC size on this machine.
Well, if core-2-core transfer is bad for performance, I expect the
performance number of pcp->high=0 to be worse than pcp->high=64 and
pcp->high=128, not as good or even better, that's what I find hard to
As for performance number being bad when pcp->high > llc, that's
understandable because there is least page/cache reuse and this is the
same for both the desktop machine and that server machine.
next prev parent reply other threads:[~2022-05-11 7:53 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-20 1:35 [mm/page_alloc] f26b3fa046: netperf.Throughput_Mbps -18.0% regression kernel test robot
2022-04-29 11:29 ` Aaron Lu
2022-04-29 13:39 ` Mel Gorman
2022-05-05 8:27 ` Aaron Lu
2022-05-05 11:09 ` Mel Gorman
2022-05-05 14:29 ` Aaron Lu
2022-05-06 8:40 ` ying.huang
2022-05-06 12:17 ` Aaron Lu
2022-05-07 0:54 ` ying.huang
2022-05-07 3:27 ` Aaron Lu
2022-05-07 7:11 ` ying.huang
2022-05-07 7:31 ` Aaron Lu
2022-05-07 7:44 ` ying.huang
2022-05-10 3:43 ` Aaron Lu
2022-05-10 6:23 ` ying.huang
2022-05-10 18:05 ` Linus Torvalds
2022-05-10 18:47 ` Waiman Long
2022-05-10 19:03 ` Linus Torvalds
2022-05-10 19:25 ` Linus Torvalds
2022-05-10 19:46 ` Waiman Long
2022-05-10 19:27 ` Peter Zijlstra
2022-05-11 1:58 ` ying.huang
2022-05-11 2:06 ` Waiman Long
2022-05-11 11:04 ` Aaron Lu
2022-05-12 3:17 ` ying.huang
2022-05-12 12:45 ` Aaron Lu
2022-05-12 17:42 ` Linus Torvalds
2022-05-12 18:06 ` Andrew Morton
2022-05-12 18:49 ` Linus Torvalds
2022-06-14 2:09 ` Feng Tang
2022-05-13 6:19 ` ying.huang
2022-05-11 3:40 ` Aaron Lu
2022-05-11 7:32 ` ying.huang
2022-05-11 7:53 ` Aaron Lu [this message]
2022-06-01 2:19 ` Aaron Lu
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).