From: Feng Tang <feng.tang@intel.com>
To: Mel Gorman <mgorman@suse.de>
Cc: Borislav Petkov <bp@suse.de>, "Luck, Tony" <tony.luck@intel.com>,
kernel test robot <rong.a.chen@intel.com>,
LKML <linux-kernel@vger.kernel.org>,
lkp@lists.01.org
Subject: Re: [LKP] Re: [x86/mce] 1de08dccd3: will-it-scale.per_process_ops -14.1% regression
Date: Tue, 25 Aug 2020 14:49:26 +0800 [thread overview]
Message-ID: <20200825064926.GB83850@shbuild999.sh.intel.com> (raw)
In-Reply-To: <20200824165653.GQ2976@suse.com>
On Mon, Aug 24, 2020 at 05:56:53PM +0100, Mel Gorman wrote:
> On Mon, Aug 24, 2020 at 06:12:38PM +0200, Borislav Petkov wrote:
> >
> > > :) Right, this is what I'm doing right now. Some test job is queued on
> > > the test box, and it may needs some iterations of new patch. Hopefully we
> > > can isolate some specific variable given some luck.
> >
> > ... yes, exactly, you need to identify the contention where this
> > happens,
> > causing a cacheline to bounce or a variable straddles across a
> > cacheline boundary, causing the read to fetch two cachelines and thus
> > causes that slowdown. And then align that var to the beginning of a
> > cacheline.
> >
>
> Given the test is malloc1, it *may* be struct per_cpu_pages embedded within
> per_cpu_pageset. The cache characteristics of per_cpu_pageset are terrible
> because of how it mixes up zone counters and per-cpu lists. However, if
> the first per_cpu_pageset is cache-aligned then every second per_cpu_pages
> will be cache-aligned and half of the lists will fit in one cache line. If
> the whole structure gets pushed out of alignment then all per_cpu_pages
> straddle cache lines, increase the overall cache footprint and potentially
> cause problems if the cache is not large enough to hold hot structures.
>
> The misses could potentially be inferred without c2c from looking at
> perf -e cache-misses on a good and bad kernel and seeing if there is a
> noticable increase in misses in mm/page_alloc.c with a focus on anything
> using per-cpu lists.
Thanks for the tip, which is useful for Xeon-Phi. I ran it with 'cache-misses'
instead of default 'cycles', and the 2 versions of perf data show similar hotspots:
92.62% 92.62% [kernel.kallsyms] [k] native_queued_spin_lock_slowpath - -
46.20% native_queued_spin_lock_slowpath;_raw_spin_lock_irqsave;release_pages;tlb_flush_mmu;tlb_finish_mmu;unmap_region;__do_munmap;__vm_munmap;__x64_sys_munmap;do_syscall_64;entry_SYSCALL_64_after_hwframe;munmap
46.13% native_queued_spin_lock_slowpath;_raw_spin_lock_irqsave;pagevec_lru_move_fn;lru_add_drain_cpu;lru_add_drain;unmap_region;__do_munmap;__vm_munmap;__x64_sys_munmap;do_syscall_64;entry_SYSCALL_64_after_hwframe;munmap
> Whether the problem is per_cpu_pages or some other structure, it's not
> struct mce's fault in all likelihood -- it's just the messenger.
Agreed. The mce patch itself is innocent, it just changes other domains'
variables' alignment indeliberately.
Thanks,
Feng
> --
> Mel Gorman
> SUSE Labs
next prev parent reply other threads:[~2020-08-25 6:49 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-25 11:44 [x86/mce] 1de08dccd3: will-it-scale.per_process_ops -14.1% regression kernel test robot
2020-04-25 13:01 ` Borislav Petkov
2020-08-18 8:29 ` [LKP] " Feng Tang
2020-08-18 20:06 ` Luck, Tony
2020-08-19 2:04 ` Feng Tang
2020-08-19 2:23 ` Luck, Tony
2020-08-19 3:04 ` Feng Tang
2020-08-19 3:15 ` Feng Tang
2020-08-21 2:02 ` Feng Tang
2020-08-24 15:14 ` Borislav Petkov
2020-08-24 15:33 ` Feng Tang
2020-08-24 15:38 ` Luck, Tony
2020-08-24 15:48 ` Feng Tang
2020-08-24 16:12 ` Borislav Petkov
2020-08-24 16:56 ` Mel Gorman
2020-08-25 6:49 ` Feng Tang [this message]
2020-08-25 6:23 ` Feng Tang
2020-08-25 16:44 ` Luck, Tony
2020-08-26 1:45 ` Feng Tang
2020-08-28 17:48 ` Borislav Petkov
2020-08-31 2:16 ` Feng Tang
2020-08-31 7:56 ` Mel Gorman
2020-08-31 8:23 ` Feng Tang
2020-08-31 8:55 ` Mel Gorman
2020-08-31 12:53 ` Feng Tang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200825064926.GB83850@shbuild999.sh.intel.com \
--to=feng.tang@intel.com \
--cc=bp@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@lists.01.org \
--cc=mgorman@suse.de \
--cc=rong.a.chen@intel.com \
--cc=tony.luck@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).