From: Jiri Olsa <jolsa@redhat.com>
To: Feng Tang <feng.tang@intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
kernel test robot <rong.a.chen@intel.com>,
Ingo Molnar <mingo@kernel.org>,
Vince Weaver <vincent.weaver@maine.edu>,
Jiri Olsa <jolsa@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>,
Ravi Bangoria <ravi.bangoria@linux.ibm.com>,
Stephane Eranian <eranian@google.com>,
Thomas Gleixner <tglx@linutronix.de>,
LKML <linux-kernel@vger.kernel.org>,
lkp@lists.01.org, andi.kleen@intel.com, ying.huang@intel.com
Subject: Re: [LKP] Re: [perf/x86] 81ec3f3c4c: will-it-scale.per_process_ops -5.5% regression
Date: Fri, 21 Feb 2020 14:20:48 +0100 [thread overview]
Message-ID: <20200221132048.GE652992@krava> (raw)
In-Reply-To: <20200221080325.GA67807@shbuild999.sh.intel.com>
On Fri, Feb 21, 2020 at 04:03:25PM +0800, Feng Tang wrote:
>
> On Wed, Feb 05, 2020 at 01:58:04PM +0100, Peter Zijlstra wrote:
> > On Wed, Feb 05, 2020 at 08:32:16PM +0800, kernel test robot wrote:
> > > Greeting,
> > >
> > > FYI, we noticed a -5.5% regression of will-it-scale.per_process_ops due to commit:
> > >
> > >
> > > commit: 81ec3f3c4c4d78f2d3b6689c9816bfbdf7417dbb ("perf/x86: Add check_period PMU callback")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > >
> >
> > I'm fairly sure this bisect/result is bogus.
>
>
> Hi Peter,
>
> Some updates:
>
> We checked more on this. We run 14 times test for it, and the
> results are consistent about the 5.5% degradation, and we
> run the same test on several other platforms, whose test results
> are also consistent, though there are no such -5.5% seen.
>
> We are also curious that the commit seems to be completely not
> relative to this scalability test of signal, which starts a task
> for each online CPU, and keeps calling raise(), and calculating
> the run numbers.
>
> One experiment we did is checking which part of the commit
> really affects the test, and it turned out to be the change of
> "struct pmu". Effectively, applying this patch upon 5.0-rc6
> which triggers the same regression.
>
> diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
> index 1d5c551..e1a0517 100644
> --- a/include/linux/perf_event.h
> +++ b/include/linux/perf_event.h
> @@ -447,6 +447,11 @@ struct pmu {
> * Filter events for PMU-specific reasons.
> */
> int (*filter_match) (struct perf_event *event); /* optional */
> +
> + /*
> + * Check period value for PERF_EVENT_IOC_PERIOD ioctl.
> + */
> + int (*check_period) (struct perf_event *event, u64 value); /* optional */
> };
>
> So likely, this commit changes the layout of the kernel text
> and data, which may trigger some cacheline level change. From
> the system map of the 2 kernels, a big trunk of symbol's address
> changes which follow the global "pmu",
nice, I wonder we could see that in perf c2c output ;-)
I'll try to run and check
thanks,
jirka
>
> 5.0-rc6-systemap:
>
> ffffffff8221d000 d pmu
> ffffffff8221d100 d pmc_reserve_mutex
> ffffffff8221d120 d amd_f15_PMC53
> ffffffff8221d160 d amd_f15_PMC50
>
> 5.0-rc6+pmu-change-systemap:
>
> ffffffff8221d000 d pmu
> ffffffff8221d120 d pmc_reserve_mutex
> ffffffff8221d140 d amd_f15_PMC53
> ffffffff8221d180 d amd_f15_PMC50
>
> But we can hardly identify which exact symbol is responsible
> for the change, as too many symbols are offseted.
>
> btw, we've seen similar case that an irrelevant commit changes
> the benchmark, like a hugetlb patch improves pagefault test on
> a platform that never uses hugetlb https://lkml.org/lkml/2020/1/14/150
>
> Thanks,
> Feng
>
> > _______________________________________________
> > LKP mailing list -- lkp@lists.01.org
> > To unsubscribe send an email to lkp-leave@lists.01.org
>
next prev parent reply other threads:[~2020-02-21 13:21 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-05 12:32 [perf/x86] 81ec3f3c4c: will-it-scale.per_process_ops -5.5% regression kernel test robot
2020-02-05 12:58 ` Peter Zijlstra
2020-02-06 3:04 ` [LKP] " Li, Philip
2020-02-21 8:03 ` Feng Tang
2020-02-21 10:58 ` Peter Zijlstra
2020-02-21 13:20 ` Jiri Olsa [this message]
2020-02-23 14:11 ` Feng Tang
2020-02-23 17:37 ` Linus Torvalds
2020-02-24 0:33 ` Feng Tang
2020-02-24 1:06 ` Linus Torvalds
2020-02-24 1:58 ` Huang, Ying
2020-02-24 2:19 ` Feng Tang
2020-02-24 13:20 ` Feng Tang
2020-02-24 19:24 ` Linus Torvalds
2020-02-24 19:42 ` Kleen, Andi
2020-02-24 20:09 ` Linus Torvalds
2020-02-24 20:47 ` Linus Torvalds
2020-02-24 21:20 ` Eric W. Biederman
2020-02-24 21:43 ` Linus Torvalds
2020-02-24 21:59 ` Eric W. Biederman
2020-02-24 22:12 ` Linus Torvalds
2020-02-25 2:57 ` Feng Tang
2020-02-25 3:15 ` Linus Torvalds
2020-02-25 4:53 ` Feng Tang
2020-02-23 19:36 ` Jiri Olsa
2020-02-21 18:05 ` Kleen, Andi
2020-02-22 12:43 ` Feng Tang
2020-02-22 17:08 ` Kleen, Andi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200221132048.GE652992@krava \
--to=jolsa@redhat.com \
--cc=acme@kernel.org \
--cc=acme@redhat.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=andi.kleen@intel.com \
--cc=eranian@google.com \
--cc=feng.tang@intel.com \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@lists.01.org \
--cc=mingo@kernel.org \
--cc=naveen.n.rao@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=ravi.bangoria@linux.ibm.com \
--cc=rong.a.chen@intel.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vincent.weaver@maine.edu \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).