From: Feng Tang <feng.tang@intel.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Jiri Olsa <jolsa@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
kernel test robot <rong.a.chen@intel.com>,
Ingo Molnar <mingo@kernel.org>,
Vince Weaver <vincent.weaver@maine.edu>,
Jiri Olsa <jolsa@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>,
Ravi Bangoria <ravi.bangoria@linux.ibm.com>,
Stephane Eranian <eranian@google.com>,
Thomas Gleixner <tglx@linutronix.de>,
LKML <linux-kernel@vger.kernel.org>,
lkp@lists.01.org, andi.kleen@intel.com, "Huang,
Ying" <ying.huang@intel.com>
Subject: Re: [LKP] Re: [perf/x86] 81ec3f3c4c: will-it-scale.per_process_ops -5.5% regression
Date: Mon, 24 Feb 2020 08:33:01 +0800 [thread overview]
Message-ID: <20200224003301.GA5061@shbuild999.sh.intel.com> (raw)
In-Reply-To: <CAHk-=wjKFTzfDWjAAabHTZcityeLpHmEQRrKdTuk0f4GWcoohQ@mail.gmail.com>
Hi Linus,
On Sun, Feb 23, 2020 at 09:37:06AM -0800, Linus Torvalds wrote:
> On Sun, Feb 23, 2020 at 6:11 AM Feng Tang <feng.tang@intel.com> wrote:
> >
> > I tried to use perf-c2c on one platform (not the one that show
> > the 5.5% regression), and found the main "hitm" points to the
> > "root_user" global data, as there is a task for each CPU doing
> > the signal stress test, and both __sigqueue_alloc() and
> > __sigqueue_free() will call get_user() and free_uid() to inc/dec
> > this root_user's refcount.
>
> What's around it for you?
>
> There might be that 'uidhash_lock' spinlock right next to it, and
> maybe that exacerbates the issue?
The system map shows:
ffffffff8225b520 d __syscall_meta__ptrace
ffffffff8225b560 d args__ptrace
ffffffff8225b580 d types__ptrace
ffffffff8225b5c0 D root_user
ffffffff8225b680 D init_user_ns
ffffffff8225b880 d ratelimit_state.56624
ffffffff8225b8c0 d event_exit__sigsuspend
I also searched the uidhack_lock,
ffffffff82b04c80 b uidhash_lock
> > Then I added some alignement inside struct "user_struct" (for
> > "root_user"), then the -5.5% is gone, with a +2.6% instead.
>
> Do you actually need to align things inside the struct, or is it
> sufficient to just align the structure itself?
Initially I justed add the ____cacheline_aligned after the definition
of struct 'user_struct', which only makes the following 'init_user_ns'
aligned, and test result doesn't show much change.
Then I added
struct user_struct {
+ char dummy[0] ____cacheline_aligned;
to make 'root_user' itself aligned.
> IOW, is the cache conflicts _within_ the user_struct itself, or is it
> with some nearby data (like that uidhash_lock or whatever?)
From the perf c2c data, and the source code checking, the conflicts
only happens for root_user.__count, and root_user.sigpending, as
all running tasks are accessing this global data for get/put and
other operations.
> > One thing I don't understand is, this -5.5% only happens in
> > one 2 sockets, 96C/192T Cascadelake platform, as we've run
> > the same test on several different platforms. In therory,
> > the false sharing may also take effect?
>
> Is that the biggest machine you have access to?
>
> Maybe it just isn't noticeable with smaller core counts. A lot of
> conflict loads tend to have "exponential" behavior - when things get
> overloaded, performance plummets because it just makes things worse as
> everybody gets slower at that contention point and now it gets even
> more contended...
No, it's not the biggest, I tried another machine 'Xeon Phi(TM) CPU 7295',
which has 72C/288T, and the regression is not seen. This is the part
confusing me :)
Also I've tried to limit the concurrent task number from 192 to 96/48/24/6/1,
and the regression number did get smaller following the task decrease.
Thanks,
Feng
> Linus
next prev parent reply other threads:[~2020-02-24 0:33 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-05 12:32 [perf/x86] 81ec3f3c4c: will-it-scale.per_process_ops -5.5% regression kernel test robot
2020-02-05 12:58 ` Peter Zijlstra
2020-02-06 3:04 ` [LKP] " Li, Philip
2020-02-21 8:03 ` Feng Tang
2020-02-21 10:58 ` Peter Zijlstra
2020-02-21 13:20 ` Jiri Olsa
2020-02-23 14:11 ` Feng Tang
2020-02-23 17:37 ` Linus Torvalds
2020-02-24 0:33 ` Feng Tang [this message]
2020-02-24 1:06 ` Linus Torvalds
2020-02-24 1:58 ` Huang, Ying
2020-02-24 2:19 ` Feng Tang
2020-02-24 13:20 ` Feng Tang
2020-02-24 19:24 ` Linus Torvalds
2020-02-24 19:42 ` Kleen, Andi
2020-02-24 20:09 ` Linus Torvalds
2020-02-24 20:47 ` Linus Torvalds
2020-02-24 21:20 ` Eric W. Biederman
2020-02-24 21:43 ` Linus Torvalds
2020-02-24 21:59 ` Eric W. Biederman
2020-02-24 22:12 ` Linus Torvalds
2020-02-25 2:57 ` Feng Tang
2020-02-25 3:15 ` Linus Torvalds
2020-02-25 4:53 ` Feng Tang
2020-02-23 19:36 ` Jiri Olsa
2020-02-21 18:05 ` Kleen, Andi
2020-02-22 12:43 ` Feng Tang
2020-02-22 17:08 ` Kleen, Andi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200224003301.GA5061@shbuild999.sh.intel.com \
--to=feng.tang@intel.com \
--cc=acme@kernel.org \
--cc=acme@redhat.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=andi.kleen@intel.com \
--cc=eranian@google.com \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@lists.01.org \
--cc=mingo@kernel.org \
--cc=naveen.n.rao@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=ravi.bangoria@linux.ibm.com \
--cc=rong.a.chen@intel.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vincent.weaver@maine.edu \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).