From: "Paul E. McKenney" <paulmck@kernel.org>
To: Joel Fernandes <joel@joelfernandes.org>
Cc: Hillf Danton <hdanton@sina.com>,
Kalesh Singh <kaleshsingh@google.com>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2] EXP rcu: Move expedited grace period (GP) work to RT kthread_worker
Date: Wed, 13 Apr 2022 11:07:09 -0700 [thread overview]
Message-ID: <20220413180709.GN4285@paulmck-ThinkPad-P17-Gen-1> (raw)
In-Reply-To: <CAEXW_YTmZnk_kFw48HeyyFTXZzfj1cPdw+BaOra14JiWJh6kNg@mail.gmail.com>
On Wed, Apr 13, 2022 at 01:21:20PM -0400, Joel Fernandes wrote:
> Hi Paul,
>
>
> On Wed, Apr 13, 2022 at 8:07 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> >
> > On Wed, Apr 13, 2022 at 07:37:11PM +0800, Hillf Danton wrote:
> > > On Sat, 9 Apr 2022 08:56:12 -0700 Paul E. McKenney wrote:
> > > > On Sat, Apr 09, 2022 at 03:17:40PM +0800, Hillf Danton wrote:
> > > > > On Fri, 8 Apr 2022 10:53:53 -0700 Kalesh Singh wrote
> > > > > > Thanks for the discussion everyone.
> > > > > >
> > > > > > We didn't fully switch to kthread workers to avoid changing the
> > > > > > behavior for users that dont need this low latency exp GPs. Another
> > > > > > (and perhaps more important) reason is because kthread_worker offers
> > > > > > reduced concurrency than workqueues which Pual reported can pose
> > > > > > issues on systems with a large number of CPUs.
> > > > >
> > > > > A second ... what issues were reported wrt concurrency, given the output
> > > > > of grep -nr workqueue block mm drivers.
> > > > >
> > > > > Feel free to post a URL link to the issues.
> > > >
> > > > The issues can be easily seen by inspecting kthread_queue_work() and
> > > > the functions that it invokes. In contrast, normal workqueues uses
> > > > per-CPU mechanisms to avoid contention, as can equally easily be seen
> > > > by inspecting queue_work_on() and the functions that it invokes.
> > >
> > > The worker from kthread_create_worker() roughly matches unbound workqueue
> > > that can get every CPU overloaded, thus the difference in implementation
> > > details between kthread worker and WQ worker (either bound or unbound) can
> > > be safely ignored if the kthread method works, given that prioirty is barely
> > > a cure to concurrency issues.
> >
> > Please look again, this time taking lock contention in to account,
> > keeping in mind that systems with several hundred CPUs are reasonably
> > common and that systems with more than a thousand CPUs are not unheard of.
>
> You are talking about lock contention in the kthread_worker infra
> which unbound WQ does not suffer from, right? I don't think the worker
> lock contention will be an issue unless several
> synchronize_rcu_expedited() calls are trying to queue work at the same
> time. Did I miss something? Considering synchronize_rcu_expedited()
> can block in the normal case (blocking is a pretty heavy operation
> involving the scheduler and load balancers), I don't see how
> contending on the worker infra locks can be an issue. If it was
> call_rcu() , then I can relate to any contention since that executes
> much more often.
Think in terms of a system with 1536 CPUs (which IBM would be extremely
happy to sell you, last I checked). This has 96 leaf rcu_node structures.
Keeping that in mind, take another look at that code.
And in the past there have been real systems with 256 leaf rcu_node
structures.
> I think the argument about too many things being RT is stronger though.
Fair enough. Except that this could be dealt with by conditionally
setting SCHED_FIFO. But the lock contention would remain.
Thanx, Paul
> Thanks,
>
> Joel
>
>
> >
> >
> > Thanx, Paul
> >
> > > Hillf
> > > >
> > > > Please do feel free to take a look.
> > > >
> > > > If taking a look does not convince you, please construct some in-kernel
> > > > benchmarks to test the scalability of these two mechanisms. Please note
> > > > that some care will be required to make sure that you are doing a valid
> > > > apples-to-apples comparison.
> > > >
> > > > Thanx, Paul
> > > >
prev parent reply other threads:[~2022-04-13 18:07 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-08 4:57 [PATCH v2] EXP rcu: Move expedited grace period (GP) work to RT kthread_worker Kalesh Singh
2022-04-08 10:21 ` Joel Fernandes
2022-04-08 15:25 ` Paul E. McKenney
2022-04-08 15:39 ` Steven Rostedt
2022-04-08 15:58 ` Paul E. McKenney
2022-04-08 21:27 ` Steven Rostedt
2022-04-08 10:42 ` Joel Fernandes
2022-04-08 14:34 ` Paul E. McKenney
2022-04-08 14:41 ` Joel Fernandes
2022-04-08 15:34 ` Paul E. McKenney
2022-04-08 17:14 ` Joel Fernandes
2022-04-08 17:39 ` Paul E. McKenney
2022-04-08 17:53 ` Kalesh Singh
2022-04-08 21:42 ` Steven Rostedt
2022-04-08 22:06 ` Kalesh Singh
[not found] ` <20220409071740.6024-1-hdanton@sina.com>
2022-04-09 15:56 ` Paul E. McKenney
[not found] ` <20220413113711.1263-1-hdanton@sina.com>
2022-04-13 14:07 ` Paul E. McKenney
2022-04-13 17:21 ` Joel Fernandes
2022-04-13 18:07 ` Paul E. McKenney [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220413180709.GN4285@paulmck-ThinkPad-P17-Gen-1 \
--to=paulmck@kernel.org \
--cc=hdanton@sina.com \
--cc=joel@joelfernandes.org \
--cc=kaleshsingh@google.com \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).