From: Joel Fernandes <joel@joelfernandes.org>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Ingo Molnar <mingo@redhat.com>,
Josh Triplett <josh@joshtriplett.org>,
Lai Jiangshan <jiangshanlai@gmail.com>,
Marco Elver <elver@google.com>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
rcu <rcu@vger.kernel.org>, Steven Rostedt <rostedt@goodmis.org>,
"Uladzislau Rezki (Sony)" <urezki@gmail.com>,
Frederic Weisbecker <fweisbec@gmail.com>,
Neeraj upadhyay <neeraj.iitr10@gmail.com>,
Alan Stern <stern@rowland.harvard.edu>
Subject: Re: [PATCH v7 6/6] rcu/segcblist: Add additional comments to explain smp_mb()
Date: Wed, 21 Oct 2020 11:57:04 -0700 [thread overview]
Message-ID: <CAEXW_YTtYspPNw_eL1vmGXhY8nJ8uQonSc5KuA1weYv3G+bWPg@mail.gmail.com> (raw)
In-Reply-To: <20201019123730.GA34192@lothringen>
On Mon, Oct 19, 2020 at 5:37 AM Frederic Weisbecker <frederic@kernel.org> wrote:
>
[..]
> > >
> > > I'm very likely missing something obvious somewhere.
> > >
> > > CPU 0 CPU 1
> > > rcu_barrier() call_rcu()/rcu_segcblist_enqueue()
> > > ------------ --------
> > >
> > > smp_mb();
> > > inc_len();
> > > smp_mb();
> > > queue callback;
> > > for_each_possible_cpu(cpu)
> > > if (!rcu_segcblist_n_cbs(&rdp->cblist))
> > > continue;
> > >
> >
> > > invoke_callback
> >
> > If CPU 0 saw the enqueue of the callback (that is the CPU 1's writes to the
> > segcb_list propagated to CPU 0), then it would have also seen the
> > effects of the inc_len. I forced this case in my last litmus test by this
> > code in P1():
>
> But then I can't find to which part of rcu_barrier() this refers to.
> I see the len read before anything else.
>
> >
> > r1 = READ_ONCE(*enq);
> > smp_mb(); /* barrier Just for test purpose to show that the.. */
> > /* ..rcu_barrier() saw list modification */
> >
> > On the other hand, if CPU 0 did not see the enqueue, then there is really no
> > issue. Since that is the same case where call_rcu() happened _after_ the
> > rcu_barrier() and there's no race. rcu_barrier() does not need to wait if
> > there was no callback enqueued.
> >
> > This is not exactly the easiest thing to explain, hence the litmus.
>
> Now, reading the documentation of rcu_barrier() (thanks to you!):
>
> Pseudo-code using rcu_barrier() is as follows:
>
> 1. Prevent any new RCU callbacks from being posted.
> 2. Execute rcu_barrier().
> 3. Allow the module to be unloaded.
>
Basically, you are saying that if all CPUs agree that len == 0
henceforth (through other memory barriers), then callback enqueuing
does not need a memory barrier before setting length to 0.
I think that makes sense but is it worth removing the memory barrier
before WRITE(len, 1) and hoping after #1, the caller would have
ensured things are fine? Also I am not sure if the above is the only
usecase for rcu_barrier().
> I think with point 1, it is assumed that the caller of rcu_barrier() must have
> not only stopped but also sync'ed with the possible enqueuers. Correct me if I'm wrong
> here. So for example if a kthread used to post the module RCU callbacks, calling kthread_stop()
> does the job as it prevents from further RCU callbacks from being enqueued and it also syncs
> with the kthread thanks to the completion implied by any caller of kthread_stop() which then
> sees what the kthread has read and written, including RCU callbacks enqueued. So if the caller
> of kthread_stop() calls rcu_barrier() right after, rcu_barrier() should see at least the len
> corresponding to the last enqueue.
>
> cancel_work_sync() also seem to really sync as well. I'm less sure about del_timer_sync().
>
> Say we have:
>
> expire_timers (CPU 0) CPU 1
> ------------- -----------
> detach_timer(timer)
> raw_spin_unlock(&base->lock);
> call_timer_fn(timer, fn, baseclk);
> -> enqueue callback
> //would need at least smp_wmb() here
> base->running_timer = NULL;
>
> del_timer_sync() {
> raw_spin_lock(&base->lock);
> if (base->running_timer != timer)
> ret = detach_if_pending(timer, base, true);
> if (!timer_pending())
> return 0;
> raw_spin_unlock(&base->lock);
> }
> //would need at least smp_rmb() here
Regarding "would need at least smp_rmb.." :
But the rcu_barrier() has the control dependency we discussed in last
emails, between READ(len) and whatever follows the rcu_barrier().
That itself will provide the ordering right?
> //although rcu_seq_start() implies a full barrier
> rcu_barrier() {
> // Sees rcu_segcblist_n_cbs(rdp(CPU 0)->cblist) == 0
> // So ignore it
>
>
> But I'm sure I'm missing something obvious. That's my specialism.
I could be missing something too :-/. But I'll include this patch in
my next posting anyway and let us also maybe see if Paul disagrees.
thanks,
- Joel
next prev parent reply other threads:[~2020-10-21 18:57 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-15 0:22 [PATCH v7 0/6] Add support for length of each segment in the segcblist Joel Fernandes (Google)
2020-10-15 0:22 ` [PATCH v7 1/6] rcu/tree: Make rcu_do_batch count how many callbacks were executed Joel Fernandes (Google)
2020-10-15 0:22 ` [PATCH v7 2/6] rcu/segcblist: Add counters to segcblist datastructure Joel Fernandes (Google)
2020-10-15 12:21 ` Frederic Weisbecker
2020-10-17 1:31 ` joel
2020-10-21 15:33 ` joel
2020-10-21 21:53 ` Frederic Weisbecker
2020-10-21 22:31 ` Joel Fernandes
2020-10-18 8:23 ` [rcu/segcblist] e08055898f: WARNING:at_kernel/rcu/srcutree.c:#cleanup_srcu_struct kernel test robot
2020-10-21 14:40 ` joel
2020-10-15 0:22 ` [PATCH v7 3/6] rcu/trace: Add tracing for how segcb list changes Joel Fernandes (Google)
2020-10-15 0:22 ` [PATCH v7 4/6] rcu/segcblist: Remove useless rcupdate.h include Joel Fernandes (Google)
2020-10-15 0:23 ` [PATCH v7 5/6] rcu/tree: Remove redundant smp_mb() in rcu_do_batch Joel Fernandes (Google)
2020-10-15 0:23 ` [PATCH v7 6/6] rcu/segcblist: Add additional comments to explain smp_mb() Joel Fernandes (Google)
2020-10-15 13:35 ` Frederic Weisbecker
2020-10-17 1:27 ` joel
2020-10-17 3:19 ` joel
2020-10-17 13:29 ` Frederic Weisbecker
2020-10-18 0:35 ` joel
2020-10-19 12:37 ` Frederic Weisbecker
2020-10-21 18:57 ` Joel Fernandes [this message]
2020-10-21 21:16 ` Frederic Weisbecker
2020-10-17 20:24 ` Alan Stern
2020-10-18 20:45 ` Joel Fernandes
2020-10-18 21:15 ` Alan Stern
2020-10-17 14:31 ` Alan Stern
2020-10-18 20:16 ` Joel Fernandes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAEXW_YTtYspPNw_eL1vmGXhY8nJ8uQonSc5KuA1weYv3G+bWPg@mail.gmail.com \
--to=joel@joelfernandes.org \
--cc=elver@google.com \
--cc=frederic@kernel.org \
--cc=fweisbec@gmail.com \
--cc=jiangshanlai@gmail.com \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@redhat.com \
--cc=neeraj.iitr10@gmail.com \
--cc=paulmck@kernel.org \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=stern@rowland.harvard.edu \
--cc=urezki@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).