Re: question about RCU dynticks_nesting

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, May 04, 2015 at 11:59:05AM -0400, Rik van Riel wrote:
> On 05/04/2015 05:26 AM, Paolo Bonzini wrote:
> 
> > Isn't this racy?
> > 
> > 	synchronize_rcu CPU		nohz CPU
> > 	---------------------------------------------------------
> > 					set flag = 0
> > 	read flag = 0
> > 					return to userspace
> > 	set TIF_NOHZ
> > 
> > and there's no guarantee that TIF_NOHZ is ever processed by the nohz CPU.
> 
> Looking at the code some more, a flag is not going to be enough.
> 
> An irq can hit while we are in kernel mode, leading to the
> task's "rcu active" counter being incremented twice.
> 
> However, currently the RCU code seems to use a much more
> complex counting scheme, with a different increment for
> kernel/task use, and irq use.
> 
> This counter seems to be modeled on the task preempt_counter,
> where we do care about whether we are in task context, irq
> context, or softirq context.
> 
> On the other hand, the RCU code only seems to care about
> whether or not a CPU is in an extended quiescent state,
> or is potentially in an RCU critical section.
> 
> Paul, what is the reason for RCU using a complex counter,
> instead of a simple increment for each potential kernel/RCU
> entry, like rcu_read_lock() does with CONFIG_PREEMPT_RCU
> enabled?

Heh!  I found out why the hard way.

You see, there are architectures where a CPU can enter an interrupt level
without ever exiting, and perhaps vice versa.  But only if that CPU is
non-idle at the time.  So, when a CPU enters idle, it is necessary to
reset the interrupt nesting to zero.  But that means that it is in turn
necessary to count task-level nesting separately from interrupt-level
nesting, so that we can determine when the CPU goes idle from a task-level
viewpoint.  Hence the use of masks and fields within the counter.

It -might- be possible to simplify this somewhat, especially now that
we have unified idle loops.  Except that I don't trust the architectures
to be reasonable about this at this point.  Furthermore, the associated
nesting checks do trigger when people are making certain types of changes
to architectures, so it is a useful debugging tool.  Which is another
reason that I am reluctant to change it.

> In fact, would we be able to simply use tsk->rcu_read_lock_nesting
> as an indicator of whether or not we should bother waiting on that
> task or CPU when doing synchronize_rcu?

Depends on exactly what you are asking.  If you are asking if I could add
a few more checks to preemptible RCU and speed up grace-period detection
in a number of cases, the answer is very likely "yes".  This is on my
list, but not particularly high priority.  If you are asking whether
CPU 0 could access ->rcu_read_lock_nesting of some task running on
some other CPU, in theory, the answer is "yes", but in practice that
would require putting full memory barriers in both rcu_read_lock()
and rcu_read_unlock(), so the real answer is "no".

Or am I missing your point?

							Thanx, Paul