NMI reentrant RCU list for -rt kernels

NMI reentrant RCU list for -rt kernels

[Mathieu Desnoyers]

Hi Paul,

Following your presentation on RCU lists for -rt kernel, discussing with Bill
Huey led me to the following idea that could solve the problem of NMI reentrancy
of RCU read side in the -rt kernels.

If we consider that the RCU list modification that makes the read side
lock preemptible is only needed for very long code paths, we could leave the
original RCU implementation along with the preemptible one, so that very short
and frequent code paths could benefit of using the very cheap preempt count
protection without having a too big impact on the scheduler latency.

For instance, my LTTng tracer disables the preemption for about 95 ns, which I
doubt would be a problem for real-time behavior. I could easily fix maximum
a maximum list size so it can be run in a constant time.

So, basically, the idea is to have two RCU API that could take names like :
atomic_rcu_* and rcu_*

Does this idea make sense ?

Mathieu


OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 

Re: NMI reentrant RCU list for -rt kernels

[Esben Nielsen]

On Sat, 22 Jul 2006, Mathieu Desnoyers wrote:

> Hi Paul,
>
> Following your presentation on RCU lists for -rt kernel, discussing with Bill
> Huey led me to the following idea that could solve the problem of NMI reentrancy
> of RCU read side in the -rt kernels.
>
> If we consider that the RCU list modification that makes the read side
> lock preemptible is only needed for very long code paths, we could leave the
> original RCU implementation along with the preemptible one, so that very short
> and frequent code paths could benefit of using the very cheap preempt count
> protection without having a too big impact on the scheduler latency.
>
> For instance, my LTTng tracer disables the preemption for about 95 ns, which I
> doubt would be a problem for real-time behavior. I could easily fix maximum
> a maximum list size so it can be run in a constant time.
>
> So, basically, the idea is to have two RCU API that could take names like :
> atomic_rcu_* and rcu_*
>
> Does this idea make sense ?

No,

1) Can you readily identify the very short code pathes? What about future 
code added to the kernel?
2) Having two parellel systems is a bad idea.
3) I believe RCU can be made much cheaper than the 
current implementation which look horrible.

I remember once discussing RCU on the list. I came up with the idea 
rcu_read_lock()/unlock() to be implemented as a per-task counter just as
preempt_disable()/disable(). The run-queue then has a sum of all the 
counters of tasks on that cpu (minus the counter for the current task).
I even made some sample code...
The only reason this wasn't considered working was the migration from CPU 
to CPU. I frankly can't see why this couldn't be fixed.

So the answer to you is: No. Fix the preemptible RCU instead. You have an 
idea above.

Esben


>
> Mathieu
>
>
> OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
> Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: NMI reentrant RCU list for -rt kernels

[Bill Huey]

On Sat, Jul 22, 2006 at 07:14:22PM +0100, Esben Nielsen wrote:
> On Sat, 22 Jul 2006, Mathieu Desnoyers wrote:
> >So, basically, the idea is to have two RCU API that could take names like :
> >atomic_rcu_* and rcu_*
> >
> >Does this idea make sense ?
> 
> No,
> 
> 1) Can you readily identify the very short code pathes? What about future 
> code added to the kernel?
> 2) Having two parellel systems is a bad idea.
> 3) I believe RCU can be made much cheaper than the 
> current implementation which look horrible.
> 
> I remember once discussing RCU on the list. I came up with the idea 
> rcu_read_lock()/unlock() to be implemented as a per-task counter just as
> preempt_disable()/disable(). The run-queue then has a sum of all the 
> counters of tasks on that cpu (minus the counter for the current task).
> I even made some sample code...
> The only reason this wasn't considered working was the migration from CPU 
> to CPU. I frankly can't see why this couldn't be fixed.
> 
> So the answer to you is: No. Fix the preemptible RCU instead. You have an 
> idea above.

Hello,

For Mathieu's uses, it's critical to have a short a path as possible in his
instrumentation code since the results can be effected by it as well as general
impact on the kernel.

The reason why the old RCU read-side logic is ok is that in the -rt kernel RCU
is use to protect things like dcache_lock and other large kernel subsystems. A
non-preemptible RCU would otherwise make all locks in the file system with a
RCU critical section above it in the lock graph non-preemptible or else it
violates the locking rules resulting in dead locking. Since Mathieu's NMI code
doesn't take other kernel locks outside of his own code, it won't create a
situation where it forces parts of the -rt system back below an RCU read-side
section to be non-preemptible.

A preempt_disable/enable should be good enough to restore the previous RCU
behavior just for Mathieu's NMI code with maybe a different function for RCU
synchronization. Making RCU safe for NMI isn't really necessary, but I'm sure
it won't stop you (Paul) from trying. :)

bill

Re: NMI reentrant RCU list for -rt kernels

[Bill Huey]

On Mon, Jul 24, 2006 at 10:24:20AM -0700, Bill Huey wrote:
> On Sat, Jul 22, 2006 at 07:14:22PM +0100, Esben Nielsen wrote:
> > On Sat, 22 Jul 2006, Mathieu Desnoyers wrote:

[discussion about making preemptible RCU both safe and fast for use in
NMI]

> > I remember once discussing RCU on the list. I came up with the idea 
> > rcu_read_lock()/unlock() to be implemented as a per-task counter just as
> > preempt_disable()/disable(). The run-queue then has a sum of all the 
> > counters of tasks on that cpu (minus the counter for the current task).
> > I even made some sample code...
> > The only reason this wasn't considered working was the migration from CPU 
> > to CPU. I frankly can't see why this couldn't be fixed.

Forgot to mention that after talking to Paul that one of his ideas for
NMI safe preemptible RCU was very close to what preemption threshold (threadX
RTOS) and that if he was going to go that route that he should think about if
he should generalize it for the scheduler.

> > So the answer to you is: No. Fix the preemptible RCU instead. You have an 
> > idea above.
> 
> Hello,
> 
> For Mathieu's uses, it's critical to have a short a path as possible in his
> instrumentation code since the results can be effected by it as well as general
> impact on the kernel.
> 
> The reason why the old RCU read-side logic is ok is that in the -rt kernel RCU
> is use to protect things like dcache_lock and other large kernel subsystems. A
> non-preemptible RCU would otherwise make all locks in the file system with a
> RCU critical section above it in the lock graph non-preemptible or else it
> violates the locking rules resulting in dead locking. Since Mathieu's NMI code
> doesn't take other kernel locks outside of his own code, it won't create a
> situation where it forces parts of the -rt system back below an RCU read-side
> section to be non-preemptible.
> 
> A preempt_disable/enable should be good enough to restore the previous RCU
> behavior just for Mathieu's NMI code with maybe a different function for RCU
> synchronization. Making RCU safe for NMI isn't really necessary, but I'm sure
> it won't stop you (Paul) from trying. :)

bill