[lttng-dev] QSBR urcu question

[lttng-dev] QSBR urcu question

[lbj via lttng-dev]

Hello all,

I am using the QSBR flavor of URCU and I have a question: is it ok to call rcu_register_thread more than once per thread? Similarly, is it ok to call rcu_unregister_thread more than once per thread? I am using a library that uses a threadpool I have no control over. A thread from its threadpool invokes a callback function of mine. Threads may be reused. I would like to call rcu_register_thread at the beginning of my callback and rcu_unregister_thread at the end. The thread in question may exit or may be reused at some later point, I have no control over that. So I want to make sure that periodically registering and unregistering the same thread is ok. Thanks for any information.

Jeff
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu question

[Mathieu Desnoyers via lttng-dev]

----- On Apr 5, 2021, at 1:43 PM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Hello all,
> 
> I am using the QSBR flavor of URCU and I have a question: is it ok to call
> rcu_register_thread more than once per thread?

Yes, but you should unregister before registering again.

> Similarly, is it ok to call
> rcu_unregister_thread more than once per thread?

Likewise: you should not have 2 unregistrations back-to-back without a
registration in between.

Note however that frequent registration/unregistration of a thread can lead to
significant performance overhead and lock contention on the rcu_registry_lock,
which is likely something you will want to avoid.

> I am using a library that uses
> a threadpool I have no control over. A thread from its threadpool invokes a
> callback function of mine. Threads may be reused. I would like to call
> rcu_register_thread at the beginning of my callback and rcu_unregister_thread
> at the end. The thread in question may exit or may be reused at some later
> point, I have no control over that. So I want to make sure that periodically
> registering and unregistering the same thread is ok. Thanks for any
> information.

You may want to use the "urcu-bp" flavor instead. It has a slightly higher
overhead than qsbr, but won't require as much fine-tuned integration with the
threadpool runtime.

But if you really want to do an efficient integration of urcu-qsbr with the threadpool
without changing any of the threadpool, here are a few ideas though:

- you could keep a "qsbr_registered" state in a TLS variable owned by your application,
- lazily perform qsbr registration if needed, based on the "qsbr_registered" state,
- put thread "offline" before returning control to the threadpool,
- put thread "online" when entering your code again,
- if the threadpool can call your application before the thread exits, then use that
  to unregister qsbr. Else, you could always use a dummy pthread key to have an application
  destructor called on thread exit, and unregister qsbr that way,

Hoping this help!

Best regards,

Mathieu

> 
> Jeff
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] QSBR urcu read lock question

[lbj via lttng-dev]

Hello all,

I have two different entities that are both protected by QSBR rcu: a policy and a hashtable. In the reclamation thread for the policy I would like to take a read lock so that I can safely iterate through the hashtable. I dont see anything wrong with this, but I just wanted to make sure it was ok since taking an rcu read lock in an rcu reclamation thread seems like it may be a bit suspect. Thanks for any insights, let me know if clarification is needed!

Jeff
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[Mathieu Desnoyers via lttng-dev]

----- On Apr 13, 2021, at 11:19 PM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Hello all,
> 
> I have two different entities that are both protected by QSBR rcu: a policy and
> a hashtable. In the reclamation thread for the policy I would like to take a
> read lock so that I can safely iterate through the hashtable. I dont see
> anything wrong with this, but I just wanted to make sure it was ok since taking
> an rcu read lock in an rcu reclamation thread seems like it may be a bit
> suspect. Thanks for any insights, let me know if clarification is needed!

When you say "the reclamation thread for the policy", do you refer to a call-rcu
worker thread ?

Also, you are aware that RCU read-side lock/unlock are effectively no-ops for
QSBR rcu, right ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[lbj via lttng-dev]

Hi Mathieu,
 When I say “reclamation thread” I do mean the thread launched by call_rcu that is typically responsible for memory deallocations. Is is possible/recommended to register for rcu and then take an rcu-reader lock in such a thread? That is my main question.

As for reader locks being no-ops in QSBR, I read that but dont quite understand it. Something must be preventing memory reclamation of rcu protected elements when I take that lock. 

My specific situation is: I have a QSBR rcu protected “policy” object (just a regular old C++ object that periodically gets refreshed and must be atomically updated because worker cores are reading it while spinning, and they cant slow down). When a new policy is received we invoke call_rcu on the old policy. call_rcu will eventually launch a thread in which the old policy’s resources are reclaimed. In this thread I would like to iterate through another, separate structure, which is also QSBR rcu protected (a urcu hashtable). To do so safely, presumably I must use an rcu readlock. I just want to make sure such a scenario is reasonable and at very least not contra-indicated. Thanks!

Jeff

 

Sent from my iPhone

> On Apr 15, 2021, at 8:20 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
> ----- On Apr 13, 2021, at 11:19 PM, lttng-dev lttng-dev@lists.lttng.org wrote:
> 
>> Hello all,
>> 
>> I have two different entities that are both protected by QSBR rcu: a policy and
>> a hashtable. In the reclamation thread for the policy I would like to take a
>> read lock so that I can safely iterate through the hashtable. I dont see
>> anything wrong with this, but I just wanted to make sure it was ok since taking
>> an rcu read lock in an rcu reclamation thread seems like it may be a bit
>> suspect. Thanks for any insights, let me know if clarification is needed!
> 
> When you say "the reclamation thread for the policy", do you refer to a call-rcu
> worker thread ?
> 
> Also, you are aware that RCU read-side lock/unlock are effectively no-ops for
> QSBR rcu, right ?
> 
> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[Mathieu Desnoyers via lttng-dev]

----- On Apr 15, 2021, at 8:41 AM, lbj lbj137@yahoo.com wrote:

> Hi Mathieu,
> When I say “reclamation thread” I do mean the thread launched by call_rcu that
> is typically responsible for memory deallocations. Is is possible/recommended
> to register for rcu and then take an rcu-reader lock in such a thread? That is
> my main question.
> 
> As for reader locks being no-ops in QSBR, I read that but dont quite understand
> it. Something must be preventing memory reclamation of rcu protected elements
> when I take that lock.

Note that a RCU read-side "lock" is really just a marker about the beginning/end
of a transaction which delays grace periods. We use the name "lock" to match
the kernel RCU APIs, but it should not be considered as doing any kind of mutual
exclusion.

> 
> My specific situation is: I have a QSBR rcu protected “policy” object (just a
> regular old C++ object that periodically gets refreshed and must be atomically
> updated because worker cores are reading it while spinning, and they cant slow
> down). When a new policy is received we invoke call_rcu on the old policy.
> call_rcu will eventually launch a thread in which the old policy’s resources
> are reclaimed. In this thread I would like to iterate through another, separate
> structure, which is also QSBR rcu protected (a urcu hashtable). To do so
> safely, presumably I must use an rcu readlock. I just want to make sure such a
> scenario is reasonable and at very least not contra-indicated. Thanks!

If you look at urcu-call-rcu-impl.h, you will notice that call_rcu_thread()
indeed registers itself as a reader thread.

So the call-rcu callbacks can indeed take a RCU read-side lock, but for QSBR
the story does not end there, because due to the nature of QSBR, the read-side
lock is indeed a no-op, and it relies instead on all registered QSBR reader
threads to periodically invoke urcu_qsbr_quiescent_state() to report that they
are in a quiescent state, or invoke urcu_qsbr_thread_offline() if they expect to be
in a quiescent state for a long period of time (e.g. blocking), followed by
urcu_qsbr_thread_online().

And indeed, the call_rcu_thread puts itself in "offline" mode while awaiting for
grace periods (this is implicitly done within the qsbr synchronize_rcu() implementation)
and when sleeping.

So yes, you should be able to have a RCU read-side from within a call-rcu worker
thread, and it's OK to assume you can do a RCU traversal with the QSBR urcu flavor
from a call-rcu worker thread as well.

Thanks,

Mathieu

> 
> Jeff
> 
> 
> 
> Sent from my iPhone
> 
>> On Apr 15, 2021, at 8:20 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> wrote:
>> 
>> ----- On Apr 13, 2021, at 11:19 PM, lttng-dev lttng-dev@lists.lttng.org wrote:
>> 
>>> Hello all,
>>> 
>>> I have two different entities that are both protected by QSBR rcu: a policy and
>>> a hashtable. In the reclamation thread for the policy I would like to take a
>>> read lock so that I can safely iterate through the hashtable. I dont see
>>> anything wrong with this, but I just wanted to make sure it was ok since taking
>>> an rcu read lock in an rcu reclamation thread seems like it may be a bit
>>> suspect. Thanks for any insights, let me know if clarification is needed!
>> 
>> When you say "the reclamation thread for the policy", do you refer to a call-rcu
>> worker thread ?
>> 
>> Also, you are aware that RCU read-side lock/unlock are effectively no-ops for
>> QSBR rcu, right ?
>> 
>> Thanks,
>> 
>> Mathieu
>> 
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
> > http://www.efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[lbj via lttng-dev]

Mathieu,
Thanks so much for your wealth if information and timely responses, they are greatly appreciated. Final question: is there any harm in explicitly calling rcu_thread_online/rcu_thread_offline from within my call_rcu callback function? From what you described it sounds like it would be redundant, but presumably would be harmless. Correct? Thanks again.
Jeff

Sent from my iPhone

> On Apr 15, 2021, at 9:04 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
> ----- On Apr 15, 2021, at 8:41 AM, lbj lbj137@yahoo.com wrote:
> 
>> Hi Mathieu,
>> When I say “reclamation thread” I do mean the thread launched by call_rcu that
>> is typically responsible for memory deallocations. Is is possible/recommended
>> to register for rcu and then take an rcu-reader lock in such a thread? That is
>> my main question.
>> 
>> As for reader locks being no-ops in QSBR, I read that but dont quite understand
>> it. Something must be preventing memory reclamation of rcu protected elements
>> when I take that lock.
> 
> Note that a RCU read-side "lock" is really just a marker about the beginning/end
> of a transaction which delays grace periods. We use the name "lock" to match
> the kernel RCU APIs, but it should not be considered as doing any kind of mutual
> exclusion.
> 
>> 
>> My specific situation is: I have a QSBR rcu protected “policy” object (just a
>> regular old C++ object that periodically gets refreshed and must be atomically
>> updated because worker cores are reading it while spinning, and they cant slow
>> down). When a new policy is received we invoke call_rcu on the old policy.
>> call_rcu will eventually launch a thread in which the old policy’s resources
>> are reclaimed. In this thread I would like to iterate through another, separate
>> structure, which is also QSBR rcu protected (a urcu hashtable). To do so
>> safely, presumably I must use an rcu readlock. I just want to make sure such a
>> scenario is reasonable and at very least not contra-indicated. Thanks!
> 
> If you look at urcu-call-rcu-impl.h, you will notice that call_rcu_thread()
> indeed registers itself as a reader thread.
> 
> So the call-rcu callbacks can indeed take a RCU read-side lock, but for QSBR
> the story does not end there, because due to the nature of QSBR, the read-side
> lock is indeed a no-op, and it relies instead on all registered QSBR reader
> threads to periodically invoke urcu_qsbr_quiescent_state() to report that they
> are in a quiescent state, or invoke urcu_qsbr_thread_offline() if they expect to be
> in a quiescent state for a long period of time (e.g. blocking), followed by
> urcu_qsbr_thread_online().
> 
> And indeed, the call_rcu_thread puts itself in "offline" mode while awaiting for
> grace periods (this is implicitly done within the qsbr synchronize_rcu() implementation)
> and when sleeping.
> 
> So yes, you should be able to have a RCU read-side from within a call-rcu worker
> thread, and it's OK to assume you can do a RCU traversal with the QSBR urcu flavor
> from a call-rcu worker thread as well.
> 
> Thanks,
> 
> Mathieu
> 
>> 
>> Jeff
>> 
>> 
>> 
>> Sent from my iPhone
>> 
>>> On Apr 15, 2021, at 8:20 AM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>> wrote:
>>> 
>>> ----- On Apr 13, 2021, at 11:19 PM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>> 
>>>> Hello all,
>>>> 
>>>> I have two different entities that are both protected by QSBR rcu: a policy and
>>>> a hashtable. In the reclamation thread for the policy I would like to take a
>>>> read lock so that I can safely iterate through the hashtable. I dont see
>>>> anything wrong with this, but I just wanted to make sure it was ok since taking
>>>> an rcu read lock in an rcu reclamation thread seems like it may be a bit
>>>> suspect. Thanks for any insights, let me know if clarification is needed!
>>> 
>>> When you say "the reclamation thread for the policy", do you refer to a call-rcu
>>> worker thread ?
>>> 
>>> Also, you are aware that RCU read-side lock/unlock are effectively no-ops for
>>> QSBR rcu, right ?
>>> 
>>> Thanks,
>>> 
>>> Mathieu
>>> 
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[Mathieu Desnoyers via lttng-dev]

----- On Apr 15, 2021, at 10:54 AM, lbj lbj137@yahoo.com wrote:

> Mathieu,
> Thanks so much for your wealth if information and timely responses, they are
> greatly appreciated. Final question: is there any harm in explicitly calling
> rcu_thread_online/rcu_thread_offline from within my call_rcu callback function?
> From what you described it sounds like it would be redundant, but presumably
> would be harmless. Correct? Thanks again.

You could indeed invoke pairs of:

  rcu_thread_offline();   <--- emphasis on _offline_ here.
  [ long wait ... ]
  rcu_thread_online();

in that specific order within the call-rcu worker thread. Note that the qsbr state
of the call-rcu worker thread is "online" when it invokes the callbacks, so each callback
should make sure that state is back to "online" before it returns control back
to its caller.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[lbj via lttng-dev]

Thanks Mathieu. Is it safe to assume that if call_rcu is called twice then the callbacks are executed in the order that call_rcu was invoked? I think there is a queue and only one thread that QSBR uses to handle callbacks, i just wanted to make sure that the queue was a guaranteed fifo. 

Sent from my iPhone

> On Apr 15, 2021, at 12:00 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
> ----- On Apr 15, 2021, at 10:54 AM, lbj lbj137@yahoo.com wrote:
> 
>> Mathieu,
>> Thanks so much for your wealth if information and timely responses, they are
>> greatly appreciated. Final question: is there any harm in explicitly calling
>> rcu_thread_online/rcu_thread_offline from within my call_rcu callback function?
>> From what you described it sounds like it would be redundant, but presumably
>> would be harmless. Correct? Thanks again.
> 
> You could indeed invoke pairs of:
> 
>  rcu_thread_offline();   <--- emphasis on _offline_ here.
>  [ long wait ... ]
>  rcu_thread_online();
> 
> in that specific order within the call-rcu worker thread. Note that the qsbr state
> of the call-rcu worker thread is "online" when it invokes the callbacks, so each callback
> should make sure that state is back to "online" before it returns control back
> to its caller.
> 
> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[Mathieu Desnoyers via lttng-dev]

----- On Apr 15, 2021, at 2:11 PM, lbj lbj137@yahoo.com wrote:

> Thanks Mathieu. Is it safe to assume that if call_rcu is called twice then the
> callbacks are executed in the order that call_rcu was invoked? I think there is
> a queue and only one thread that QSBR uses to handle callbacks, i just wanted
> to make sure that the queue was a guaranteed fifo.

It may very well depend on your configuration. For instance, if an application
invokes create_all_cpu_call_rcu_data(), it will create per-cpu worker threads
on each possible CPU. In that configuration, a given thread invoking call_rcu()
twice (back to back) may be migrated from one CPU to another in between, hence
different call-rcu worker threads will be responsible for executing the callbacks,
and they can be executed in any order.

So even though by careful analysis of your specific application configuration you
may presume that the callbacks will be executed in the same order they were invoked,
this is not guaranteed by the API, so I would not recommend relying on FIFO order
assumptions.

And given that the call_rcu API does not guarantee FIFO order, we could eventually
decide to change the order of callback execution from FIFO to LIFO if it leads to
performance improvements due to better cache locality.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] QSBR urcu read lock question

[lbj via lttng-dev]

Thank you again for all your help Mathieu!

Sent from my iPhone

> On Apr 15, 2021, at 3:26 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
> ----- On Apr 15, 2021, at 2:11 PM, lbj lbj137@yahoo.com wrote:
> 
>> Thanks Mathieu. Is it safe to assume that if call_rcu is called twice then the
>> callbacks are executed in the order that call_rcu was invoked? I think there is
>> a queue and only one thread that QSBR uses to handle callbacks, i just wanted
>> to make sure that the queue was a guaranteed fifo.
> 
> It may very well depend on your configuration. For instance, if an application
> invokes create_all_cpu_call_rcu_data(), it will create per-cpu worker threads
> on each possible CPU. In that configuration, a given thread invoking call_rcu()
> twice (back to back) may be migrated from one CPU to another in between, hence
> different call-rcu worker threads will be responsible for executing the callbacks,
> and they can be executed in any order.
> 
> So even though by careful analysis of your specific application configuration you
> may presume that the callbacks will be executed in the same order they were invoked,
> this is not guaranteed by the API, so I would not recommend relying on FIFO order
> assumptions.
> 
> And given that the call_rcu API does not guarantee FIFO order, we could eventually
> decide to change the order of callback execution from FIFO to LIFO if it leads to
> performance improvements due to better cache locality.
> 
> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev