[Qestion] Is preempt_disable/enable needed in non-preemption code path

[Qestion] Is preempt_disable/enable needed in non-preemption code path

[Xu, Yanfei]

Hi experts,

I am learning rcu mechanism and its codes. When looking at the 
rcu_blocking_is_gp(), I found there is a pair preemption disable/enable 
operation in non-preemption code path. And it has been a long time. I 
can't understand why we need it? Is there some thing I missed? If not, 
can we remove the unnecessary operation like blow?


diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index da6f5213fb74..c6d95a00715e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
         if (IS_ENABLED(CONFIG_PREEMPTION))
                 return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
         might_sleep();  /* Check for RCU read-side critical section. */
-       preempt_disable();
         /*
          * If the rcu_state.n_online_cpus counter is equal to one,
          * there is only one CPU, and that CPU sees all prior accesses
@@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
          * Those memory barriers are provided by CPU-hotplug code.
          */
         ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
-       preempt_enable();
         return ret;
  }



Best regards,
Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Paul E. McKenney]

On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
> Hi experts,
> 
> I am learning rcu mechanism and its codes. When looking at the
> rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
> operation in non-preemption code path. And it has been a long time. I can't
> understand why we need it? Is there some thing I missed? If not, can we
> remove the unnecessary operation like blow?

Good point, you are right that preemption is disabled anyway in that block
of code.  However, preempt_disable() and preempt_enable() also prevent the
compiler from moving that READ_ONCE() around.  So my question to you is
whether it is safe to remove those statements entirely or whether they
should instead be replaced by barrier() or similar.

							Thanx, Paul

> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index da6f5213fb74..c6d95a00715e 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
>         if (IS_ENABLED(CONFIG_PREEMPTION))
>                 return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
>         might_sleep();  /* Check for RCU read-side critical section. */
> -       preempt_disable();
>         /*
>          * If the rcu_state.n_online_cpus counter is equal to one,
>          * there is only one CPU, and that CPU sees all prior accesses
> @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
>          * Those memory barriers are provided by CPU-hotplug code.
>          */
>         ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
> -       preempt_enable();
>         return ret;
>  }
> 
> 
> 
> Best regards,
> Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Xu, Yanfei]

On 4/15/21 11:43 PM, Paul E. McKenney wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
> 
> On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
>> Hi experts,
>>
>> I am learning rcu mechanism and its codes. When looking at the
>> rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
>> operation in non-preemption code path. And it has been a long time. I can't
>> understand why we need it? Is there some thing I missed? If not, can we
>> remove the unnecessary operation like blow?
> 
> Good point, you are right that preemption is disabled anyway in that block
> of code.  However, preempt_disable() and preempt_enable() also prevent the
> compiler from moving that READ_ONCE() around.  So my question to you is
> whether it is safe to remove those statements entirely or whether they
> should instead be replaced by barrier() or similar.

Thanks for your reply! :)

Yes, preempt_disable() and preempt_enable() defined in !preemption are 
barrier(). barrier can prevent from reordering that READ_ONCE(), but 
base on my current understanding, volatile in READ_ONCE can also tell 
the compiler not to reorder it. So, I think it's safe?

Best regards,
Yanfei

> 
>                                                          Thanx, Paul
> 
>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>> index da6f5213fb74..c6d95a00715e 100644
>> --- a/kernel/rcu/tree.c
>> +++ b/kernel/rcu/tree.c
>> @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
>>          if (IS_ENABLED(CONFIG_PREEMPTION))
>>                  return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
>>          might_sleep();  /* Check for RCU read-side critical section. */
>> -       preempt_disable();
>>          /*
>>           * If the rcu_state.n_online_cpus counter is equal to one,
>>           * there is only one CPU, and that CPU sees all prior accesses
>> @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
>>           * Those memory barriers are provided by CPU-hotplug code.
>>           */
>>          ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
>> -       preempt_enable();
>>          return ret;
>>   }
>>
>>
>>
>> Best regards,
>> Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Xu, Yanfei]

On 4/16/21 12:18 AM, Xu, Yanfei wrote:
> 
> 
> On 4/15/21 11:43 PM, Paul E. McKenney wrote:
>> [Please note: This e-mail is from an EXTERNAL e-mail address]
>>
>> On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
>>> Hi experts,
>>>
>>> I am learning rcu mechanism and its codes. When looking at the
>>> rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
>>> operation in non-preemption code path. And it has been a long time. I 
>>> can't
>>> understand why we need it? Is there some thing I missed? If not, can we
>>> remove the unnecessary operation like blow?
>>
>> Good point, you are right that preemption is disabled anyway in that 
>> block
>> of code.  However, preempt_disable() and preempt_enable() also prevent 
>> the
>> compiler from moving that READ_ONCE() around.  So my question to you is
>> whether it is safe to remove those statements entirely or whether they
>> should instead be replaced by barrier() or similar.
> 
> Thanks for your reply! :)
> 
> Yes, preempt_disable() and preempt_enable() defined in !preemption are 
> barrier(). barrier can prevent from reordering that READ_ONCE(), but 
> base on my current understanding, volatile in READ_ONCE can also tell 
> the compiler not to reorder it. So, I think it's safe?
> 
> Best regards,
> Yanfei

Hi Paul,
I objdump the function rcu_blocking_is_gp():

after dropping the barrier():
ffffffff81107c50 <rcu_blocking_is_gp>:
ffffffff81107c50:       e8 7b 2a f5 ff          callq  ffffffff8105a6d0 
<__fentry__>
ffffffff81107c55:       8b 05 41 fe 7c 01       mov 
0x17cfe41(%rip),%eax        # ffffffff828d7a9c <rcu_state+0x221c>
ffffffff81107c5b:       55                      push   %rbp
ffffffff81107c5c:       48 89 e5                mov    %rsp,%rbp
ffffffff81107c5f:       5d                      pop    %rbp
ffffffff81107c60:       83 f8 01                cmp    $0x1,%eax
ffffffff81107c63:       0f 9e c0                setle  %al
ffffffff81107c66:       0f b6 c0                movzbl %al,%eax
ffffffff81107c69:       c3                      retq
ffffffff81107c6a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)

the original codes:
ffffffff81107ba0 <rcu_blocking_is_gp>:
ffffffff81107ba0:       e8 2b 2b f5 ff          callq  ffffffff8105a6d0 
<__fentry__>
ffffffff81107ba5:       55                      push   %rbp
ffffffff81107ba6:       48 89 e5                mov    %rsp,%rbp
ffffffff81107ba9:       8b 05 ed fe 7c 01       mov 
0x17cfeed(%rip),%eax        # ffffffff828d7a9c <rcu_state+0x221c>
ffffffff81107baf:       83 f8 01                cmp    $0x1,%eax
ffffffff81107bb2:       5d                      pop    %rbp
ffffffff81107bb3:       0f 9e c0                setle  %al
ffffffff81107bb6:       0f b6 c0                movzbl %al,%eax
ffffffff81107bb9:       c3                      retq
ffffffff81107bba:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)

umm... It did been reordered by compiler after dropping the barrier(), 
however, I think the result will not be effected. Right?

Best regards,
Yanfei

> 
>>
>>                                                          Thanx, Paul
>>
>>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>>> index da6f5213fb74..c6d95a00715e 100644
>>> --- a/kernel/rcu/tree.c
>>> +++ b/kernel/rcu/tree.c
>>> @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
>>>          if (IS_ENABLED(CONFIG_PREEMPTION))
>>>                  return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
>>>          might_sleep();  /* Check for RCU read-side critical section. */
>>> -       preempt_disable();
>>>          /*
>>>           * If the rcu_state.n_online_cpus counter is equal to one,
>>>           * there is only one CPU, and that CPU sees all prior accesses
>>> @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
>>>           * Those memory barriers are provided by CPU-hotplug code.
>>>           */
>>>          ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
>>> -       preempt_enable();
>>>          return ret;
>>>   }
>>>
>>>
>>>
>>> Best regards,
>>> Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Paul E. McKenney]

On Fri, Apr 16, 2021 at 12:18:42AM +0800, Xu, Yanfei wrote:
> 
> 
> On 4/15/21 11:43 PM, Paul E. McKenney wrote:
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > 
> > On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
> > > Hi experts,
> > > 
> > > I am learning rcu mechanism and its codes. When looking at the
> > > rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
> > > operation in non-preemption code path. And it has been a long time. I can't
> > > understand why we need it? Is there some thing I missed? If not, can we
> > > remove the unnecessary operation like blow?
> > 
> > Good point, you are right that preemption is disabled anyway in that block
> > of code.  However, preempt_disable() and preempt_enable() also prevent the
> > compiler from moving that READ_ONCE() around.  So my question to you is
> > whether it is safe to remove those statements entirely or whether they
> > should instead be replaced by barrier() or similar.
> 
> Thanks for your reply! :)
> 
> Yes, preempt_disable() and preempt_enable() defined in !preemption are
> barrier(). barrier can prevent from reordering that READ_ONCE(), but base on
> my current understanding, volatile in READ_ONCE can also tell the compiler
> not to reorder it. So, I think it's safe?

Maybe.

Please keep in mind that although the compiler is prohibited from
reordering volatile accesses with each other, there is nothing stopping
it from reordering volatile accesses with non-volatile accesses.

							Thanx, Paul

> Best regards,
> Yanfei
> 
> > 
> >                                                          Thanx, Paul
> > 
> > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > index da6f5213fb74..c6d95a00715e 100644
> > > --- a/kernel/rcu/tree.c
> > > +++ b/kernel/rcu/tree.c
> > > @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
> > >          if (IS_ENABLED(CONFIG_PREEMPTION))
> > >                  return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
> > >          might_sleep();  /* Check for RCU read-side critical section. */
> > > -       preempt_disable();
> > >          /*
> > >           * If the rcu_state.n_online_cpus counter is equal to one,
> > >           * there is only one CPU, and that CPU sees all prior accesses
> > > @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
> > >           * Those memory barriers are provided by CPU-hotplug code.
> > >           */
> > >          ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
> > > -       preempt_enable();
> > >          return ret;
> > >   }
> > > 
> > > 
> > > 
> > > Best regards,
> > > Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Paul E. McKenney]

On Fri, Apr 16, 2021 at 01:01:17AM +0800, Xu, Yanfei wrote:
> 
> 
> On 4/16/21 12:18 AM, Xu, Yanfei wrote:
> > 
> > 
> > On 4/15/21 11:43 PM, Paul E. McKenney wrote:
> > > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > > 
> > > On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
> > > > Hi experts,
> > > > 
> > > > I am learning rcu mechanism and its codes. When looking at the
> > > > rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
> > > > operation in non-preemption code path. And it has been a long
> > > > time. I can't
> > > > understand why we need it? Is there some thing I missed? If not, can we
> > > > remove the unnecessary operation like blow?
> > > 
> > > Good point, you are right that preemption is disabled anyway in that
> > > block
> > > of code.  However, preempt_disable() and preempt_enable() also
> > > prevent the
> > > compiler from moving that READ_ONCE() around.  So my question to you is
> > > whether it is safe to remove those statements entirely or whether they
> > > should instead be replaced by barrier() or similar.
> > 
> > Thanks for your reply! :)
> > 
> > Yes, preempt_disable() and preempt_enable() defined in !preemption are
> > barrier(). barrier can prevent from reordering that READ_ONCE(), but
> > base on my current understanding, volatile in READ_ONCE can also tell
> > the compiler not to reorder it. So, I think it's safe?
> > 
> > Best regards,
> > Yanfei
> 
> Hi Paul,
> I objdump the function rcu_blocking_is_gp():
> 
> after dropping the barrier():
> ffffffff81107c50 <rcu_blocking_is_gp>:
> ffffffff81107c50:       e8 7b 2a f5 ff          callq  ffffffff8105a6d0
> <__fentry__>
> ffffffff81107c55:       8b 05 41 fe 7c 01       mov 0x17cfe41(%rip),%eax
> # ffffffff828d7a9c <rcu_state+0x221c>
> ffffffff81107c5b:       55                      push   %rbp
> ffffffff81107c5c:       48 89 e5                mov    %rsp,%rbp
> ffffffff81107c5f:       5d                      pop    %rbp
> ffffffff81107c60:       83 f8 01                cmp    $0x1,%eax
> ffffffff81107c63:       0f 9e c0                setle  %al
> ffffffff81107c66:       0f b6 c0                movzbl %al,%eax
> ffffffff81107c69:       c3                      retq
> ffffffff81107c6a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
> 
> the original codes:
> ffffffff81107ba0 <rcu_blocking_is_gp>:
> ffffffff81107ba0:       e8 2b 2b f5 ff          callq  ffffffff8105a6d0
> <__fentry__>
> ffffffff81107ba5:       55                      push   %rbp
> ffffffff81107ba6:       48 89 e5                mov    %rsp,%rbp
> ffffffff81107ba9:       8b 05 ed fe 7c 01       mov 0x17cfeed(%rip),%eax
> # ffffffff828d7a9c <rcu_state+0x221c>
> ffffffff81107baf:       83 f8 01                cmp    $0x1,%eax
> ffffffff81107bb2:       5d                      pop    %rbp
> ffffffff81107bb3:       0f 9e c0                setle  %al
> ffffffff81107bb6:       0f b6 c0                movzbl %al,%eax
> ffffffff81107bb9:       c3                      retq
> ffffffff81107bba:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
> 
> umm... It did been reordered by compiler after dropping the barrier(),
> however, I think the result will not be effected. Right?

Indeed, the compiler is free to reorder volatile accesses with
non-volatile accesses.

The result might not be affected by your compiler using your particular
settings for optimizations, but is that true for all optimization settings
for all compilers, both now and in the future?

							Thanx, Paul

> Best regards,
> Yanfei
> 
> > 
> > > 
> > >                                                          Thanx, Paul
> > > 
> > > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > > index da6f5213fb74..c6d95a00715e 100644
> > > > --- a/kernel/rcu/tree.c
> > > > +++ b/kernel/rcu/tree.c
> > > > @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
> > > >          if (IS_ENABLED(CONFIG_PREEMPTION))
> > > >                  return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
> > > >          might_sleep();  /* Check for RCU read-side critical section. */
> > > > -       preempt_disable();
> > > >          /*
> > > >           * If the rcu_state.n_online_cpus counter is equal to one,
> > > >           * there is only one CPU, and that CPU sees all prior accesses
> > > > @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
> > > >           * Those memory barriers are provided by CPU-hotplug code.
> > > >           */
> > > >          ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
> > > > -       preempt_enable();
> > > >          return ret;
> > > >   }
> > > > 
> > > > 
> > > > 
> > > > Best regards,
> > > > Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Xu, Yanfei]

On 4/16/21 1:07 AM, Paul E. McKenney wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
> 
> On Fri, Apr 16, 2021 at 12:18:42AM +0800, Xu, Yanfei wrote:
>>
>>
>> On 4/15/21 11:43 PM, Paul E. McKenney wrote:
>>> [Please note: This e-mail is from an EXTERNAL e-mail address]
>>>
>>> On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
>>>> Hi experts,
>>>>
>>>> I am learning rcu mechanism and its codes. When looking at the
>>>> rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
>>>> operation in non-preemption code path. And it has been a long time. I can't
>>>> understand why we need it? Is there some thing I missed? If not, can we
>>>> remove the unnecessary operation like blow?
>>>
>>> Good point, you are right that preemption is disabled anyway in that block
>>> of code.  However, preempt_disable() and preempt_enable() also prevent the
>>> compiler from moving that READ_ONCE() around.  So my question to you is
>>> whether it is safe to remove those statements entirely or whether they
>>> should instead be replaced by barrier() or similar.
>>
>> Thanks for your reply! :)
>>
>> Yes, preempt_disable() and preempt_enable() defined in !preemption are
>> barrier(). barrier can prevent from reordering that READ_ONCE(), but base on
>> my current understanding, volatile in READ_ONCE can also tell the compiler
>> not to reorder it. So, I think it's safe?
> 
> Maybe.
> 
> Please keep in mind that although the compiler is prohibited from
> reordering volatile accesses with each other, there is nothing stopping
> it from reordering volatile accesses with non-volatile accesses.

Thanks for your patient explanation!

I am trying to absorb what you said. Blow are my understanding:
1. "the compiler is prohibited from reordering volatile accesses with 
each other" means these situations:
int a;
foo()
{
     for(;;)
         READ_ONCE(a);
}

or

int a,b;
foo()
{
     int c,d;
     c = READ_ONCE(a);
     d = READ_ONCE(b);
}

2. "volatile accesses with non-volatile accesses" means d=b may happen 
before c=READ_ONCE(a) :
int a;
foo()
{
     int b = 2
     int c,d;
     c = READ_ONCE(a);
     d = b;
}
if we want to keep the ordering of volatile access "c=READ_ONCE(a)" and 
non-volatile access "d=b", we should use stronger barrier like barrier().

Hope I didn't misunderstand.

Back to rcu_blocking_is_gp(), I find this link today 
https://www.spinics.net/lists/rcu/msg03985.html
With the content in this link, I still haven't got the meaning of these 
two barrier(). I think I should learn knowledge about cpu-hotplug and 
things which talked in the link first to make sure if I am missing 
something, and then consult you. :)

Best regards,
Yanfei

> 
>                                                          Thanx, Paul
> 
>> Best regards,
>> Yanfei
>>
>>>
>>>                                                           Thanx, Paul
>>>
>>>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>>>> index da6f5213fb74..c6d95a00715e 100644
>>>> --- a/kernel/rcu/tree.c
>>>> +++ b/kernel/rcu/tree.c
>>>> @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
>>>>           if (IS_ENABLED(CONFIG_PREEMPTION))
>>>>                   return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
>>>>           might_sleep();  /* Check for RCU read-side critical section. */
>>>> -       preempt_disable();
>>>>           /*
>>>>            * If the rcu_state.n_online_cpus counter is equal to one,
>>>>            * there is only one CPU, and that CPU sees all prior accesses
>>>> @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
>>>>            * Those memory barriers are provided by CPU-hotplug code.
>>>>            */
>>>>           ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
>>>> -       preempt_enable();
>>>>           return ret;
>>>>    }
>>>>
>>>>
>>>>
>>>> Best regards,
>>>> Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Paul E. McKenney]

On Fri, Apr 16, 2021 at 06:51:10PM +0800, Xu, Yanfei wrote:
> 
> 
> On 4/16/21 1:07 AM, Paul E. McKenney wrote:
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > 
> > On Fri, Apr 16, 2021 at 12:18:42AM +0800, Xu, Yanfei wrote:
> > > 
> > > 
> > > On 4/15/21 11:43 PM, Paul E. McKenney wrote:
> > > > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > > > 
> > > > On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
> > > > > Hi experts,
> > > > > 
> > > > > I am learning rcu mechanism and its codes. When looking at the
> > > > > rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
> > > > > operation in non-preemption code path. And it has been a long time. I can't
> > > > > understand why we need it? Is there some thing I missed? If not, can we
> > > > > remove the unnecessary operation like blow?
> > > > 
> > > > Good point, you are right that preemption is disabled anyway in that block
> > > > of code.  However, preempt_disable() and preempt_enable() also prevent the
> > > > compiler from moving that READ_ONCE() around.  So my question to you is
> > > > whether it is safe to remove those statements entirely or whether they
> > > > should instead be replaced by barrier() or similar.
> > > 
> > > Thanks for your reply! :)
> > > 
> > > Yes, preempt_disable() and preempt_enable() defined in !preemption are
> > > barrier(). barrier can prevent from reordering that READ_ONCE(), but base on
> > > my current understanding, volatile in READ_ONCE can also tell the compiler
> > > not to reorder it. So, I think it's safe?
> > 
> > Maybe.
> > 
> > Please keep in mind that although the compiler is prohibited from
> > reordering volatile accesses with each other, there is nothing stopping
> > it from reordering volatile accesses with non-volatile accesses.
> 
> Thanks for your patient explanation!
> 
> I am trying to absorb what you said. Blow are my understanding:
> 1. "the compiler is prohibited from reordering volatile accesses with each
> other" means these situations:
> int a;
> foo()
> {
>     for(;;)
>         READ_ONCE(a);
> }
> 
> or
> 
> int a,b;
> foo()
> {
>     int c,d;
>     c = READ_ONCE(a);
>     d = READ_ONCE(b);
> }

Yes, in both cases the load instructions emitted for the READ_ONCE()
macros must be emitted in order.  The underlying hardware is free
to reorder.

> 2. "volatile accesses with non-volatile accesses" means d=b may happen
> before c=READ_ONCE(a) :
> int a;
> foo()
> {
>     int b = 2
>     int c,d;
>     c = READ_ONCE(a);
>     d = b;
> }
> if we want to keep the ordering of volatile access "c=READ_ONCE(a)" and
> non-volatile access "d=b", we should use stronger barrier like barrier().

Or an additional READ_ONCE() for b or a WRITE_ONCE() for d.  But again,
this would constrain only the compiler, not the hardware.

But this wouldn't matter in most cases, because both b and d are local
variables whose addresses were never taken.  So someone would need to
be using something crazy to poke into others' stacks for this to matter.

> Hope I didn't misunderstand.

It looks like you have most of it.

> Back to rcu_blocking_is_gp(), I find this link today
> https://www.spinics.net/lists/rcu/msg03985.html
> With the content in this link, I still haven't got the meaning of these two
> barrier(). I think I should learn knowledge about cpu-hotplug and things
> which talked in the link first to make sure if I am missing something, and
> then consult you. :)

That sounds like a very good approach!

Keep in mind that I am worried not just about the current state of
the code and compilers, but also their possible future states.

							Thanx, Paul

> Best regards,
> Yanfei
> 
> > 
> >                                                          Thanx, Paul
> > 
> > > Best regards,
> > > Yanfei
> > > 
> > > > 
> > > >                                                           Thanx, Paul
> > > > 
> > > > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > > > index da6f5213fb74..c6d95a00715e 100644
> > > > > --- a/kernel/rcu/tree.c
> > > > > +++ b/kernel/rcu/tree.c
> > > > > @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
> > > > >           if (IS_ENABLED(CONFIG_PREEMPTION))
> > > > >                   return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
> > > > >           might_sleep();  /* Check for RCU read-side critical section. */
> > > > > -       preempt_disable();
> > > > >           /*
> > > > >            * If the rcu_state.n_online_cpus counter is equal to one,
> > > > >            * there is only one CPU, and that CPU sees all prior accesses
> > > > > @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
> > > > >            * Those memory barriers are provided by CPU-hotplug code.
> > > > >            */
> > > > >           ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
> > > > > -       preempt_enable();
> > > > >           return ret;
> > > > >    }
> > > > > 
> > > > > 
> > > > > 
> > > > > Best regards,
> > > > > Yanfei

Re: [Qestion] Is preempt_disable/enable needed in non-preemption code path

[Xu, Yanfei]

On 4/17/21 1:26 AM, Paul E. McKenney wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
> 
> On Fri, Apr 16, 2021 at 06:51:10PM +0800, Xu, Yanfei wrote:
>>
>>
>> On 4/16/21 1:07 AM, Paul E. McKenney wrote:
>>> [Please note: This e-mail is from an EXTERNAL e-mail address]
>>>
>>> On Fri, Apr 16, 2021 at 12:18:42AM +0800, Xu, Yanfei wrote:
>>>>
>>>>
>>>> On 4/15/21 11:43 PM, Paul E. McKenney wrote:
>>>>> [Please note: This e-mail is from an EXTERNAL e-mail address]
>>>>>
>>>>> On Thu, Apr 15, 2021 at 11:04:05PM +0800, Xu, Yanfei wrote:
>>>>>> Hi experts,
>>>>>>
>>>>>> I am learning rcu mechanism and its codes. When looking at the
>>>>>> rcu_blocking_is_gp(), I found there is a pair preemption disable/enable
>>>>>> operation in non-preemption code path. And it has been a long time. I can't
>>>>>> understand why we need it? Is there some thing I missed? If not, can we
>>>>>> remove the unnecessary operation like blow?
>>>>>
>>>>> Good point, you are right that preemption is disabled anyway in that block
>>>>> of code.  However, preempt_disable() and preempt_enable() also prevent the
>>>>> compiler from moving that READ_ONCE() around.  So my question to you is
>>>>> whether it is safe to remove those statements entirely or whether they
>>>>> should instead be replaced by barrier() or similar.
>>>>
>>>> Thanks for your reply! :)
>>>>
>>>> Yes, preempt_disable() and preempt_enable() defined in !preemption are
>>>> barrier(). barrier can prevent from reordering that READ_ONCE(), but base on
>>>> my current understanding, volatile in READ_ONCE can also tell the compiler
>>>> not to reorder it. So, I think it's safe?
>>>
>>> Maybe.
>>>
>>> Please keep in mind that although the compiler is prohibited from
>>> reordering volatile accesses with each other, there is nothing stopping
>>> it from reordering volatile accesses with non-volatile accesses.
>>
>> Thanks for your patient explanation!
>>
>> I am trying to absorb what you said. Blow are my understanding:
>> 1. "the compiler is prohibited from reordering volatile accesses with each
>> other" means these situations:
>> int a;
>> foo()
>> {
>>      for(;;)
>>          READ_ONCE(a);
>> }
>>
>> or
>>
>> int a,b;
>> foo()
>> {
>>      int c,d;
>>      c = READ_ONCE(a);
>>      d = READ_ONCE(b);
>> }
> 
> Yes, in both cases the load instructions emitted for the READ_ONCE()
> macros must be emitted in order.  The underlying hardware is free
> to reorder.

Got it.
> 
>> 2. "volatile accesses with non-volatile accesses" means d=b may happen
>> before c=READ_ONCE(a) :
>> int a;
>> foo()
>> {
>>      int b = 2
>>      int c,d;
>>      c = READ_ONCE(a);
>>      d = b;
>> }
>> if we want to keep the ordering of volatile access "c=READ_ONCE(a)" and
>> non-volatile access "d=b", we should use stronger barrier like barrier().
> 
> Or an additional READ_ONCE() for b or a WRITE_ONCE() for d.  But again,
> this would constrain only the compiler, not the hardware.
> 
> But this wouldn't matter in most cases, because both b and d are local
> variables whose addresses were never taken.  So someone would need to
> be using something crazy to poke into others' stacks for this to matter.

Agree.
> 
>> Hope I didn't misunderstand.
> 
> It looks like you have most of it.
> 
>> Back to rcu_blocking_is_gp(), I find this link today
>> https://www.spinics.net/lists/rcu/msg03985.html
>> With the content in this link, I still haven't got the meaning of these two
>> barrier(). I think I should learn knowledge about cpu-hotplug and things
>> which talked in the link first to make sure if I am missing something, and
>> then consult you. :)
> 
> That sounds like a very good approach!
> 
> Keep in mind that I am worried not just about the current state of
> the code and compilers, but also their possible future states.

I see.

Thanks again.

Best regards,
Yanfei
> 
>                                                          Thanx, Paul
> 
>> Best regards,
>> Yanfei
>>
>>>
>>>                                                           Thanx, Paul
>>>
>>>> Best regards,
>>>> Yanfei
>>>>
>>>>>
>>>>>                                                            Thanx, Paul
>>>>>
>>>>>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>>>>>> index da6f5213fb74..c6d95a00715e 100644
>>>>>> --- a/kernel/rcu/tree.c
>>>>>> +++ b/kernel/rcu/tree.c
>>>>>> @@ -3703,7 +3703,6 @@ static int rcu_blocking_is_gp(void)
>>>>>>            if (IS_ENABLED(CONFIG_PREEMPTION))
>>>>>>                    return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
>>>>>>            might_sleep();  /* Check for RCU read-side critical section. */
>>>>>> -       preempt_disable();
>>>>>>            /*
>>>>>>             * If the rcu_state.n_online_cpus counter is equal to one,
>>>>>>             * there is only one CPU, and that CPU sees all prior accesses
>>>>>> @@ -3718,7 +3717,6 @@ static int rcu_blocking_is_gp(void)
>>>>>>             * Those memory barriers are provided by CPU-hotplug code.
>>>>>>             */
>>>>>>            ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
>>>>>> -       preempt_enable();
>>>>>>            return ret;
>>>>>>     }
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Yanfei