Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Qais Yousef]

+CC stable

On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> The following commit has been merged into the sched/core branch of tip:
> 
> Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> Author:        Valentin Schneider <valentin.schneider@arm.com>
> AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> Committer:     Peter Zijlstra <peterz@infradead.org>
> CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> 
> sched/tracing: Don't re-read p->state when emitting sched_switch event
> 
> As of commit
> 
>   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> 
> the following sequence becomes possible:
> 
> 		      p->__state = TASK_INTERRUPTIBLE;
> 		      __schedule()
> 			deactivate_task(p);
>   ttwu()
>     READ !p->on_rq
>     p->__state=TASK_WAKING
> 			trace_sched_switch()
> 			  __trace_sched_switch_state()
> 			    task_state_index()
> 			      return 0;
> 
> TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> the trace event.
> 
> Prevent this by pushing the value read from __schedule() down the trace
> event.
> 
> Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com

Any objection to picking this for stable? I'm interested in this one for some
Android users but prefer if it can be taken by stable rather than backport it
individually.

I think it makes sense to pick the next one in the series too.

Thanks!

--
Qais Yousef

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Greg KH]

On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> +CC stable
> 
> On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > The following commit has been merged into the sched/core branch of tip:
> > 
> > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > Committer:     Peter Zijlstra <peterz@infradead.org>
> > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > 
> > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > 
> > As of commit
> > 
> >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > 
> > the following sequence becomes possible:
> > 
> > 		      p->__state = TASK_INTERRUPTIBLE;
> > 		      __schedule()
> > 			deactivate_task(p);
> >   ttwu()
> >     READ !p->on_rq
> >     p->__state=TASK_WAKING
> > 			trace_sched_switch()
> > 			  __trace_sched_switch_state()
> > 			    task_state_index()
> > 			      return 0;
> > 
> > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > the trace event.
> > 
> > Prevent this by pushing the value read from __schedule() down the trace
> > event.
> > 
> > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> 
> Any objection to picking this for stable? I'm interested in this one for some
> Android users but prefer if it can be taken by stable rather than backport it
> individually.
> 
> I think it makes sense to pick the next one in the series too.

What commit does this fix in Linus's tree?

Once it hits Linus's tree, let stable@vger.kernel.org know what the git
commit id is in Linus's tree.

thanks,

greg k-h

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Qais Yousef]

On 03/08/22 19:10, Greg KH wrote:
> On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > +CC stable
> > 
> > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > The following commit has been merged into the sched/core branch of tip:
> > > 
> > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > 
> > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > 
> > > As of commit
> > > 
> > >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > 
> > > the following sequence becomes possible:
> > > 
> > > 		      p->__state = TASK_INTERRUPTIBLE;
> > > 		      __schedule()
> > > 			deactivate_task(p);
> > >   ttwu()
> > >     READ !p->on_rq
> > >     p->__state=TASK_WAKING
> > > 			trace_sched_switch()
> > > 			  __trace_sched_switch_state()
> > > 			    task_state_index()
> > > 			      return 0;
> > > 
> > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > the trace event.
> > > 
> > > Prevent this by pushing the value read from __schedule() down the trace
> > > event.
> > > 
> > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > 
> > Any objection to picking this for stable? I'm interested in this one for some
> > Android users but prefer if it can be taken by stable rather than backport it
> > individually.
> > 
> > I think it makes sense to pick the next one in the series too.
> 
> What commit does this fix in Linus's tree?

It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")

Not sure about the 2nd patch, but I can try to figure it out.

> Once it hits Linus's tree, let stable@vger.kernel.org know what the git
> commit id is in Linus's tree.

Sure. My bad if I rushed the request. I just wanted to know whether it's okay
for this to go into stable. If no one shouts, I'll ping once it lands in
Linus'.


Thanks!

--
Qais Yousef

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Qais Yousef]

On 03/08/22 18:51, Qais Yousef wrote:
> On 03/08/22 19:10, Greg KH wrote:
> > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > +CC stable
> > > 
> > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > The following commit has been merged into the sched/core branch of tip:
> > > > 
> > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > 
> > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > 
> > > > As of commit
> > > > 
> > > >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > 
> > > > the following sequence becomes possible:
> > > > 
> > > > 		      p->__state = TASK_INTERRUPTIBLE;
> > > > 		      __schedule()
> > > > 			deactivate_task(p);
> > > >   ttwu()
> > > >     READ !p->on_rq
> > > >     p->__state=TASK_WAKING
> > > > 			trace_sched_switch()
> > > > 			  __trace_sched_switch_state()
> > > > 			    task_state_index()
> > > > 			      return 0;
> > > > 
> > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > the trace event.
> > > > 
> > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > event.
> > > > 
> > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > 
> > > Any objection to picking this for stable? I'm interested in this one for some
> > > Android users but prefer if it can be taken by stable rather than backport it
> > > individually.
> > > 
> > > I think it makes sense to pick the next one in the series too.
> > 
> > What commit does this fix in Linus's tree?
> 
> It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")

Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
it up for v5.15+, but it impacts v5.10 too.

Thanks!

--
Qais Yousef

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Qais Yousef]

On 04/10/22 00:38, Qais Yousef wrote:
> On 03/08/22 18:51, Qais Yousef wrote:
> > On 03/08/22 19:10, Greg KH wrote:
> > > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > > +CC stable
> > > > 
> > > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > > The following commit has been merged into the sched/core branch of tip:
> > > > > 
> > > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > > 
> > > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > 
> > > > > As of commit
> > > > > 
> > > > >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > 
> > > > > the following sequence becomes possible:
> > > > > 
> > > > > 		      p->__state = TASK_INTERRUPTIBLE;
> > > > > 		      __schedule()
> > > > > 			deactivate_task(p);
> > > > >   ttwu()
> > > > >     READ !p->on_rq
> > > > >     p->__state=TASK_WAKING
> > > > > 			trace_sched_switch()
> > > > > 			  __trace_sched_switch_state()
> > > > > 			    task_state_index()
> > > > > 			      return 0;
> > > > > 
> > > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > > the trace event.
> > > > > 
> > > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > > event.
> > > > > 
> > > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > > 
> > > > Any objection to picking this for stable? I'm interested in this one for some
> > > > Android users but prefer if it can be taken by stable rather than backport it
> > > > individually.
> > > > 
> > > > I think it makes sense to pick the next one in the series too.
> > > 
> > > What commit does this fix in Linus's tree?
> > 
> > It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> 
> Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
> it up for v5.15+, but it impacts v5.10 too.

commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
subject: sched/tracing: Don't re-read p->state when emitting sched_switch event

This patch has an impact on Android 5.10 users who experience tooling breakage.
Is it possible to include in 5.10 LTS please?

It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.

Thanks

--
Qais Yousef

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Holger Hoffstätte]

On 2022-04-11 00:06, Qais Yousef wrote:
> On 04/10/22 00:38, Qais Yousef wrote:
>> On 03/08/22 18:51, Qais Yousef wrote:
>>> On 03/08/22 19:10, Greg KH wrote:
>>>> On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
>>>>> +CC stable
>>>>>
>>>>> On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
>>>>>> The following commit has been merged into the sched/core branch of tip:
>>>>>>
>>>>>> Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>> Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>> Author:        Valentin Schneider <valentin.schneider@arm.com>
>>>>>> AuthorDate:    Thu, 20 Jan 2022 16:25:19
>>>>>> Committer:     Peter Zijlstra <peterz@infradead.org>
>>>>>> CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
>>>>>>
>>>>>> sched/tracing: Don't re-read p->state when emitting sched_switch event
>>>>>>
>>>>>> As of commit
>>>>>>
>>>>>>    c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>>>>>
>>>>>> the following sequence becomes possible:
>>>>>>
>>>>>> 		      p->__state = TASK_INTERRUPTIBLE;
>>>>>> 		      __schedule()
>>>>>> 			deactivate_task(p);
>>>>>>    ttwu()
>>>>>>      READ !p->on_rq
>>>>>>      p->__state=TASK_WAKING
>>>>>> 			trace_sched_switch()
>>>>>> 			  __trace_sched_switch_state()
>>>>>> 			    task_state_index()
>>>>>> 			      return 0;
>>>>>>
>>>>>> TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
>>>>>> the trace event.
>>>>>>
>>>>>> Prevent this by pushing the value read from __schedule() down the trace
>>>>>> event.
>>>>>>
>>>>>> Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
>>>>>> Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
>>>>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>>>>> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
>>>>>> Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
>>>>>
>>>>> Any objection to picking this for stable? I'm interested in this one for some
>>>>> Android users but prefer if it can be taken by stable rather than backport it
>>>>> individually.
>>>>>
>>>>> I think it makes sense to pick the next one in the series too.
>>>>
>>>> What commit does this fix in Linus's tree?
>>>
>>> It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>
>> Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
>> it up for v5.15+, but it impacts v5.10 too.
> 
> commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> subject: sched/tracing: Don't re-read p->state when emitting sched_switch event
> 
> This patch has an impact on Android 5.10 users who experience tooling breakage.
> Is it possible to include in 5.10 LTS please?
> 
> It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.
> 

https://lore.kernel.org/stable/Yk2PQzynOVOzJdPo@kroah.com/

However, since then further investigation (still in progress) has shown that this
may have been the fault of the tool in question, so if you can verify that tracing
sched still works for you with this patch in 5.15.x then by all means
let's merge it.

-h

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Holger Hoffstätte]

On 2022-04-11 01:22, Holger Hoffstätte wrote:
> On 2022-04-11 00:06, Qais Yousef wrote:
>> On 04/10/22 00:38, Qais Yousef wrote:
>>> On 03/08/22 18:51, Qais Yousef wrote:
>>>> On 03/08/22 19:10, Greg KH wrote:
>>>>> On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
>>>>>> +CC stable
>>>>>>
>>>>>> On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
>>>>>>> The following commit has been merged into the sched/core branch of tip:
>>>>>>>
>>>>>>> Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>>> Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>>> Author:        Valentin Schneider <valentin.schneider@arm.com>
>>>>>>> AuthorDate:    Thu, 20 Jan 2022 16:25:19
>>>>>>> Committer:     Peter Zijlstra <peterz@infradead.org>
>>>>>>> CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
>>>>>>>
>>>>>>> sched/tracing: Don't re-read p->state when emitting sched_switch event
>>>>>>>
>>>>>>> As of commit
>>>>>>>
>>>>>>>    c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>>>>>>
>>>>>>> the following sequence becomes possible:
>>>>>>>
>>>>>>>               p->__state = TASK_INTERRUPTIBLE;
>>>>>>>               __schedule()
>>>>>>>             deactivate_task(p);
>>>>>>>    ttwu()
>>>>>>>      READ !p->on_rq
>>>>>>>      p->__state=TASK_WAKING
>>>>>>>             trace_sched_switch()
>>>>>>>               __trace_sched_switch_state()
>>>>>>>                 task_state_index()
>>>>>>>                   return 0;
>>>>>>>
>>>>>>> TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
>>>>>>> the trace event.
>>>>>>>
>>>>>>> Prevent this by pushing the value read from __schedule() down the trace
>>>>>>> event.
>>>>>>>
>>>>>>> Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
>>>>>>> Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
>>>>>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>>>>>> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
>>>>>>> Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
>>>>>>
>>>>>> Any objection to picking this for stable? I'm interested in this one for some
>>>>>> Android users but prefer if it can be taken by stable rather than backport it
>>>>>> individually.
>>>>>>
>>>>>> I think it makes sense to pick the next one in the series too.
>>>>>
>>>>> What commit does this fix in Linus's tree?
>>>>
>>>> It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>>
>>> Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
>>> it up for v5.15+, but it impacts v5.10 too.
>>
>> commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>> subject: sched/tracing: Don't re-read p->state when emitting sched_switch event
>>
>> This patch has an impact on Android 5.10 users who experience tooling breakage.
>> Is it possible to include in 5.10 LTS please?
>>
>> It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.
>>
> 
> https://lore.kernel.org/stable/Yk2PQzynOVOzJdPo@kroah.com/
> 
> However, since then further investigation (still in progress) has shown that this
> may have been the fault of the tool in question, so if you can verify that tracing
> sched still works for you with this patch in 5.15.x then by all means
> let's merge it.

So it turns out the lockup is indeed the fault of the tool, which contains multiple
kernel-version dependent tracepoint definitions and now fails with this
patch.

Greg, please re-enqueue this patch where necessary (5.10, 5.15+)
  
-h

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Greg KH]

On Mon, Apr 11, 2022 at 09:18:19AM +0200, Holger Hoffstätte wrote:
> On 2022-04-11 01:22, Holger Hoffstätte wrote:
> > On 2022-04-11 00:06, Qais Yousef wrote:
> > > On 04/10/22 00:38, Qais Yousef wrote:
> > > > On 03/08/22 18:51, Qais Yousef wrote:
> > > > > On 03/08/22 19:10, Greg KH wrote:
> > > > > > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > > > > > +CC stable
> > > > > > > 
> > > > > > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > > > > > The following commit has been merged into the sched/core branch of tip:
> > > > > > > > 
> > > > > > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > > > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19
> > > > > > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > > > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > > > > > 
> > > > > > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > > > > 
> > > > > > > > As of commit
> > > > > > > > 
> > > > > > > >    c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > > > > 
> > > > > > > > the following sequence becomes possible:
> > > > > > > > 
> > > > > > > >               p->__state = TASK_INTERRUPTIBLE;
> > > > > > > >               __schedule()
> > > > > > > >             deactivate_task(p);
> > > > > > > >    ttwu()
> > > > > > > >      READ !p->on_rq
> > > > > > > >      p->__state=TASK_WAKING
> > > > > > > >             trace_sched_switch()
> > > > > > > >               __trace_sched_switch_state()
> > > > > > > >                 task_state_index()
> > > > > > > >                   return 0;
> > > > > > > > 
> > > > > > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > > > > > the trace event.
> > > > > > > > 
> > > > > > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > > > > > event.
> > > > > > > > 
> > > > > > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > > > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > > > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > > > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > > > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > > > > > 
> > > > > > > Any objection to picking this for stable? I'm interested in this one for some
> > > > > > > Android users but prefer if it can be taken by stable rather than backport it
> > > > > > > individually.
> > > > > > > 
> > > > > > > I think it makes sense to pick the next one in the series too.
> > > > > > 
> > > > > > What commit does this fix in Linus's tree?
> > > > > 
> > > > > It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > 
> > > > Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
> > > > it up for v5.15+, but it impacts v5.10 too.
> > > 
> > > commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > subject: sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > 
> > > This patch has an impact on Android 5.10 users who experience tooling breakage.
> > > Is it possible to include in 5.10 LTS please?
> > > 
> > > It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.
> > > 
> > 
> > https://lore.kernel.org/stable/Yk2PQzynOVOzJdPo@kroah.com/
> > 
> > However, since then further investigation (still in progress) has shown that this
> > may have been the fault of the tool in question, so if you can verify that tracing
> > sched still works for you with this patch in 5.15.x then by all means
> > let's merge it.
> 
> So it turns out the lockup is indeed the fault of the tool, which contains multiple
> kernel-version dependent tracepoint definitions and now fails with this
> patch.

What tools is this?

> Greg, please re-enqueue this patch where necessary (5.10, 5.15+)

If I queue it up again, will the tools keep breaking?

thanks,

greg k-hj

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Holger Hoffstätte]

On 2022-04-11 09:28, Greg KH wrote:
> On Mon, Apr 11, 2022 at 09:18:19AM +0200, Holger Hoffstätte wrote:
>> On 2022-04-11 01:22, Holger Hoffstätte wrote:
>>> On 2022-04-11 00:06, Qais Yousef wrote:
>>>> On 04/10/22 00:38, Qais Yousef wrote:
>>>>> On 03/08/22 18:51, Qais Yousef wrote:
>>>>>> On 03/08/22 19:10, Greg KH wrote:
>>>>>>> On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
>>>>>>>> +CC stable
>>>>>>>>
>>>>>>>> On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
>>>>>>>>> The following commit has been merged into the sched/core branch of tip:
>>>>>>>>>
>>>>>>>>> Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>>>>> Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>>>>>>> Author:        Valentin Schneider <valentin.schneider@arm.com>
>>>>>>>>> AuthorDate:    Thu, 20 Jan 2022 16:25:19
>>>>>>>>> Committer:     Peter Zijlstra <peterz@infradead.org>
>>>>>>>>> CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
>>>>>>>>>
>>>>>>>>> sched/tracing: Don't re-read p->state when emitting sched_switch event
>>>>>>>>>
>>>>>>>>> As of commit
>>>>>>>>>
>>>>>>>>>     c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>>>>>>>>
>>>>>>>>> the following sequence becomes possible:
>>>>>>>>>
>>>>>>>>>                p->__state = TASK_INTERRUPTIBLE;
>>>>>>>>>                __schedule()
>>>>>>>>>              deactivate_task(p);
>>>>>>>>>     ttwu()
>>>>>>>>>       READ !p->on_rq
>>>>>>>>>       p->__state=TASK_WAKING
>>>>>>>>>              trace_sched_switch()
>>>>>>>>>                __trace_sched_switch_state()
>>>>>>>>>                  task_state_index()
>>>>>>>>>                    return 0;
>>>>>>>>>
>>>>>>>>> TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
>>>>>>>>> the trace event.
>>>>>>>>>
>>>>>>>>> Prevent this by pushing the value read from __schedule() down the trace
>>>>>>>>> event.
>>>>>>>>>
>>>>>>>>> Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
>>>>>>>>> Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
>>>>>>>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>>>>>>>> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
>>>>>>>>> Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
>>>>>>>>
>>>>>>>> Any objection to picking this for stable? I'm interested in this one for some
>>>>>>>> Android users but prefer if it can be taken by stable rather than backport it
>>>>>>>> individually.
>>>>>>>>
>>>>>>>> I think it makes sense to pick the next one in the series too.
>>>>>>>
>>>>>>> What commit does this fix in Linus's tree?
>>>>>>
>>>>>> It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
>>>>>
>>>>> Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
>>>>> it up for v5.15+, but it impacts v5.10 too.
>>>>
>>>> commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
>>>> subject: sched/tracing: Don't re-read p->state when emitting sched_switch event
>>>>
>>>> This patch has an impact on Android 5.10 users who experience tooling breakage.
>>>> Is it possible to include in 5.10 LTS please?
>>>>
>>>> It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.
>>>>
>>>
>>> https://lore.kernel.org/stable/Yk2PQzynOVOzJdPo@kroah.com/
>>>
>>> However, since then further investigation (still in progress) has shown that this
>>> may have been the fault of the tool in question, so if you can verify that tracing
>>> sched still works for you with this patch in 5.15.x then by all means
>>> let's merge it.
>>
>> So it turns out the lockup is indeed the fault of the tool, which contains multiple
>> kernel-version dependent tracepoint definitions and now fails with this
>> patch.
> 
> What tools is this?

sysdig - which uses a helper kernel module which accesses tracepoints, but of course
(as I just found) with copypasta'd TP definitions, which broke with this patch due to
the additional parameter in the function signature. It's been prone to breakage forever
because of a lack of a stable kernel ABI.

Took me a while to find/figure out, but IMHO better safe than sorry. We've had
autoselected scheduler patches before that looked fine but really were not.

> 
>> Greg, please re-enqueue this patch where necessary (5.10, 5.15+)
> 
> If I queue it up again, will the tools keep breaking?

Yes, but that's their problem with an out-of-tree module; a few more #ifdefs
are not going to make a big difference.

thanks
Holger

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Greg KH]

On Sun, Apr 10, 2022 at 12:38:29AM +0100, Qais Yousef wrote:
> On 03/08/22 18:51, Qais Yousef wrote:
> > On 03/08/22 19:10, Greg KH wrote:
> > > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > > +CC stable
> > > > 
> > > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > > The following commit has been merged into the sched/core branch of tip:
> > > > > 
> > > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > > 
> > > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > 
> > > > > As of commit
> > > > > 
> > > > >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > 
> > > > > the following sequence becomes possible:
> > > > > 
> > > > > 		      p->__state = TASK_INTERRUPTIBLE;
> > > > > 		      __schedule()
> > > > > 			deactivate_task(p);
> > > > >   ttwu()
> > > > >     READ !p->on_rq
> > > > >     p->__state=TASK_WAKING
> > > > > 			trace_sched_switch()
> > > > > 			  __trace_sched_switch_state()
> > > > > 			    task_state_index()
> > > > > 			      return 0;
> > > > > 
> > > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > > the trace event.
> > > > > 
> > > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > > event.
> > > > > 
> > > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > > 
> > > > Any objection to picking this for stable? I'm interested in this one for some
> > > > Android users but prefer if it can be taken by stable rather than backport it
> > > > individually.
> > > > 
> > > > I think it makes sense to pick the next one in the series too.
> > > 
> > > What commit does this fix in Linus's tree?
> > 
> > It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> 
> Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
> it up for v5.15+, but it impacts v5.10 too.

It does not apply to 5.10 at all, how did you test this?

{sigh}

Again, if you want this applied to any stable trees, please test that it
works and send the properly backported patches.

thanks,

greg k-h

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Greg KH]

On Mon, Apr 11, 2022 at 10:05:19AM +0200, Holger Hoffstätte wrote:
> On 2022-04-11 09:28, Greg KH wrote:
> > On Mon, Apr 11, 2022 at 09:18:19AM +0200, Holger Hoffstätte wrote:
> > > On 2022-04-11 01:22, Holger Hoffstätte wrote:
> > > > On 2022-04-11 00:06, Qais Yousef wrote:
> > > > > On 04/10/22 00:38, Qais Yousef wrote:
> > > > > > On 03/08/22 18:51, Qais Yousef wrote:
> > > > > > > On 03/08/22 19:10, Greg KH wrote:
> > > > > > > > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > > > > > > > +CC stable
> > > > > > > > > 
> > > > > > > > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > > > > > > > The following commit has been merged into the sched/core branch of tip:
> > > > > > > > > > 
> > > > > > > > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > > > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > > > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > > > > > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19
> > > > > > > > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > > > > > > > 
> > > > > > > > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > > > > > > 
> > > > > > > > > > As of commit
> > > > > > > > > > 
> > > > > > > > > >     c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > > > > > > 
> > > > > > > > > > the following sequence becomes possible:
> > > > > > > > > > 
> > > > > > > > > >                p->__state = TASK_INTERRUPTIBLE;
> > > > > > > > > >                __schedule()
> > > > > > > > > >              deactivate_task(p);
> > > > > > > > > >     ttwu()
> > > > > > > > > >       READ !p->on_rq
> > > > > > > > > >       p->__state=TASK_WAKING
> > > > > > > > > >              trace_sched_switch()
> > > > > > > > > >                __trace_sched_switch_state()
> > > > > > > > > >                  task_state_index()
> > > > > > > > > >                    return 0;
> > > > > > > > > > 
> > > > > > > > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > > > > > > > the trace event.
> > > > > > > > > > 
> > > > > > > > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > > > > > > > event.
> > > > > > > > > > 
> > > > > > > > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > > > > > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > > > > > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > > > > > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > > > > > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > > > > > > > 
> > > > > > > > > Any objection to picking this for stable? I'm interested in this one for some
> > > > > > > > > Android users but prefer if it can be taken by stable rather than backport it
> > > > > > > > > individually.
> > > > > > > > > 
> > > > > > > > > I think it makes sense to pick the next one in the series too.
> > > > > > > > 
> > > > > > > > What commit does this fix in Linus's tree?
> > > > > > > 
> > > > > > > It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > > 
> > > > > > Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
> > > > > > it up for v5.15+, but it impacts v5.10 too.
> > > > > 
> > > > > commit: fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > subject: sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > 
> > > > > This patch has an impact on Android 5.10 users who experience tooling breakage.
> > > > > Is it possible to include in 5.10 LTS please?
> > > > > 
> > > > > It was already picked up for 5.15+ by AUTOSEL and only 5.10 is missing.
> > > > > 
> > > > 
> > > > https://lore.kernel.org/stable/Yk2PQzynOVOzJdPo@kroah.com/
> > > > 
> > > > However, since then further investigation (still in progress) has shown that this
> > > > may have been the fault of the tool in question, so if you can verify that tracing
> > > > sched still works for you with this patch in 5.15.x then by all means
> > > > let's merge it.
> > > 
> > > So it turns out the lockup is indeed the fault of the tool, which contains multiple
> > > kernel-version dependent tracepoint definitions and now fails with this
> > > patch.
> > 
> > What tools is this?
> 
> sysdig - which uses a helper kernel module which accesses tracepoints, but of course
> (as I just found) with copypasta'd TP definitions, which broke with this patch due to
> the additional parameter in the function signature. It's been prone to breakage forever
> because of a lack of a stable kernel ABI.
> 
> Took me a while to find/figure out, but IMHO better safe than sorry. We've had
> autoselected scheduler patches before that looked fine but really were not.

Thanks for the info, sysdig does do some pretty intrusive things but I
thought it had moved over to being almost all bpf code.

Anyway, I'll wait for Qais to submit a properly backported and tested
set of patches before picking this up again due to all of the problems
I've had with this patch.

thanks,

greg k-h

Re: [tip: sched/core] sched/tracing: Don't re-read p->state when emitting sched_switch event

[Qais Yousef]

On 04/11/22 15:22, Greg KH wrote:
> On Sun, Apr 10, 2022 at 12:38:29AM +0100, Qais Yousef wrote:
> > On 03/08/22 18:51, Qais Yousef wrote:
> > > On 03/08/22 19:10, Greg KH wrote:
> > > > On Tue, Mar 08, 2022 at 06:02:40PM +0000, Qais Yousef wrote:
> > > > > +CC stable
> > > > > 
> > > > > On 03/01/22 15:24, tip-bot2 for Valentin Schneider wrote:
> > > > > > The following commit has been merged into the sched/core branch of tip:
> > > > > > 
> > > > > > Commit-ID:     fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > Gitweb:        https://git.kernel.org/tip/fa2c3254d7cfff5f7a916ab928a562d1165f17bb
> > > > > > Author:        Valentin Schneider <valentin.schneider@arm.com>
> > > > > > AuthorDate:    Thu, 20 Jan 2022 16:25:19 
> > > > > > Committer:     Peter Zijlstra <peterz@infradead.org>
> > > > > > CommitterDate: Tue, 01 Mar 2022 16:18:39 +01:00
> > > > > > 
> > > > > > sched/tracing: Don't re-read p->state when emitting sched_switch event
> > > > > > 
> > > > > > As of commit
> > > > > > 
> > > > > >   c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > > > > > 
> > > > > > the following sequence becomes possible:
> > > > > > 
> > > > > > 		      p->__state = TASK_INTERRUPTIBLE;
> > > > > > 		      __schedule()
> > > > > > 			deactivate_task(p);
> > > > > >   ttwu()
> > > > > >     READ !p->on_rq
> > > > > >     p->__state=TASK_WAKING
> > > > > > 			trace_sched_switch()
> > > > > > 			  __trace_sched_switch_state()
> > > > > > 			    task_state_index()
> > > > > > 			      return 0;
> > > > > > 
> > > > > > TASK_WAKING isn't in TASK_REPORT, so the task appears as TASK_RUNNING in
> > > > > > the trace event.
> > > > > > 
> > > > > > Prevent this by pushing the value read from __schedule() down the trace
> > > > > > event.
> > > > > > 
> > > > > > Reported-by: Abhijeet Dharmapurikar <adharmap@quicinc.com>
> > > > > > Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
> > > > > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > > > > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> > > > > > Link: https://lore.kernel.org/r/20220120162520.570782-2-valentin.schneider@arm.com
> > > > > 
> > > > > Any objection to picking this for stable? I'm interested in this one for some
> > > > > Android users but prefer if it can be taken by stable rather than backport it
> > > > > individually.
> > > > > 
> > > > > I think it makes sense to pick the next one in the series too.
> > > > 
> > > > What commit does this fix in Linus's tree?
> > > 
> > > It should be this one: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
> > 
> > Should this be okay to be picked up by stable now? I can see AUTOSEL has picked
> > it up for v5.15+, but it impacts v5.10 too.
> 
> It does not apply to 5.10 at all, how did you test this?
> 
> {sigh}
> 
> Again, if you want this applied to any stable trees, please test that it
> works and send the properly backported patches.

Sorry. I'm picking up pieces after a colleague left and was under the
impression that 5.10 Android users already use as-is, but due to GKI just need
it formally in stable. Few things got lost in translation, I am very sorry!

Let me sort all of this out properly including the testing part for 5.15 too.

Thanks!

--
Qais Yousef