[ANNOUNCE] 4.19.106-rt44

[ANNOUNCE] 4.19.106-rt44

[Steven Rostedt]

Dear RT Folks,

I'm pleased to announce the 4.19.106-rt44 stable release.


This release is just an update to the new stable 4.19.106 version
and no RT specific changes have been made.


You can get this release via the git tree at:

  git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git

  branch: v4.19-rt
  Head SHA1: 0f2960c75dd68d339f0aff2935f51652b5625fbf


Or to build 4.19.106-rt44 directly, the following patches should be applied:

  http://www.kernel.org/pub/linux/kernel/v4.x/linux-4.19.tar.xz

  http://www.kernel.org/pub/linux/kernel/v4.x/patch-4.19.106.xz

  http://www.kernel.org/pub/linux/kernel/projects/rt/4.19/patch-4.19.106-rt44.patch.xz




Enjoy,

-- Steve

Re: [ANNOUNCE] 4.19.106-rt44

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

Hi!

> I'm pleased to announce the 4.19.106-rt44 stable release.
> 
> 
> This release is just an update to the new stable 4.19.106 version
> and no RT specific changes have been made.
> 
> 
> You can get this release via the git tree at:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git
> 
>   branch: v4.19-rt
>   Head SHA1: 0f2960c75dd68d339f0aff2935f51652b5625fbf

This brought some problems for me. de0-nano board now fails to boot in
cca 50% of cases if I move these patches on top of -cip tree.

This is example of failed job:

https://lava.ciplatform.org/scheduler/job/13037

de0-nano is 32-bit arm, should be based on Altera SoCFPGA if I understand
things correctly.

"fc9f4631a290 irqwork: push most work into softirq context" touches
area of the panic above. I tried to revert it on top of the full
series, and tests passed twice so far...

Best regards,
									
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]


On Thu 2020-03-19 22:48:35, Pavel Machek wrote:
> Hi!
> 
> > I'm pleased to announce the 4.19.106-rt44 stable release.
> > 
> > 
> > This release is just an update to the new stable 4.19.106 version
> > and no RT specific changes have been made.
> > 
> > 
> > You can get this release via the git tree at:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git
> > 
> >   branch: v4.19-rt
> >   Head SHA1: 0f2960c75dd68d339f0aff2935f51652b5625fbf
> 
> This brought some problems for me. de0-nano board now fails to boot in
> cca 50% of cases if I move these patches on top of -cip tree.
> 
> This is example of failed job:
> 
> https://lava.ciplatform.org/scheduler/job/13037
> 
> de0-nano is 32-bit arm, should be based on Altera SoCFPGA if I understand
> things correctly.
> 
> "fc9f4631a290 irqwork: push most work into softirq context" touches
> area of the panic above. I tried to revert it on top of the full
> series, and tests passed twice so far...

Test passed 7 times now. So yes, reverting this fixes de0-nano
boot. Any ideas what might be wrong?

I'll be running it few more times.

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/127953471

Best regards,
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Steven Rostedt]

On Fri, 20 Mar 2020 00:22:25 +0100
Pavel Machek <pavel@denx.de> wrote:

> On Thu 2020-03-19 22:48:35, Pavel Machek wrote:
> > Hi!

Hi Pavel!

> >   
> > > I'm pleased to announce the 4.19.106-rt44 stable release.
> > > 
> > > 
> > > This release is just an update to the new stable 4.19.106 version
> > > and no RT specific changes have been made.
> > > 
> > > 
> > > You can get this release via the git tree at:
> > > 
> > >   git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git
> > > 
> > >   branch: v4.19-rt
> > >   Head SHA1: 0f2960c75dd68d339f0aff2935f51652b5625fbf  
> > 
> > This brought some problems for me. de0-nano board now fails to boot in
> > cca 50% of cases if I move these patches on top of -cip tree.
> > 
> > This is example of failed job:
> > 
> > https://lava.ciplatform.org/scheduler/job/13037
> > 
> > de0-nano is 32-bit arm, should be based on Altera SoCFPGA if I understand
> > things correctly.
> > 
> > "fc9f4631a290 irqwork: push most work into softirq context" touches
> > area of the panic above. I tried to revert it on top of the full
> > series, and tests passed twice so far...  
> 
> Test passed 7 times now. So yes, reverting this fixes de0-nano
> boot. Any ideas what might be wrong?
> 
> I'll be running it few more times.
> 
> https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/127953471
> 

Looks like you are running this without PREEMPT_RT enabled.

Does this patch help?

-- Steve


diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 2940622da5b3..0ca75c77536b 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -146,8 +146,9 @@ bool irq_work_needs_cpu(void)
 	raised = this_cpu_ptr(&raised_list);
 	lazy = this_cpu_ptr(&lazy_list);
 
-	if (llist_empty(raised) && llist_empty(lazy))
-		return false;
+	if (llist_empty(raised) || arch_irq_work_has_interrupt())
+		if (llist_empty(lazy))
+			return false;
 
 	/* All work should have been flushed before going offline */
 	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

Hi!

> > > > I'm pleased to announce the 4.19.106-rt44 stable release.
> > > > 
> > > > 
> > > > This release is just an update to the new stable 4.19.106 version
> > > > and no RT specific changes have been made.
> > > > 
> > > > 
> > > > You can get this release via the git tree at:
> > > > 
> > > >   git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git
> > > > 
> > > >   branch: v4.19-rt
> > > >   Head SHA1: 0f2960c75dd68d339f0aff2935f51652b5625fbf  
> > > 
> > > This brought some problems for me. de0-nano board now fails to boot in
> > > cca 50% of cases if I move these patches on top of -cip tree.
> > > 
> > > This is example of failed job:
> > > 
> > > https://lava.ciplatform.org/scheduler/job/13037
> > > 
> > > de0-nano is 32-bit arm, should be based on Altera SoCFPGA if I understand
> > > things correctly.
> > > 
> > > "fc9f4631a290 irqwork: push most work into softirq context" touches
> > > area of the panic above. I tried to revert it on top of the full
> > > series, and tests passed twice so far...  
> > 
> > Test passed 7 times now. So yes, reverting this fixes de0-nano
> > boot. Any ideas what might be wrong?
> > 
> > I'll be running it few more times.
> > 
> > https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/127953471
> > 
> 
> Looks like you are running this without PREEMPT_RT enabled.

Yes. We still need to set up proper realtime testing.

> Does this patch help?

I don't think so. It also failed, and the failure seems to be
identical to me.

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
https://lava.ciplatform.org/scheduler/job/13110

Best regards,
								Pavel

> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index 2940622da5b3..0ca75c77536b 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -146,8 +146,9 @@ bool irq_work_needs_cpu(void)
>  	raised = this_cpu_ptr(&raised_list);
>  	lazy = this_cpu_ptr(&lazy_list);
>  
> -	if (llist_empty(raised) && llist_empty(lazy))
> -		return false;
> +	if (llist_empty(raised) || arch_irq_work_has_interrupt())
> +		if (llist_empty(lazy))
> +			return false;
>  
>  	/* All work should have been flushed before going offline */
>  	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Steven Rostedt]

On Fri, 20 Mar 2020 20:54:32 +0100
Pavel Machek <pavel@denx.de> wrote:

> > Does this patch help?  
> 
> I don't think so. It also failed, and the failure seems to be
> identical to me.
> 
> https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
> https://lava.ciplatform.org/scheduler/job/13110
> 

Can you send me a patch that shows the difference between the revert that
you say works, and the upstream v4.19-rt tree (let me know which version
of v4.19-rt you are basing it on).

Thanks!

-- Steve

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

Hi!

On Fri 2020-03-20 16:05:45, Steven Rostedt wrote:
> On Fri, 20 Mar 2020 20:54:32 +0100
> Pavel Machek <pavel@denx.de> wrote:
> 
> > > Does this patch help?  
> > 
> > I don't think so. It also failed, and the failure seems to be
> > identical to me.
> > 
> > https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
> > https://lava.ciplatform.org/scheduler/job/13110
> > 
> 
> Can you send me a patch that shows the difference between the revert that
> you say works, and the upstream v4.19-rt tree (let me know which version
> of v4.19-rt you are basing it on).

I was using -rt44, and yes, I can probably generate better diffs.

But I guess I found it with code review: how does this look to you? I
applied it on top of your fix, and am testing. 2 successes so far.

Signed-off-by: Pavel Machek <pavel@denx.de>

Best regards,
								Pavel

commit aa034c3060dbab96a7b6d6bf827504b394bed15b
Author: Pavel Machek <pavel@ucw.cz>
Date:   Sat Mar 21 22:58:43 2020 +0100

    It sounds like irq_work_queue() was queueing on wrong list?

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 0ca75c77536b..dd654865c219 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -81,7 +81,8 @@ bool irq_work_queue(struct irq_work *work)
 
 	/* Queue the entry and raise the IPI if needed. */
 	preempt_disable();
-	if (IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	if ((IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	    || (work->flags & IRQ_WORK_LAZY))
 		list = this_cpu_ptr(&lazy_list);
 	else
 		list = this_cpu_ptr(&raised_list);



      	    	       	      	  	   	    	   Pavel
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 3190 bytes --]

Hi!

> > > > Does this patch help?  
> > > 
> > > I don't think so. It also failed, and the failure seems to be
> > > identical to me.
> > > 
> > > https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
> > > https://lava.ciplatform.org/scheduler/job/13110
> > > 
> > 
> > Can you send me a patch that shows the difference between the revert that
> > you say works, and the upstream v4.19-rt tree (let me know which version
> > of v4.19-rt you are basing it on).
> 
> I was using -rt44, and yes, I can probably generate better diffs.
> 
> But I guess I found it with code review: how does this look to you? I
> applied it on top of your fix, and am testing. 2 successes so far.

And I'd recommend some kind of cleanup on top. The code is really
"interesting" and we don't want to have two copies. Totally untested.

Looking at the code, it could be probably cleaned up further.

Signed-off-by: Pavel Machek <pavel@denx.de>

Best regards,
								Pavel

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index dd654865c219..88211b87d4e3 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -70,10 +70,19 @@ static void __irq_work_queue_local(struct irq_work *work, struct llist_head *lis
 		arch_irq_work_raise();
 }
 
+static struct llist_head *irq_work_get_list(struct irq_work *work)
+{
+	if ((IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	    || (work->flags & IRQ_WORK_LAZY))
+		return &lazy_list;
+	else
+		return &raised_list;
+}
+
 /* Enqueue the irq work @work on the current CPU */
 bool irq_work_queue(struct irq_work *work)
 {
-	struct llist_head *list;
+    	struct llist_head *list;
 
 	/* Only queue if not already pending */
 	if (!irq_work_claim(work))
@@ -81,12 +90,7 @@ bool irq_work_queue(struct irq_work *work)
 
 	/* Queue the entry and raise the IPI if needed. */
 	preempt_disable();
-	if ((IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
-	    || (work->flags & IRQ_WORK_LAZY))
-		list = this_cpu_ptr(&lazy_list);
-	else
-		list = this_cpu_ptr(&raised_list);
-
+	list = this_cpu_ptr(irq_work_get_list(work));
 	__irq_work_queue_local(work, list);
 	preempt_enable();
 
@@ -107,7 +111,6 @@ bool irq_work_queue_on(struct irq_work *work, int cpu)
 
 #else /* CONFIG_SMP: */
 	struct llist_head *list;
-	bool lazy_work, realtime = IS_ENABLED(CONFIG_PREEMPT_RT_FULL);
 
 	/* All work should have been flushed before going offline */
 	WARN_ON_ONCE(cpu_is_offline(cpu));
@@ -117,13 +120,8 @@ bool irq_work_queue_on(struct irq_work *work, int cpu)
 		return false;
 
 	preempt_disable();
-
-	lazy_work = work->flags & IRQ_WORK_LAZY;
-
-	if (lazy_work || (realtime && !(work->flags & IRQ_WORK_HARD_IRQ)))
-		list = &per_cpu(lazy_list, cpu);
-	else
-		list = &per_cpu(raised_list, cpu);
+	list = irq_work_get_list(work);
+	list = &per_cpu(list, cpu);
 
 	if (cpu != smp_processor_id()) {
 		/* Arch remote IPI send/receive backend aren't NMI safe */

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

4.4-rt ... seems to have same ireqwork problem was Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2002 bytes --]

Hi!

> > > > > Does this patch help?  
> > > > 
> > > > I don't think so. It also failed, and the failure seems to be
> > > > identical to me.
> > > > 
> > > > https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
> > > > https://lava.ciplatform.org/scheduler/job/13110
> > > > 
> > > 
> > > Can you send me a patch that shows the difference between the revert that
> > > you say works, and the upstream v4.19-rt tree (let me know which version
> > > of v4.19-rt you are basing it on).
> > 
> > I was using -rt44, and yes, I can probably generate better diffs.
> > 
> > But I guess I found it with code review: how does this look to you? I
> > applied it on top of your fix, and am testing. 2 successes so far.
> 
> And I'd recommend some kind of cleanup on top. The code is really
> "interesting" and we don't want to have two copies. Totally untested.
> 
> Looking at the code, it could be probably cleaned up further.

It seems 4.4 branch has same problem. Unfortunately, our testing lab
does not help in this case, so .. this is completely untested. Problem
was found by code inspection.

Best regards,
								Pavel

Signed-off-by: Pavel Machek <pavel@denx.de>
Fixes: fc9f4631a290 ("irqwork: push most work into softirq context")

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 2899ba0d23d1..19896e6f1b2a 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -78,7 +78,8 @@ bool irq_work_queue_on(struct irq_work *work, int cpu)
 	if (!irq_work_claim(work))
 		return false;
 
-	if (IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	if ((IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	    || (work->flags & IRQ_WORK_LAZY))
 		list = &per_cpu(lazy_list, cpu);
 	else
 		list = &per_cpu(raised_list, cpu);



-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: 4.19.106-rt44 -- boot problems with irqwork: push most work into softirq context

[Steven Rostedt]

On Sun, 22 Mar 2020 00:00:28 +0100
Pavel Machek <pavel@denx.de> wrote:

> Hi!
> 
> > > > > Does this patch help?    
> > > > 
> > > > I don't think so. It also failed, and the failure seems to be
> > > > identical to me.
> > > > 
> > > > https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
> > > > https://lava.ciplatform.org/scheduler/job/13110
> > > >   
> > > 
> > > Can you send me a patch that shows the difference between the revert that
> > > you say works, and the upstream v4.19-rt tree (let me know which version
> > > of v4.19-rt you are basing it on).  
> > 
> > I was using -rt44, and yes, I can probably generate better diffs.
> > 
> > But I guess I found it with code review: how does this look to you? I
> > applied it on top of your fix, and am testing. 2 successes so far.  
> 
> And I'd recommend some kind of cleanup on top. The code is really
> "interesting" and we don't want to have two copies. Totally untested.
> 
> Looking at the code, it could be probably cleaned up further.
> 
> Signed-off-by: Pavel Machek <pavel@denx.de>
> 
> Best regards,
> 								Pavel

I applied this patch, does this work for you. It's slightly different than
yours as I thought only the condition needed to be saved, not the lists
themselves.

-- Steve

Index: stable-rt.git/kernel/irq_work.c
===================================================================
--- stable-rt.git.orig/kernel/irq_work.c	2020-03-30 15:11:13.849875145 -0400
+++ stable-rt.git/kernel/irq_work.c	2020-03-30 15:18:54.365242025 -0400
@@ -70,6 +70,12 @@ static void __irq_work_queue_local(struc
 		arch_irq_work_raise();
 }
 
+static inline bool use_lazy_list(struct irq_work *work)
+{
+	return (IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+		|| (work->flags & IRQ_WORK_LAZY);
+}
+
 /* Enqueue the irq work @work on the current CPU */
 bool irq_work_queue(struct irq_work *work)
 {
@@ -81,11 +87,10 @@ bool irq_work_queue(struct irq_work *wor
 
 	/* Queue the entry and raise the IPI if needed. */
 	preempt_disable();
-	if (IS_ENABLED(CONFIG_PREEMPT_RT_FULL) && !(work->flags & IRQ_WORK_HARD_IRQ))
+	if (use_lazy_list(work))
 		list = this_cpu_ptr(&lazy_list);
 	else
 		list = this_cpu_ptr(&raised_list);
-
 	__irq_work_queue_local(work, list);
 	preempt_enable();
 
@@ -106,7 +111,6 @@ bool irq_work_queue_on(struct irq_work *
 
 #else /* CONFIG_SMP: */
 	struct llist_head *list;
-	bool lazy_work, realtime = IS_ENABLED(CONFIG_PREEMPT_RT_FULL);
 
 	/* All work should have been flushed before going offline */
 	WARN_ON_ONCE(cpu_is_offline(cpu));
@@ -116,10 +120,7 @@ bool irq_work_queue_on(struct irq_work *
 		return false;
 
 	preempt_disable();
-
-	lazy_work = work->flags & IRQ_WORK_LAZY;
-
-	if (lazy_work || (realtime && !(work->flags & IRQ_WORK_HARD_IRQ)))
+	if (use_lazy_list(work))
 		list = &per_cpu(lazy_list, cpu);
 	else
 		list = &per_cpu(raised_list, cpu);