[PATCH] generic: Add the exception case checking routine for ppi interrupt

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[MaJun]

From: Ma Jun <majun258@huawei.com>

During system booting, if the interrupt which has no action registered
is triggered, it would cause system panic when try to access the
action member.

Signed-off-by: Ma Jun <majun258@huawei.com>
---
 kernel/irq/chip.c |   20 ++++++++++++++++----
 1 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index 8114d06..9a0e872 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -766,11 +766,23 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc)
  */
 void handle_percpu_devid_irq(unsigned int irq, struct irq_desc *desc)
 {
-	struct irq_chip *chip = irq_desc_get_chip(desc);
-	struct irqaction *action = desc->action;
-	void *dev_id = raw_cpu_ptr(action->percpu_dev_id);
+	struct irq_chip *chip = NULL;
+	struct irqaction *action;
+	void *dev_id;
 	irqreturn_t res;
 
+	action = desc->action;
+
+	/* Unexpected interrupt in some execption case
+	 * we just send eoi to end this interrupt
+	 */
+	if (unlikely(!action)) {
+		mask_irq(desc);
+		goto out;
+	}
+	dev_id = raw_cpu_ptr(action->percpu_dev_id);
+
+	chip = irq_desc_get_chip(desc);
 	kstat_incr_irqs_this_cpu(irq, desc);
 
 	if (chip->irq_ack)
@@ -779,7 +791,7 @@ void handle_percpu_devid_irq(unsigned int irq, struct irq_desc *desc)
 	trace_irq_handler_entry(irq, action);
 	res = action->handler(irq, dev_id);
 	trace_irq_handler_exit(irq, action, res);
-
+out:
 	if (chip->irq_eoi)
 		chip->irq_eoi(&desc->irq_data);
 }
-- 
1.7.1

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Marc Zyngier]

On 30/08/16 05:17, MaJun wrote:
> From: Ma Jun <majun258@huawei.com>
> 
> During system booting, if the interrupt which has no action registered
> is triggered, it would cause system panic when try to access the
> action member.

And why would that interrupt be enabled? If you enable a PPI before
registering a handler, you're doing something wrong.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[majun (F)]

? 2016/8/30 16:50, Marc Zyngier ??:
> On 30/08/16 05:17, MaJun wrote:
>> From: Ma Jun <majun258@huawei.com>
>>
>> During system booting, if the interrupt which has no action registered
>> is triggered, it would cause system panic when try to access the
>> action member.
> 
> And why would that interrupt be enabled? If you enable a PPI before
> registering a handler, you're doing something wrong.
> 

Actually,the problem described above happened during the capture kernel booting.

In my system, sometimes there is a pending physical timer interrupt(30)
when the first kernel panic and the status is kept until the capture kernel booting.

So, this interrupt will be handled during capture kernel booting.

Becasue we use virt timer interrupt but not physical timer interrupt in capture kernel,
the interrupt 30 has no action handler.

Besides, I think we need to do exception check in this function just
like "handle_fasteoi_irq" does.

Thanks
MaJun

> Thanks,
> 
> 	M.
> 

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Marc Zyngier]

+Mark

On 30/08/16 11:35, majun (F) wrote:
> 
> 
> ? 2016/8/30 16:50, Marc Zyngier ??:
>> On 30/08/16 05:17, MaJun wrote:
>>> From: Ma Jun <majun258@huawei.com>
>>>
>>> During system booting, if the interrupt which has no action registered
>>> is triggered, it would cause system panic when try to access the
>>> action member.
>>
>> And why would that interrupt be enabled? If you enable a PPI before
>> registering a handler, you're doing something wrong.
>>
> 
> Actually,the problem described above happened during the capture
> kernel booting.
> 
> In my system, sometimes there is a pending physical timer
> interrupt(30) when the first kernel panic and the status is kept
> until the capture kernel booting.

And that's perfectly fine. The interrupt can be pending forever, as it
shouldn't get enabled.

> So, this interrupt will be handled during capture kernel booting.

Why? Who enables it?

> 
> Becasue we use virt timer interrupt but not physical timer interrupt
> in capture kernel, the interrupt 30 has no action handler.

Again: who enables this interrupt? Whichever driver enables it should be
fixed.

> Besides, I think we need to do exception check in this function just
> like "handle_fasteoi_irq" does.

I respectfully disagree. This will only hide a whole class of silly
bugs, and I'd rather squash them instead of papering over them.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Mark Rutland]

On Tue, Aug 30, 2016 at 12:07:36PM +0100, Marc Zyngier wrote:
> +Mark
> On 30/08/16 11:35, majun (F) wrote:
> > ? 2016/8/30 16:50, Marc Zyngier ??:
> >> On 30/08/16 05:17, MaJun wrote:
> >>> From: Ma Jun <majun258@huawei.com>
> >>>
> >>> During system booting, if the interrupt which has no action registered
> >>> is triggered, it would cause system panic when try to access the
> >>> action member.
> >>
> >> And why would that interrupt be enabled? If you enable a PPI before
> >> registering a handler, you're doing something wrong.
> > 
> > Actually,the problem described above happened during the capture
> > kernel booting.
> > 
> > In my system, sometimes there is a pending physical timer
> > interrupt(30) when the first kernel panic and the status is kept
> > until the capture kernel booting.
> 
> And that's perfectly fine. The interrupt can be pending forever, as it
> shouldn't get enabled.
> 
> > So, this interrupt will be handled during capture kernel booting.
> 
> Why? Who enables it?
> 
> > Becasue we use virt timer interrupt but not physical timer interrupt
> > in capture kernel, the interrupt 30 has no action handler.
> 
> Again: who enables this interrupt? Whichever driver enables it should be
> fixed.

I'm also at a loss.

In this case, arch_timer_uses_ppi must be VIRT_PPI. So in
arch_timer_register(), we'll only request_percpu_irq the virt PPI.
arch_timer_has_nonsecure_ppi() will be false, given arch_timer_uses_ppi
is VIRT_PPI, so in arch_timer_starting_cpu() we'll only
enable_percpu_irq() the virt PPI.

We don't fiddle with arch_timer_uses_ppi after calling
arch_timer_register(). So I can't see how we could enable another IRQ in
this case.

Looking at the driver in virt/kvm/arm/arch_timer.c, we only enable what
we've succesfully requested, so it doesnt' seem like there's an issue
there.

>From a quick look at teh GIC driver, it looks like we reset PPIs
correctly, so it doesn't look like we have a "latent enable".

Thanks,
Mark.

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[majun (F)]

Hi Marc & Mark:

? 2016/8/30 19:21, Mark Rutland ??:
> On Tue, Aug 30, 2016 at 12:07:36PM +0100, Marc Zyngier wrote:
>> +Mark
>> On 30/08/16 11:35, majun (F) wrote:
>>> ? 2016/8/30 16:50, Marc Zyngier ??:
>>>> On 30/08/16 05:17, MaJun wrote:
>>>>> From: Ma Jun <majun258@huawei.com>
>>>>>
>>>>> During system booting, if the interrupt which has no action registered
>>>>> is triggered, it would cause system panic when try to access the
>>>>> action member.
>>>>
>>>> And why would that interrupt be enabled? If you enable a PPI before
>>>> registering a handler, you're doing something wrong.
>>>
>>> Actually,the problem described above happened during the capture
>>> kernel booting.
>>>
>>> In my system, sometimes there is a pending physical timer
>>> interrupt(30) when the first kernel panic and the status is kept
>>> until the capture kernel booting.
>>
>> And that's perfectly fine. The interrupt can be pending forever, as it
>> shouldn't get enabled.
>>
>>> So, this interrupt will be handled during capture kernel booting.
>>
>> Why? Who enables it?
>>
>>> Becasue we use virt timer interrupt but not physical timer interrupt
>>> in capture kernel, the interrupt 30 has no action handler.
>>
>> Again: who enables this interrupt? Whichever driver enables it should be
>> fixed.
> 
> I'm also at a loss.
> 
> In this case, arch_timer_uses_ppi must be VIRT_PPI. So in
> arch_timer_register(), we'll only request_percpu_irq the virt PPI.
> arch_timer_has_nonsecure_ppi() will be false, given arch_timer_uses_ppi
> is VIRT_PPI, so in arch_timer_starting_cpu() we'll only
> enable_percpu_irq() the virt PPI.
> 
> We don't fiddle with arch_timer_uses_ppi after calling
> arch_timer_register(). So I can't see how we could enable another IRQ in
> this case.
> 
> Looking at the driver in virt/kvm/arm/arch_timer.c, we only enable what
> we've succesfully requested, so it doesnt' seem like there's an issue
> there.
> 
>>From a quick look at teh GIC driver, it looks like we reset PPIs
> correctly, so it doesn't look like we have a "latent enable".
> 

I just checked the status of irq 30 during capture kernel booting.

The irq 30 status is: mask, pending after arch_timer_starting_cpu() called.
Because irq 30 triggered only 1 time during capture kernel booting,
I think this problem maybe happened in the case like:
1:irq 30 triggered, but not acked by cpu yet.
2:local_irq_disable() called
3:system reboot -->capture kernel booting
4:local_irq_enable()
5:irq 30 acked by CPU.

Is this case possible?

Thanks
MaJun

> Thanks,
> Mark.
> 
> .
> 

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Marc Zyngier]

On 31/08/16 07:35, majun (F) wrote:
> Hi Marc & Mark:
> 
> ? 2016/8/30 19:21, Mark Rutland ??:
>> On Tue, Aug 30, 2016 at 12:07:36PM +0100, Marc Zyngier wrote:
>>> +Mark
>>> On 30/08/16 11:35, majun (F) wrote:
>>>> ? 2016/8/30 16:50, Marc Zyngier ??:
>>>>> On 30/08/16 05:17, MaJun wrote:
>>>>>> From: Ma Jun <majun258@huawei.com>
>>>>>>
>>>>>> During system booting, if the interrupt which has no action registered
>>>>>> is triggered, it would cause system panic when try to access the
>>>>>> action member.
>>>>>
>>>>> And why would that interrupt be enabled? If you enable a PPI before
>>>>> registering a handler, you're doing something wrong.
>>>>
>>>> Actually,the problem described above happened during the capture
>>>> kernel booting.
>>>>
>>>> In my system, sometimes there is a pending physical timer
>>>> interrupt(30) when the first kernel panic and the status is kept
>>>> until the capture kernel booting.
>>>
>>> And that's perfectly fine. The interrupt can be pending forever, as it
>>> shouldn't get enabled.
>>>
>>>> So, this interrupt will be handled during capture kernel booting.
>>>
>>> Why? Who enables it?
>>>
>>>> Becasue we use virt timer interrupt but not physical timer interrupt
>>>> in capture kernel, the interrupt 30 has no action handler.
>>>
>>> Again: who enables this interrupt? Whichever driver enables it should be
>>> fixed.
>>
>> I'm also at a loss.
>>
>> In this case, arch_timer_uses_ppi must be VIRT_PPI. So in
>> arch_timer_register(), we'll only request_percpu_irq the virt PPI.
>> arch_timer_has_nonsecure_ppi() will be false, given arch_timer_uses_ppi
>> is VIRT_PPI, so in arch_timer_starting_cpu() we'll only
>> enable_percpu_irq() the virt PPI.
>>
>> We don't fiddle with arch_timer_uses_ppi after calling
>> arch_timer_register(). So I can't see how we could enable another IRQ in
>> this case.
>>
>> Looking at the driver in virt/kvm/arm/arch_timer.c, we only enable what
>> we've succesfully requested, so it doesnt' seem like there's an issue
>> there.
>>
>> >From a quick look at teh GIC driver, it looks like we reset PPIs
>> correctly, so it doesn't look like we have a "latent enable".
>>
> 
> I just checked the status of irq 30 during capture kernel booting.
> 
> The irq 30 status is: mask, pending after arch_timer_starting_cpu() called.
> Because irq 30 triggered only 1 time during capture kernel booting,
> I think this problem maybe happened in the case like:
> 1:irq 30 triggered, but not acked by cpu yet.
> 2:local_irq_disable() called
> 3:system reboot -->capture kernel booting
> 4:local_irq_enable()
> 5:irq 30 acked by CPU.
> 
> Is this case possible?

I can't see how, because you've missed:

3b: All PPIs are disabled as each CPU comes up

So for (5) to occur, I can only see two possibilities:
(a) either something else is enabling the timer PPI
(b) your GIC doesn't correctly retire a pending PPI that is being disabled

I'm discounting (b) because I can't see how the system would work
otherwise, so (a) must be happening somehow.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[majun (F)]

? 2016/8/31 16:35, Marc Zyngier ??:
> On 31/08/16 07:35, majun (F) wrote:
[...]
>>>
>>
>> I just checked the status of irq 30 during capture kernel booting.
>>
>> The irq 30 status is: mask, pending after arch_timer_starting_cpu() called.
>> Because irq 30 triggered only 1 time during capture kernel booting,
>> I think this problem maybe happened in the case like:
>> 1:irq 30 triggered, but not acked by cpu yet.
>> 2:local_irq_disable() called
>> 3:system reboot -->capture kernel booting
>> 4:local_irq_enable()
>> 5:irq 30 acked by CPU.
>>
>> Is this case possible?
> 
> I can't see how, because you've missed:
> 
> 3b: All PPIs are disabled as each CPU comes up
> 
> So for (5) to occur, I can only see two possibilities:
> (a) either something else is enabling the timer PPI

I checked the whole process, the irq 30 alway keeping disabled.

> (b) your GIC doesn't correctly retire a pending PPI that is being disabled

According to our hardware guy said, GIC in our system has problem in this case.
Usually, when we mask irq 30, the interrupt which in pending status but not acked by cpu
should be released/cleared by hardware, but actually, we did't do like this in our system.

So, this conclusion just same as you assumption.

Do you have any suggestion or workaround for this problem?

Thanks!
Majun

> 
> I'm discounting (b) because I can't see how the system would work
> otherwise, so (a) must be happening somehow.
> 


> Thanks,
> 
> 	M.
> 

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Marc Zyngier]

On 01/09/16 09:15, majun (F) wrote:
> 
> 
> ? 2016/8/31 16:35, Marc Zyngier ??:
>> On 31/08/16 07:35, majun (F) wrote:
> [...]
>>>>
>>>
>>> I just checked the status of irq 30 during capture kernel booting.
>>>
>>> The irq 30 status is: mask, pending after arch_timer_starting_cpu() called.
>>> Because irq 30 triggered only 1 time during capture kernel booting,
>>> I think this problem maybe happened in the case like:
>>> 1:irq 30 triggered, but not acked by cpu yet.
>>> 2:local_irq_disable() called
>>> 3:system reboot -->capture kernel booting
>>> 4:local_irq_enable()
>>> 5:irq 30 acked by CPU.
>>>
>>> Is this case possible?
>>
>> I can't see how, because you've missed:
>>
>> 3b: All PPIs are disabled as each CPU comes up
>>
>> So for (5) to occur, I can only see two possibilities:
>> (a) either something else is enabling the timer PPI
> 
> I checked the whole process, the irq 30 alway keeping disabled.
> 
>> (b) your GIC doesn't correctly retire a pending PPI that is being disabled
> 
> According to our hardware guy said, GIC in our system has problem in this case.
> Usually, when we mask irq 30, the interrupt which in pending status but not acked by cpu
> should be released/cleared by hardware, but actually, we did't do like this in our system.

That's crazy. This means that you cannot reliably mask interrupts. :-(
Does this only affect PPIs? Or does it affect all interrupt types?

> So, this conclusion just same as you assumption.
> 
> Do you have any suggestion or workaround for this problem?

Well, this issue goes way beyond the hack you wanted to add to the
generic code, and it should probably be addressed in the GIC code
itself, as an implementation specific workaround. Without knowing the
details of the erratum, it is difficult to think of that would be
required. I can come up with something like this:

	irqnr = gic_read_iar();
	if (unlikely(!is_enabled(irqnr))) {
		gic_write_eoir(irqnr);
		if (static_key_true(&supports_deactivate))
			gic_write_dir(irqnr);
		set_pending(irqnr);
		continue;
	}

Performance will suffer (an extra MMIO access on the fast path). If LPIs
are also affected, then the ITS code also needs to be involved, and
that's not going to be pretty either. This code will have to be enabled
at runtime, and handled like other erratum we have in this code.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Thomas Gleixner]

On Thu, 1 Sep 2016, Marc Zyngier wrote:
> On 01/09/16 09:15, majun (F) wrote:
> Well, this issue goes way beyond the hack you wanted to add to the
> generic code, and it should probably be addressed in the GIC code
> itself, as an implementation specific workaround. Without knowing the
> details of the erratum, it is difficult to think of that would be
> required. I can come up with something like this:
> 
> 	irqnr = gic_read_iar();
> 	if (unlikely(!is_enabled(irqnr))) {
> 		gic_write_eoir(irqnr);
> 		if (static_key_true(&supports_deactivate))
> 			gic_write_dir(irqnr);
> 		set_pending(irqnr);
> 		continue;
> 	}
> 
> Performance will suffer (an extra MMIO access on the fast path). If LPIs
> are also affected, then the ITS code also needs to be involved, and
> that's not going to be pretty either. This code will have to be enabled
> at runtime, and handled like other erratum we have in this code.

So that's certainly a required workaround at the gic level. Though I really
think that we should make handle_percpu_devid_irq robust against a spurious
interrupt.

>  void handle_percpu_devid_irq(unsigned int irq, struct irq_desc *desc)
>  {
> -	struct irq_chip *chip = irq_desc_get_chip(desc);
> -	struct irqaction *action = desc->action;
> -	void *dev_id = raw_cpu_ptr(action->percpu_dev_id);
> +	struct irq_chip *chip = NULL;
> +	struct irqaction *action;
> +	void *dev_id;
>  	irqreturn_t res;
>  
> +	action = desc->action;
> +
> +	/* Unexpected interrupt in some execption case
> +	 * we just send eoi to end this interrupt
> +	 */
> +	if (unlikely(!action)) {
> +		mask_irq(desc);

This is wrong. mask_irq() does not work for percpu interrupts. Aside of that
this completely lacks any debug information which tells us that there is
something wrong in the system. I'm going to apply the patch below for
robustness sake.

Thanks,

	tglx

8<----------------------
Subject: genirq: Robustify handle_percpu_devid_irq()
From: Thomas Gleixner <tglx@linutronix.de>
Date: Fri, 02 Sep 2016 14:45:19 +0200

The percpu_devid handler is not robust against spurious interrupts. If a
spurious interrupt happens and no action is installed then the handler crashes
with a NULL pointer dereference.

Add a sanity check for this and log the wreckage once in dmesg.

Reported-by: Majun <majun258@huawei.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 kernel/irq/chip.c |   18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -756,7 +756,6 @@ void handle_percpu_devid_irq(struct irq_
 {
 	struct irq_chip *chip = irq_desc_get_chip(desc);
 	struct irqaction *action = desc->action;
-	void *dev_id = raw_cpu_ptr(action->percpu_dev_id);
 	unsigned int irq = irq_desc_get_irq(desc);
 	irqreturn_t res;
 
@@ -765,9 +764,20 @@ void handle_percpu_devid_irq(struct irq_
 	if (chip->irq_ack)
 		chip->irq_ack(&desc->irq_data);
 
-	trace_irq_handler_entry(irq, action);
-	res = action->handler(irq, dev_id);
-	trace_irq_handler_exit(irq, action, res);
+	if (likely(action)) {
+		trace_irq_handler_entry(irq, action);
+		res = action->handler(irq, raw_cpu_ptr(action->percpu_dev_id));
+		trace_irq_handler_exit(irq, action, res);
+	} else {
+		unsigned int cpu = smp_processor_id();
+		bool enabled = cpumask_test_cpu(cpu, desc->percpu_enabled);
+
+		if (enabled)
+			irq_percpu_disable(desc, cpu);
+
+		pr_err_once("Spurious%s percpu IRQ%u on CPU%u\n",
+			    enabled ? " and unmasked" : "", irq, cpu);
+	}
 
 	if (chip->irq_eoi)
 		chip->irq_eoi(&desc->irq_data);

[PATCH] generic: Add the exception case checking routine for ppi interrupt

[Marc Zyngier]

On 02/09/16 14:08, Thomas Gleixner wrote:
> On Thu, 1 Sep 2016, Marc Zyngier wrote:
>> On 01/09/16 09:15, majun (F) wrote:
>> Well, this issue goes way beyond the hack you wanted to add to the
>> generic code, and it should probably be addressed in the GIC code
>> itself, as an implementation specific workaround. Without knowing the
>> details of the erratum, it is difficult to think of that would be
>> required. I can come up with something like this:
>>
>> 	irqnr = gic_read_iar();
>> 	if (unlikely(!is_enabled(irqnr))) {
>> 		gic_write_eoir(irqnr);
>> 		if (static_key_true(&supports_deactivate))
>> 			gic_write_dir(irqnr);
>> 		set_pending(irqnr);
>> 		continue;
>> 	}
>>
>> Performance will suffer (an extra MMIO access on the fast path). If LPIs
>> are also affected, then the ITS code also needs to be involved, and
>> that's not going to be pretty either. This code will have to be enabled
>> at runtime, and handled like other erratum we have in this code.
> 
> So that's certainly a required workaround at the gic level. Though I really
> think that we should make handle_percpu_devid_irq robust against a spurious
> interrupt.
> 
>>  void handle_percpu_devid_irq(unsigned int irq, struct irq_desc *desc)
>>  {
>> -	struct irq_chip *chip = irq_desc_get_chip(desc);
>> -	struct irqaction *action = desc->action;
>> -	void *dev_id = raw_cpu_ptr(action->percpu_dev_id);
>> +	struct irq_chip *chip = NULL;
>> +	struct irqaction *action;
>> +	void *dev_id;
>>  	irqreturn_t res;
>>  
>> +	action = desc->action;
>> +
>> +	/* Unexpected interrupt in some execption case
>> +	 * we just send eoi to end this interrupt
>> +	 */
>> +	if (unlikely(!action)) {
>> +		mask_irq(desc);
> 
> This is wrong. mask_irq() does not work for percpu interrupts. Aside of that
> this completely lacks any debug information which tells us that there is
> something wrong in the system. I'm going to apply the patch below for
> robustness sake.
> 
> Thanks,
> 
> 	tglx
> 
> 8<----------------------
> Subject: genirq: Robustify handle_percpu_devid_irq()
> From: Thomas Gleixner <tglx@linutronix.de>
> Date: Fri, 02 Sep 2016 14:45:19 +0200
> 
> The percpu_devid handler is not robust against spurious interrupts. If a
> spurious interrupt happens and no action is installed then the handler crashes
> with a NULL pointer dereference.
> 
> Add a sanity check for this and log the wreckage once in dmesg.
> 
> Reported-by: Majun <majun258@huawei.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Looks fine to me.

Acked-by: Marc Zyngier <marc.zyngier@arm.com>

	M.
-- 
Jazz is not dead. It just smells funny...