[PATCH 0/2] Fix NMI backtrace for Inforce IFC6410

[PATCH 0/2] Fix NMI backtrace for Inforce IFC6410

[Daniel Thompson]

After spotting some problems with the SysRq-L on Inforce IFC6410 I ran
some tests (see patch 2) and concluded that certain SGI IDs cannot be
used by Linux on these platforms because they have been reserved for use
by the secure world.

This patchset includes both a patch to resolve the problem on Inforce
IFC6410 and a patch to detect and report reserved SGI IDs during boot.
The patch (deliberately) keeps quiet on platforms where the kernel's
world is not subject to any restrictions.


Daniel Thompson (2):
  arm: Fix "NMI" backtrace for Inforce IFC6410
  irqchip/gic: Identify and report any reserved SGI IDs

 arch/arm/kernel/smp.c     |  3 ++-
 drivers/irqchip/irq-gic.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)

--
2.5.0

[PATCH 1/2] arm: Fix "NMI" backtrace for Inforce IFC6410

[Daniel Thompson]

SysRq-L does not generate a backtrace from all CPUs when I test it
on my Inforce IFC6410 platform (Snapdragon 600). The stack dump code,
triggered by IPI_CPU_BACKTRACE, never runs on the other CPUs.
Eventually we hit the 10 second timeout and a subset of the expected
stack dumps on are shown on the console.

It is likely this is because SGI IDs 14 and 15 have been reserved for
use by secure world on this platform. For IFC6410 platform the code
works as expected when IPI_CPU_BACKTRACE is set to any value in the
interval 9..13.

Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
 arch/arm/kernel/smp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index b26361355dae..78205927fcd4 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -73,7 +73,8 @@ enum ipi_msg_type {
 	IPI_CPU_STOP,
 	IPI_IRQ_WORK,
 	IPI_COMPLETION,
-	IPI_CPU_BACKTRACE = 15,
+	IPI_CPU_BACKTRACE = 13,
+	/* 14 and 15 are reserved; they do not work on some Krait CPUs */
 };
 
 static DECLARE_COMPLETION(cpu_running);
-- 
2.5.0

[PATCH 2/2] irqchip/gic: Identify and report any reserved SGI IDs

[Daniel Thompson]

It is possible for the secure world to reserve certain SGI IDs for itself.
Currently we have limited visibility of which IDs are safe to use for IPIs.

Modify the GIC initialization code to actively search for reserved SGI IDs
and report if any are found. Warn even more loudly if the reserved SGIs
overlap with the normal IPI range.

When run on an Inforce IFC6410 (Snapdragon 600) this code produces the
following messages:
~~~ cut here ~~~
CPU0: Detected reserved SGI IDs: 14-15
CPU1: Detected reserved SGI IDs: 15
CPU2: Detected reserved SGI IDs: 15
CPU3: Detected reserved SGI IDs: 15
~~~ cut here ~~~

Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
 drivers/irqchip/irq-gic.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index abf2ffaed392..541622da7049 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -490,6 +490,7 @@ static void gic_cpu_init(struct gic_chip_data *gic)
 	void __iomem *base = gic_data_cpu_base(gic);
 	unsigned int cpu_mask, cpu = smp_processor_id();
 	int i;
+	DECLARE_BITMAP(sgi_mask, 16);
 
 	/*
 	 * Setting up the CPU map is only relevant for the primary GIC
@@ -511,6 +512,58 @@ static void gic_cpu_init(struct gic_chip_data *gic)
 		for (i = 0; i < NR_GIC_CPU_IF; i++)
 			if (i != cpu)
 				gic_cpu_map[i] &= ~cpu_mask;
+
+		/*
+		 * Fiddle with the SGI set/clear registers to try identify
+		 * any IPIs that are reserved for secure world.
+		 */
+		bitmap_fill(sgi_mask, 16);
+
+		for (i = 0; i < 16; i++) {
+			void __iomem *set_reg =
+			    dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3);
+			void __iomem *clear_reg =
+			    dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3);
+			unsigned long mask = cpu_mask << (8*(i%4));
+			unsigned long flags, pending, after_clear, after_set;
+
+			local_irq_save(flags);
+
+			/* record original value */
+			pending = readl_relaxed(set_reg);
+
+			/* clear, test, set, and test again */
+			writel_relaxed(mask, clear_reg);
+			after_clear = readl_relaxed(set_reg);
+			writel_relaxed(mask, set_reg);
+			after_set = readl_relaxed(set_reg);
+
+			/* restore original value */
+			writel_relaxed(mask & ~pending, clear_reg);
+
+			local_irq_restore(flags);
+
+			if (mask & ~after_clear && mask & after_set)
+				clear_bit(i, sgi_mask);
+		}
+
+		/*
+		 * Show the SGI mask if it is "interesting". Here interesting
+		 * means that the set/clear register is implemented
+		 * (mask is not full) and it tells us that the secure world
+		 * has reserved some SGIs (mask is not empty).
+		 */
+		if (!bitmap_full(sgi_mask, 16) && !bitmap_empty(sgi_mask, 16))
+			pr_info("CPU%d: Detected reserved SGI IDs: %*pbl\n",
+				cpu, 16, sgi_mask);
+
+		/*
+		 * Yell if the reserved IDs make the system unviable.
+		 */
+		if (!bitmap_full(sgi_mask, 16) &&
+		    find_first_bit(sgi_mask, 16) < NR_IPI)
+			pr_crit("CPU%d: Not enough SGI IDs; expect failure\n",
+				cpu);
 	}
 
 	gic_cpu_config(dist_base, NULL);
-- 
2.5.0

Re: [PATCH 2/2] irqchip/gic: Identify and report any reserved SGI IDs

[Marc Zyngier]

Hi Daniel,

On 16/12/15 17:08, Daniel Thompson wrote:
> It is possible for the secure world to reserve certain SGI IDs for itself.
> Currently we have limited visibility of which IDs are safe to use for IPIs.
> 
> Modify the GIC initialization code to actively search for reserved SGI IDs
> and report if any are found. Warn even more loudly if the reserved SGIs
> overlap with the normal IPI range.
> 
> When run on an Inforce IFC6410 (Snapdragon 600) this code produces the
> following messages:
> ~~~ cut here ~~~
> CPU0: Detected reserved SGI IDs: 14-15
> CPU1: Detected reserved SGI IDs: 15
> CPU2: Detected reserved SGI IDs: 15
> CPU3: Detected reserved SGI IDs: 15
> ~~~ cut here ~~~
> 
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> ---
>  drivers/irqchip/irq-gic.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 53 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
> index abf2ffaed392..541622da7049 100644
> --- a/drivers/irqchip/irq-gic.c
> +++ b/drivers/irqchip/irq-gic.c
> @@ -490,6 +490,7 @@ static void gic_cpu_init(struct gic_chip_data *gic)
>  	void __iomem *base = gic_data_cpu_base(gic);
>  	unsigned int cpu_mask, cpu = smp_processor_id();
>  	int i;
> +	DECLARE_BITMAP(sgi_mask, 16);
>  
>  	/*
>  	 * Setting up the CPU map is only relevant for the primary GIC
> @@ -511,6 +512,58 @@ static void gic_cpu_init(struct gic_chip_data *gic)
>  		for (i = 0; i < NR_GIC_CPU_IF; i++)
>  			if (i != cpu)
>  				gic_cpu_map[i] &= ~cpu_mask;
> +
> +		/*
> +		 * Fiddle with the SGI set/clear registers to try identify
> +		 * any IPIs that are reserved for secure world.
> +		 */
> +		bitmap_fill(sgi_mask, 16);
> +
> +		for (i = 0; i < 16; i++) {
> +			void __iomem *set_reg =
> +			    dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3);
> +			void __iomem *clear_reg =
> +			    dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3);
> +			unsigned long mask = cpu_mask << (8*(i%4));
> +			unsigned long flags, pending, after_clear, after_set;

Please make these u32, as unsigned long is 64bit on arm64. Another thing
to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can
save yourself some this hassle shifting things around and just write a
single byte. You're already writing 16 times anyway...

Another thing to consider is that these locations are only defined on
GICv2 and not GICv1, so this patch is likely to cause trouble on older HW.

> +
> +			local_irq_save(flags);

Why do you need to do this? The CPU interface is not enabled yet, so I
can't see how you could get an interrupt on this CPU.

> +
> +			/* record original value */
> +			pending = readl_relaxed(set_reg);
> +
> +			/* clear, test, set, and test again */
> +			writel_relaxed(mask, clear_reg);
> +			after_clear = readl_relaxed(set_reg);
> +			writel_relaxed(mask, set_reg);
> +			after_set = readl_relaxed(set_reg);

It should be enough to write to the SET register, and read back, as the
bit is RAZ/WI when the interrupt is Group-0.

> +
> +			/* restore original value */
> +			writel_relaxed(mask & ~pending, clear_reg);
> +
> +			local_irq_restore(flags);
> +
> +			if (mask & ~after_clear && mask & after_set)
> +				clear_bit(i, sgi_mask);
> +		}
> +
> +		/*
> +		 * Show the SGI mask if it is "interesting". Here interesting
> +		 * means that the set/clear register is implemented
> +		 * (mask is not full) and it tells us that the secure world
> +		 * has reserved some SGIs (mask is not empty).
> +		 */
> +		if (!bitmap_full(sgi_mask, 16) && !bitmap_empty(sgi_mask, 16))
> +			pr_info("CPU%d: Detected reserved SGI IDs: %*pbl\n",
> +				cpu, 16, sgi_mask);
> +
> +		/*
> +		 * Yell if the reserved IDs make the system unviable.
> +		 */
> +		if (!bitmap_full(sgi_mask, 16) &&
> +		    find_first_bit(sgi_mask, 16) < NR_IPI)
> +			pr_crit("CPU%d: Not enough SGI IDs; expect failure\n",
> +				cpu);
>  	}
>  
>  	gic_cpu_config(dist_base, NULL);
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/2] irqchip/gic: Identify and report any reserved SGI IDs

[Daniel Thompson]

On Wed, Dec 16, 2015 at 05:47:09PM +0000, Marc Zyngier wrote:
> Hi Daniel,

Hi Marc

Thanks for the review.


> On 16/12/15 17:08, Daniel Thompson wrote:
> > It is possible for the secure world to reserve certain SGI IDs for itself.
> > Currently we have limited visibility of which IDs are safe to use for IPIs.
> > 
> > Modify the GIC initialization code to actively search for reserved SGI IDs
> > and report if any are found. Warn even more loudly if the reserved SGIs
> > overlap with the normal IPI range.
> > 
> > When run on an Inforce IFC6410 (Snapdragon 600) this code produces the
> > following messages:
> > ~~~ cut here ~~~
> > CPU0: Detected reserved SGI IDs: 14-15
> > CPU1: Detected reserved SGI IDs: 15
> > CPU2: Detected reserved SGI IDs: 15
> > CPU3: Detected reserved SGI IDs: 15
> > ~~~ cut here ~~~
> > 
> > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>

BTW you *didn't* say "this code is pointless and I hate it"...

Does that mean I should be looking at adding similar code for GICv3+? I 
wanted to guage reactions to this sort of diagnostics before getting
carried away!


> > +
> > +		/*
> > +		 * Fiddle with the SGI set/clear registers to try identify
> > +		 * any IPIs that are reserved for secure world.
> > +		 */
> > +		bitmap_fill(sgi_mask, 16);
> > +
> > +		for (i = 0; i < 16; i++) {
> > +			void __iomem *set_reg =
> > +			    dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3);
> > +			void __iomem *clear_reg =
> > +			    dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3);
> > +			unsigned long mask = cpu_mask << (8*(i%4));
> > +			unsigned long flags, pending, after_clear, after_set;
> 
> Please make these u32, as unsigned long is 64bit on arm64. Another thing
> to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can
> save yourself some this hassle shifting things around and just write a
> single byte. You're already writing 16 times anyway...

Will do both.


> Another thing to consider is that these locations are only defined on
> GICv2 and not GICv1, so this patch is likely to cause trouble on older HW.

As presented the code relies on the RAZ/WI property of reserved
registers to avoid issues on GICv1; it does not report anything if there
appear to be know working SGIs on the assumption we are actually running
on a GICv1.

You'd prefer an explicit version check?


> > +
> > +			local_irq_save(flags);
> 
> Why do you need to do this? The CPU interface is not enabled yet, so I
> can't see how you could get an interrupt on this CPU.

Agreed. Can get rid of these.


> > +
> > +			/* record original value */
> > +			pending = readl_relaxed(set_reg);
> > +
> > +			/* clear, test, set, and test again */
> > +			writel_relaxed(mask, clear_reg);
> > +			after_clear = readl_relaxed(set_reg);
> > +			writel_relaxed(mask, set_reg);
> > +			after_set = readl_relaxed(set_reg);
> 
> It should be enough to write to the SET register, and read back, as the
> bit is RAZ/WI when the interrupt is Group-0.

Good point. Will simplify.


Daniel.

Re: [PATCH 2/2] irqchip/gic: Identify and report any reserved SGI IDs

[Marc Zyngier]

On Thu, 17 Dec 2015 19:26:10 +0000
Daniel Thompson <daniel.thompson@linaro.org> wrote:

> On Wed, Dec 16, 2015 at 05:47:09PM +0000, Marc Zyngier wrote:
> > Hi Daniel,
> 
> Hi Marc
> 
> Thanks for the review.
> 
> 
> > On 16/12/15 17:08, Daniel Thompson wrote:
> > > It is possible for the secure world to reserve certain SGI IDs for itself.
> > > Currently we have limited visibility of which IDs are safe to use for IPIs.
> > > 
> > > Modify the GIC initialization code to actively search for reserved SGI IDs
> > > and report if any are found. Warn even more loudly if the reserved SGIs
> > > overlap with the normal IPI range.
> > > 
> > > When run on an Inforce IFC6410 (Snapdragon 600) this code produces the
> > > following messages:
> > > ~~~ cut here ~~~
> > > CPU0: Detected reserved SGI IDs: 14-15
> > > CPU1: Detected reserved SGI IDs: 15
> > > CPU2: Detected reserved SGI IDs: 15
> > > CPU3: Detected reserved SGI IDs: 15
> > > ~~~ cut here ~~~
> > > 
> > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> 
> BTW you *didn't* say "this code is pointless and I hate it"...

Well, this was obviously used to detect another issue (patch #1), so I
can't see any harm in trying to sanitize things, as long as it doesn't
break anything else. I imagine this patch will also trigger on my
sunxi platforms, which use a SGI to implement PSCI in secure mode.

> Does that mean I should be looking at adding similar code for GICv3+? I 
> wanted to guage reactions to this sort of diagnostics before getting
> carried away!

Could be useful as well.

> 
> 
> > > +
> > > +		/*
> > > +		 * Fiddle with the SGI set/clear registers to try identify
> > > +		 * any IPIs that are reserved for secure world.
> > > +		 */
> > > +		bitmap_fill(sgi_mask, 16);
> > > +
> > > +		for (i = 0; i < 16; i++) {
> > > +			void __iomem *set_reg =
> > > +			    dist_base + GIC_DIST_SGI_PENDING_SET + (i & ~3);
> > > +			void __iomem *clear_reg =
> > > +			    dist_base + GIC_DIST_SGI_PENDING_CLEAR + (i & ~3);
> > > +			unsigned long mask = cpu_mask << (8*(i%4));
> > > +			unsigned long flags, pending, after_clear, after_set;
> > 
> > Please make these u32, as unsigned long is 64bit on arm64. Another thing
> > to note is that GICD_CPEND{S,C}SGIRn are byte accessible, so you can
> > save yourself some this hassle shifting things around and just write a
> > single byte. You're already writing 16 times anyway...
> 
> Will do both.
> 
> 
> > Another thing to consider is that these locations are only defined on
> > GICv2 and not GICv1, so this patch is likely to cause trouble on older HW.
> 
> As presented the code relies on the RAZ/WI property of reserved
> registers to avoid issues on GICv1; it does not report anything if there
> appear to be know working SGIs on the assumption we are actually running
> on a GICv1.
> 
> You'd prefer an explicit version check?

I'd rather be cautious and check for the architecture version,
specially if you settle for the byte access mentioned above (a GICv1
may not support byte access and explode unexpectedly). ICPIDR2.ArchRev
should be the right thing to check.

> 
> 
> > > +
> > > +			local_irq_save(flags);
> > 
> > Why do you need to do this? The CPU interface is not enabled yet, so I
> > can't see how you could get an interrupt on this CPU.
> 
> Agreed. Can get rid of these.
> 
> 
> > > +
> > > +			/* record original value */
> > > +			pending = readl_relaxed(set_reg);
> > > +
> > > +			/* clear, test, set, and test again */
> > > +			writel_relaxed(mask, clear_reg);
> > > +			after_clear = readl_relaxed(set_reg);
> > > +			writel_relaxed(mask, set_reg);
> > > +			after_set = readl_relaxed(set_reg);
> > 
> > It should be enough to write to the SET register, and read back, as the
> > bit is RAZ/WI when the interrupt is Group-0.
> 
> Good point. Will simplify.

I'd also suggest moving the whole thing to a separate function that'd
get called from gic_cpu_init().

Thanks,

	M.
-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 1/2] arm: Fix "NMI" backtrace for Inforce IFC6410

[Marc Zyngier]

On 16/12/15 17:08, Daniel Thompson wrote:
> SysRq-L does not generate a backtrace from all CPUs when I test it
> on my Inforce IFC6410 platform (Snapdragon 600). The stack dump code,
> triggered by IPI_CPU_BACKTRACE, never runs on the other CPUs.
> Eventually we hit the 10 second timeout and a subset of the expected
> stack dumps on are shown on the console.
> 
> It is likely this is because SGI IDs 14 and 15 have been reserved for
> use by secure world on this platform. For IFC6410 platform the code
> works as expected when IPI_CPU_BACKTRACE is set to any value in the
> interval 9..13.
> 
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> ---
>  arch/arm/kernel/smp.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> index b26361355dae..78205927fcd4 100644
> --- a/arch/arm/kernel/smp.c
> +++ b/arch/arm/kernel/smp.c
> @@ -73,7 +73,8 @@ enum ipi_msg_type {
>  	IPI_CPU_STOP,
>  	IPI_IRQ_WORK,
>  	IPI_COMPLETION,
> -	IPI_CPU_BACKTRACE = 15,
> +	IPI_CPU_BACKTRACE = 13,
> +	/* 14 and 15 are reserved; they do not work on some Krait CPUs */
>  };
>  
>  static DECLARE_COMPLETION(cpu_running);
> 

It looks to me that we're just moving the goal posts and keep using a
SGI that is likely to be reserved by the secure side. Quoting the GIC spec:

In any system that implements the ARM Security Extensions, to support a
consistent model for message passing between processors, ARM strongly
recommends that all processors reserve:
* ID0-ID7 for Non-secure interrupts
* ID8-ID15 for Secure interrupts.

Of course, this is only a recommendation, but it is one that is likely
to be followed...

Now, we are already using our 8 "non-secure" SGIs, but we can
immediately reclaim one (IPI_CALL_FUNC_SINGLE is not that useful
anymore). I'll cook a patch for that.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 2/2] irqchip/gic: Identify and report any reserved SGI IDs

[Daniel Thompson]

On 18/12/15 07:39, Marc Zyngier wrote:
>>> On 16/12/15 17:08, Daniel Thompson wrote:
>>>> It is possible for the secure world to reserve certain SGI IDs for itself.
>>>> Currently we have limited visibility of which IDs are safe to use for IPIs.
>>>>
>>>> Modify the GIC initialization code to actively search for reserved SGI IDs
>>>> and report if any are found. Warn even more loudly if the reserved SGIs
>>>> overlap with the normal IPI range.
>>>>
>>>> When run on an Inforce IFC6410 (Snapdragon 600) this code produces the
>>>> following messages:
>>>> ~~~ cut here ~~~
>>>> CPU0: Detected reserved SGI IDs: 14-15
>>>> CPU1: Detected reserved SGI IDs: 15
>>>> CPU2: Detected reserved SGI IDs: 15
>>>> CPU3: Detected reserved SGI IDs: 15
>>>> ~~~ cut here ~~~
>>>>
>>>> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
...

>>> Another thing to consider is that these locations are only defined on
>>> GICv2 and not GICv1, so this patch is likely to cause trouble on older HW.
>>
>> As presented the code relies on the RAZ/WI property of reserved
>> registers to avoid issues on GICv1; it does not report anything if there
>> appear to be know working SGIs on the assumption we are actually running
>> on a GICv1.
>>
>> You'd prefer an explicit version check?
>
> I'd rather be cautious and check for the architecture version,
> specially if you settle for the byte access mentioned above (a GICv1
> may not support byte access and explode unexpectedly). ICPIDR2.ArchRev
> should be the right thing to check.

Will do.


>>>> +
>>>> +			/* record original value */
>>>> +			pending = readl_relaxed(set_reg);
>>>> +
>>>> +			/* clear, test, set, and test again */
>>>> +			writel_relaxed(mask, clear_reg);
>>>> +			after_clear = readl_relaxed(set_reg);
>>>> +			writel_relaxed(mask, set_reg);
>>>> +			after_set = readl_relaxed(set_reg);
>>>
>>> It should be enough to write to the SET register, and read back, as the
>>> bit is RAZ/WI when the interrupt is Group-0.
>>
>> Good point. Will simplify.
>
> I'd also suggest moving the whole thing to a separate function that'd
> get called from gic_cpu_init().

Will do.

I think I will also upgrade the pr_crit() to a WARN_ON() and hard code 
the check to 8 rather than NR_IPI.

Currently NR_IPI is small on arm64 so it would be good for us to shout 
loudly about latent firmware/secure-zone misconfiguration.


Daniel.