[PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Marc Zyngier]

For as long as SMP ARM has existed, IPIs have been handled as
something special. The arch code and the interrupt controller exchange
a couple of hooks (one to generate an IPI, another to handle it).

Although this is perfectly manageable, it prevents the use of features
that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
also means that each interrupt controller driver has to follow an
architecture-specific interface instead of just implementing the base
irqchip functionnalities. The arch code also duplicates a number of
things that the core irq code already does (such as calling
set_irq_regs(), irq_enter()...).

This series tries to remedy this on arm/arm64 by offering a new
registration interface where the irqchip gives the arch code a range
of interrupts to use for IPIs. The arch code requests these as normal
interrupts.

The bulk of the work is at the interrupt controller level, where all 3
irqchips used on arm64 get converted.

Finally, the arm64 code drops the legacy registration interface. The
same thing could be done on 32bit as well once the two remaining
irqchips using that interface get converted.

There is probably more that could be done: statistics are still
architecture-private code, for example, and no attempt is made to
solve that (apart from hidding the IRQs from /proc/interrupt).

This has been tested on a bunch of 32 and 64bit guests.

Marc Zyngier (11):
  genirq: Add fasteoi IPI flow
  genirq: Allow interrupts to be excluded from /proc/interrupts
  arm64: Allow IPIs to be handled as normal interrupts
  ARM: Allow IPIs to be handled as normal interrupts
  irqchip/gic-v3: Describe the SGI range
  irqchip/gic-v3: Configure SGIs as standard interrupts
  irqchip/gic: Refactor SMP configuration
  irqchip/gic: Configure SGIs as standard interrupts
  irqchip/gic-common: Don't enable SGIs by default
  irqchip/bcm2836: Configure mailbox interrupts as standard interrupts
  arm64: Kill __smp_cross_call and co

 arch/arm/Kconfig                  |   1 +
 arch/arm/include/asm/smp.h        |   5 +
 arch/arm/kernel/smp.c             |  97 +++++++++++---
 arch/arm64/Kconfig                |   1 +
 arch/arm64/include/asm/irq_work.h |   4 +-
 arch/arm64/include/asm/smp.h      |   6 +-
 arch/arm64/kernel/smp.c           |  98 +++++++++++----
 drivers/irqchip/irq-bcm2836.c     | 151 ++++++++++++++++++----
 drivers/irqchip/irq-gic-common.c  |   3 -
 drivers/irqchip/irq-gic-v3.c      | 109 ++++++++++------
 drivers/irqchip/irq-gic.c         | 203 ++++++++++++++++++------------
 include/linux/irq.h               |   4 +-
 kernel/irq/chip.c                 |  26 ++++
 kernel/irq/debugfs.c              |   1 +
 kernel/irq/proc.c                 |   2 +-
 kernel/irq/settings.h             |   7 ++
 16 files changed, 515 insertions(+), 203 deletions(-)

-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 01/11] genirq: Add fasteoi IPI flow

[Marc Zyngier]

For irqchips using the fasteoi flow, IPIs are a bit special.

They need to be EOId early (before calling the handler), as
funny things may happen in the handler (they do not necessarily
behave like a normal interrupt), and that the arch code is
already handling the stats.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 include/linux/irq.h |  1 +
 kernel/irq/chip.c   | 26 ++++++++++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index 8d5bc2c237d7..726f94d8b8cc 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -621,6 +621,7 @@ static inline int irq_set_parent(int irq, int parent_irq)
  */
 extern void handle_level_irq(struct irq_desc *desc);
 extern void handle_fasteoi_irq(struct irq_desc *desc);
+extern void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc);
 extern void handle_edge_irq(struct irq_desc *desc);
 extern void handle_edge_eoi_irq(struct irq_desc *desc);
 extern void handle_simple_irq(struct irq_desc *desc);
diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index 41e7e37a0928..7b0b789cfed4 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -955,6 +955,32 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
 		chip->irq_eoi(&desc->irq_data);
 }
 
+/**
+ * handle_percpu_devid_fasteoi_ipi - Per CPU local IPI handler with per cpu
+ *				     dev ids
+ * @desc:	the interrupt description structure for this irq
+ *
+ * The biggest differences with the IRQ version are that:
+ * - the interrupt is EOIed early, as the IPI could result in a context
+ *   switch, and we need to make sure the IPI can fire again
+ * - Stats are usually handled at the architecture level, so we ignore them
+ *   here
+ */
+void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc)
+{
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	struct irqaction *action = desc->action;
+	unsigned int irq = irq_desc_get_irq(desc);
+	irqreturn_t res;
+
+	if (chip->irq_eoi)
+		chip->irq_eoi(&desc->irq_data);
+
+	trace_irq_handler_entry(irq, action);
+	res = action->handler(irq, raw_cpu_ptr(action->percpu_dev_id));
+	trace_irq_handler_exit(irq, action, res);
+}
+
 /**
  * handle_percpu_devid_fasteoi_nmi - Per CPU local NMI handler with per cpu
  *				     dev ids
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/11] genirq: Add fasteoi IPI flow

[Florian Fainelli]

On 5/19/2020 9:17 AM, Marc Zyngier wrote:
> For irqchips using the fasteoi flow, IPIs are a bit special.
> 
> They need to be EOId early (before calling the handler), as
> funny things may happen in the handler (they do not necessarily
> behave like a normal interrupt), and that the arch code is
> already handling the stats.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  include/linux/irq.h |  1 +
>  kernel/irq/chip.c   | 26 ++++++++++++++++++++++++++
>  2 files changed, 27 insertions(+)
> 
> diff --git a/include/linux/irq.h b/include/linux/irq.h
> index 8d5bc2c237d7..726f94d8b8cc 100644
> --- a/include/linux/irq.h
> +++ b/include/linux/irq.h
> @@ -621,6 +621,7 @@ static inline int irq_set_parent(int irq, int parent_irq)
>   */
>  extern void handle_level_irq(struct irq_desc *desc);
>  extern void handle_fasteoi_irq(struct irq_desc *desc);
> +extern void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc);
>  extern void handle_edge_irq(struct irq_desc *desc);
>  extern void handle_edge_eoi_irq(struct irq_desc *desc);
>  extern void handle_simple_irq(struct irq_desc *desc);
> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> index 41e7e37a0928..7b0b789cfed4 100644
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -955,6 +955,32 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
>  		chip->irq_eoi(&desc->irq_data);
>  }
>  
> +/**
> + * handle_percpu_devid_fasteoi_ipi - Per CPU local IPI handler with per cpu
> + *				     dev ids
> + * @desc:	the interrupt description structure for this irq
> + *
> + * The biggest differences with the IRQ version are that:
> + * - the interrupt is EOIed early, as the IPI could result in a context
> + *   switch, and we need to make sure the IPI can fire again
> + * - Stats are usually handled at the architecture level, so we ignore them
> + *   here
> + */
> +void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc)
> +{
> +	struct irq_chip *chip = irq_desc_get_chip(desc);
> +	struct irqaction *action = desc->action;
> +	unsigned int irq = irq_desc_get_irq(desc);
> +	irqreturn_t res;

Should not this have a:

	if (!irq_settings_is_no_accounting(desc))
		__kstat_incr_irqs_this_cpu(desc);

here in case you are using that handler with a SGI interrupt which is
not used as an IPI?

> +
> +	if (chip->irq_eoi)
> +		chip->irq_eoi(&desc->irq_data);
> +
> +	trace_irq_handler_entry(irq, action);
> +	res = action->handler(irq, raw_cpu_ptr(action->percpu_dev_id));
> +	trace_irq_handler_exit(irq, action, res);
> +}
> +
>  /**
>   * handle_percpu_devid_fasteoi_nmi - Per CPU local NMI handler with per cpu
>   *				     dev ids
> 

-- 
Florian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/11] genirq: Add fasteoi IPI flow

[Marc Zyngier]

On Tue, 19 May 2020 12:47:24 -0700
Florian Fainelli <f.fainelli@gmail.com> wrote:

> On 5/19/2020 9:17 AM, Marc Zyngier wrote:
> > For irqchips using the fasteoi flow, IPIs are a bit special.
> > 
> > They need to be EOId early (before calling the handler), as
> > funny things may happen in the handler (they do not necessarily
> > behave like a normal interrupt), and that the arch code is
> > already handling the stats.
> > 
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  include/linux/irq.h |  1 +
> >  kernel/irq/chip.c   | 26 ++++++++++++++++++++++++++
> >  2 files changed, 27 insertions(+)
> > 
> > diff --git a/include/linux/irq.h b/include/linux/irq.h
> > index 8d5bc2c237d7..726f94d8b8cc 100644
> > --- a/include/linux/irq.h
> > +++ b/include/linux/irq.h
> > @@ -621,6 +621,7 @@ static inline int irq_set_parent(int irq, int parent_irq)
> >   */
> >  extern void handle_level_irq(struct irq_desc *desc);
> >  extern void handle_fasteoi_irq(struct irq_desc *desc);
> > +extern void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc);
> >  extern void handle_edge_irq(struct irq_desc *desc);
> >  extern void handle_edge_eoi_irq(struct irq_desc *desc);
> >  extern void handle_simple_irq(struct irq_desc *desc);
> > diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> > index 41e7e37a0928..7b0b789cfed4 100644
> > --- a/kernel/irq/chip.c
> > +++ b/kernel/irq/chip.c
> > @@ -955,6 +955,32 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
> >  		chip->irq_eoi(&desc->irq_data);
> >  }
> >  
> > +/**
> > + * handle_percpu_devid_fasteoi_ipi - Per CPU local IPI handler with per cpu
> > + *				     dev ids
> > + * @desc:	the interrupt description structure for this irq
> > + *
> > + * The biggest differences with the IRQ version are that:
> > + * - the interrupt is EOIed early, as the IPI could result in a context
> > + *   switch, and we need to make sure the IPI can fire again
> > + * - Stats are usually handled at the architecture level, so we ignore them
> > + *   here
> > + */
> > +void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc)
> > +{
> > +	struct irq_chip *chip = irq_desc_get_chip(desc);
> > +	struct irqaction *action = desc->action;
> > +	unsigned int irq = irq_desc_get_irq(desc);
> > +	irqreturn_t res;  
> 
> Should not this have a:
> 
> 	if (!irq_settings_is_no_accounting(desc))
> 		__kstat_incr_irqs_this_cpu(desc);
> 
> here in case you are using that handler with a SGI interrupt which is
> not used as an IPI?

I see you coming... ;-)

Yes, could do. TBH, I plan on killing the "no accounting" altogether
and move to the core IRQ one, so this would only be temporary. All I
need is to update the rest of the arm32 interrupt controllers...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/11] genirq: Add fasteoi IPI flow

[Valentin Schneider]

On 19/05/20 17:17, Marc Zyngier wrote:
> For irqchips using the fasteoi flow, IPIs are a bit special.
>
> They need to be EOId early (before calling the handler), as
> funny things may happen in the handler (they do not necessarily
> behave like a normal interrupt), and that the arch code is
> already handling the stats.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  include/linux/irq.h |  1 +
>  kernel/irq/chip.c   | 26 ++++++++++++++++++++++++++
>  2 files changed, 27 insertions(+)
>
> diff --git a/include/linux/irq.h b/include/linux/irq.h
> index 8d5bc2c237d7..726f94d8b8cc 100644
> --- a/include/linux/irq.h
> +++ b/include/linux/irq.h
> @@ -621,6 +621,7 @@ static inline int irq_set_parent(int irq, int parent_irq)
>   */
>  extern void handle_level_irq(struct irq_desc *desc);
>  extern void handle_fasteoi_irq(struct irq_desc *desc);
> +extern void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc);
>  extern void handle_edge_irq(struct irq_desc *desc);
>  extern void handle_edge_eoi_irq(struct irq_desc *desc);
>  extern void handle_simple_irq(struct irq_desc *desc);
> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> index 41e7e37a0928..7b0b789cfed4 100644
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -955,6 +955,32 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
>               chip->irq_eoi(&desc->irq_data);
>  }
>
> +/**
> + * handle_percpu_devid_fasteoi_ipi - Per CPU local IPI handler with per cpu
> + *				     dev ids
> + * @desc:	the interrupt description structure for this irq
> + *
> + * The biggest differences with the IRQ version are that:
> + * - the interrupt is EOIed early, as the IPI could result in a context
> + *   switch, and we need to make sure the IPI can fire again
> + * - Stats are usually handled at the architecture level, so we ignore them
> + *   here
> + */
> +void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc)
> +{
> +	struct irq_chip *chip = irq_desc_get_chip(desc);
> +	struct irqaction *action = desc->action;
> +	unsigned int irq = irq_desc_get_irq(desc);
> +	irqreturn_t res;
> +

Tied to the following patch, does that want something like

+	if (!irq_settings_is_no_accounting(desc))
+		__kstat_incr_irqs_this_cpu(desc);
+

While I'm at it, now that we would have IPIs as 'normal' interrupts, what
prevents us from getting rid of the arch-side accounting? Is it just that
we are keeping it as long as handle_IPI() lives, or are there more hurdles
hidden around?

> +	if (chip->irq_eoi)
> +		chip->irq_eoi(&desc->irq_data);
> +
> +	trace_irq_handler_entry(irq, action);
> +	res = action->handler(irq, raw_cpu_ptr(action->percpu_dev_id));
> +	trace_irq_handler_exit(irq, action, res);
> +}
> +
>  /**
>   * handle_percpu_devid_fasteoi_nmi - Per CPU local NMI handler with per cpu
>   *				     dev ids

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/11] genirq: Add fasteoi IPI flow

[Valentin Schneider]

On 19/05/20 23:25, Valentin Schneider wrote:
> Tied to the following patch, does that want something like
>
> +	if (!irq_settings_is_no_accounting(desc))
> +		__kstat_incr_irqs_this_cpu(desc);
> +

And ofc after updating my inbox I see this has already been suggested, oh
well.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 01/11] genirq: Add fasteoi IPI flow

[Marc Zyngier]

On Tue, 19 May 2020 23:25:43 +0100
Valentin Schneider <valentin.schneider@arm.com> wrote:

> On 19/05/20 17:17, Marc Zyngier wrote:
> > For irqchips using the fasteoi flow, IPIs are a bit special.
> >
> > They need to be EOId early (before calling the handler), as
> > funny things may happen in the handler (they do not necessarily
> > behave like a normal interrupt), and that the arch code is
> > already handling the stats.
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  include/linux/irq.h |  1 +
> >  kernel/irq/chip.c   | 26 ++++++++++++++++++++++++++
> >  2 files changed, 27 insertions(+)
> >
> > diff --git a/include/linux/irq.h b/include/linux/irq.h
> > index 8d5bc2c237d7..726f94d8b8cc 100644
> > --- a/include/linux/irq.h
> > +++ b/include/linux/irq.h
> > @@ -621,6 +621,7 @@ static inline int irq_set_parent(int irq, int parent_irq)
> >   */
> >  extern void handle_level_irq(struct irq_desc *desc);
> >  extern void handle_fasteoi_irq(struct irq_desc *desc);
> > +extern void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc);
> >  extern void handle_edge_irq(struct irq_desc *desc);
> >  extern void handle_edge_eoi_irq(struct irq_desc *desc);
> >  extern void handle_simple_irq(struct irq_desc *desc);
> > diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> > index 41e7e37a0928..7b0b789cfed4 100644
> > --- a/kernel/irq/chip.c
> > +++ b/kernel/irq/chip.c
> > @@ -955,6 +955,32 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
> >               chip->irq_eoi(&desc->irq_data);
> >  }
> >
> > +/**
> > + * handle_percpu_devid_fasteoi_ipi - Per CPU local IPI handler with per cpu
> > + *				     dev ids
> > + * @desc:	the interrupt description structure for this irq
> > + *
> > + * The biggest differences with the IRQ version are that:
> > + * - the interrupt is EOIed early, as the IPI could result in a context
> > + *   switch, and we need to make sure the IPI can fire again
> > + * - Stats are usually handled at the architecture level, so we ignore them
> > + *   here
> > + */
> > +void handle_percpu_devid_fasteoi_ipi(struct irq_desc *desc)
> > +{
> > +	struct irq_chip *chip = irq_desc_get_chip(desc);
> > +	struct irqaction *action = desc->action;
> > +	unsigned int irq = irq_desc_get_irq(desc);
> > +	irqreturn_t res;
> > +  
> 
> Tied to the following patch, does that want something like
> 
> +	if (!irq_settings_is_no_accounting(desc))
> +		__kstat_incr_irqs_this_cpu(desc);
> +
> 
> While I'm at it, now that we would have IPIs as 'normal' interrupts, what
> prevents us from getting rid of the arch-side accounting? Is it just that
> we are keeping it as long as handle_IPI() lives, or are there more hurdles
> hidden around?

See my reply to Florian. I need to mop the rest of the 32bit irqchips
without having the HW (hip04 is almost a copy of the GIC driver, and
the Marvell horror is very RPi like).

Once this is done, we can kill the home-brewed stuff and rely on core
infrastructure.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 02/11] genirq: Allow interrupts to be excluded from /proc/interrupts

[Marc Zyngier]

A number of architectures implement IPI statistics directly,
bypassing the core kstat_irqs accounting. As we move IPIs to
being actual IRQs, we would end-up with a confusing display
in /proc/interrupts (where the IPIs would appear twice).

In order to solve this, allow interrupts to be flagged as
"no accounting", which excludes them from /proc/interrupts.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 include/linux/irq.h   | 3 ++-
 kernel/irq/debugfs.c  | 1 +
 kernel/irq/proc.c     | 2 +-
 kernel/irq/settings.h | 7 +++++++
 4 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index 726f94d8b8cc..4433c8e765c8 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -97,13 +97,14 @@ enum {
 	IRQ_PER_CPU_DEVID	= (1 << 17),
 	IRQ_IS_POLLED		= (1 << 18),
 	IRQ_DISABLE_UNLAZY	= (1 << 19),
+	IRQ_NO_ACCOUNTING	= (1 << 20),
 };
 
 #define IRQF_MODIFY_MASK	\
 	(IRQ_TYPE_SENSE_MASK | IRQ_NOPROBE | IRQ_NOREQUEST | \
 	 IRQ_NOAUTOEN | IRQ_MOVE_PCNTXT | IRQ_LEVEL | IRQ_NO_BALANCING | \
 	 IRQ_PER_CPU | IRQ_NESTED_THREAD | IRQ_NOTHREAD | IRQ_PER_CPU_DEVID | \
-	 IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY)
+	 IRQ_IS_POLLED | IRQ_DISABLE_UNLAZY | IRQ_NO_ACCOUNTING)
 
 #define IRQ_NO_BALANCING_MASK	(IRQ_PER_CPU | IRQ_NO_BALANCING)
 
diff --git a/kernel/irq/debugfs.c b/kernel/irq/debugfs.c
index 4f9f844074db..9a22eab5537f 100644
--- a/kernel/irq/debugfs.c
+++ b/kernel/irq/debugfs.c
@@ -131,6 +131,7 @@ static const struct irq_bit_descr irqdesc_states[] = {
 	BIT_MASK_DESCR(_IRQ_PER_CPU_DEVID),
 	BIT_MASK_DESCR(_IRQ_IS_POLLED),
 	BIT_MASK_DESCR(_IRQ_DISABLE_UNLAZY),
+	BIT_MASK_DESCR(_IRQ_NO_ACCOUNTING),
 };
 
 static const struct irq_bit_descr irqdesc_istates[] = {
diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 32c071d7bc03..87780af13c12 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -485,7 +485,7 @@ int show_interrupts(struct seq_file *p, void *v)
 
 	rcu_read_lock();
 	desc = irq_to_desc(i);
-	if (!desc)
+	if (!desc || irq_settings_is_no_accounting(desc))
 		goto outsparse;
 
 	if (desc->kstat_irqs)
diff --git a/kernel/irq/settings.h b/kernel/irq/settings.h
index e43795cd2ccf..732fc3f2c686 100644
--- a/kernel/irq/settings.h
+++ b/kernel/irq/settings.h
@@ -17,6 +17,7 @@ enum {
 	_IRQ_PER_CPU_DEVID	= IRQ_PER_CPU_DEVID,
 	_IRQ_IS_POLLED		= IRQ_IS_POLLED,
 	_IRQ_DISABLE_UNLAZY	= IRQ_DISABLE_UNLAZY,
+	_IRQ_NO_ACCOUNTING	= IRQ_NO_ACCOUNTING,
 	_IRQF_MODIFY_MASK	= IRQF_MODIFY_MASK,
 };
 
@@ -31,6 +32,7 @@ enum {
 #define IRQ_PER_CPU_DEVID	GOT_YOU_MORON
 #define IRQ_IS_POLLED		GOT_YOU_MORON
 #define IRQ_DISABLE_UNLAZY	GOT_YOU_MORON
+#define IRQ_NO_ACCOUNTING	GOT_YOU_MORON
 #undef IRQF_MODIFY_MASK
 #define IRQF_MODIFY_MASK	GOT_YOU_MORON
 
@@ -167,3 +169,8 @@ static inline void irq_settings_clr_disable_unlazy(struct irq_desc *desc)
 {
 	desc->status_use_accessors &= ~_IRQ_DISABLE_UNLAZY;
 }
+
+static inline bool irq_settings_is_no_accounting(struct irq_desc *desc)
+{
+	return desc->status_use_accessors & _IRQ_NO_ACCOUNTING;
+}
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 03/11] arm64: Allow IPIs to be handled as normal interrupts

[Marc Zyngier]

In order to deal with IPIs as normal interrupts, let's add
a new way to register them with the architecture code.

set_smp_ipi_range() takes a range of interrupts, and allows
the arch code to request them as if the were normal interrupts.
A standard handler is then called by the core IRQ code to deal
with the IPI.

This means that we don't need to call irq_enter/irq_exit, and
that we don't need to deal with set_irq_regs either. So let's
move the dispatcher into its own function, and leave handle_IPI()
as a compatibility function.

On the sending side, let's make use of ipi_send_mask, which
already exists for this purpose.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/smp.h |  5 ++
 arch/arm64/kernel/smp.c      | 92 +++++++++++++++++++++++++++++++-----
 3 files changed, 86 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 5d513f461957..6a6271281016 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -99,6 +99,7 @@ config ARM64
 	select GENERIC_CPU_VULNERABILITIES
 	select GENERIC_EARLY_IOREMAP
 	select GENERIC_IDLE_POLL_SETUP
+	select GENERIC_IRQ_IPI
 	select GENERIC_IRQ_MULTI_HANDLER
 	select GENERIC_IRQ_PROBE
 	select GENERIC_IRQ_SHOW
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 40d5ba029615..42f366ba88bf 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -82,6 +82,11 @@ extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
 
 extern void (*__smp_cross_call)(const struct cpumask *, unsigned int);
 
+/*
+ * Register IPI interrupts with the arch SMP code
+ */
+extern void set_smp_ipi_range(int ipi_base, int nr_ipi);
+
 /*
  * Called from the secondary holding pen, this is the secondary CPU entry point.
  */
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 061f60fe452f..93ba0025e7b9 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -67,6 +67,13 @@ struct secondary_data secondary_data;
 /* Number of CPUs which aren't online, but looping in kernel text. */
 int cpus_stuck_in_kernel;
 
+static int ipi_irq_base;
+static int nr_ipi = NR_IPI;
+static struct irq_desc *ipi_desc[NR_IPI];
+
+static void ipi_setup(int cpu);
+static void ipi_teardown(int cpu);
+
 enum ipi_msg_type {
 	IPI_RESCHEDULE,
 	IPI_CALL_FUNC,
@@ -247,6 +254,8 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 */
 	notify_cpu_starting(cpu);
 
+	ipi_setup(cpu);
+
 	store_cpu_topology(cpu);
 	numa_add_cpu(cpu);
 
@@ -374,6 +383,8 @@ void cpu_die(void)
 
 	local_daif_mask();
 
+	ipi_teardown(cpu);
+
 	/* Tell __cpu_die() that this CPU is now safe to dispose of */
 	(void)cpu_report_death();
 
@@ -900,10 +911,9 @@ static void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
 /*
  * Main handler for inter-processor interrupts
  */
-void handle_IPI(int ipinr, struct pt_regs *regs)
+static void do_handle_IPI(int ipinr)
 {
 	unsigned int cpu = smp_processor_id();
-	struct pt_regs *old_regs = set_irq_regs(regs);
 
 	if ((unsigned)ipinr < NR_IPI) {
 		trace_ipi_entry_rcuidle(ipi_types[ipinr]);
@@ -916,21 +926,16 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 		break;
 
 	case IPI_CALL_FUNC:
-		irq_enter();
 		generic_smp_call_function_interrupt();
-		irq_exit();
 		break;
 
 	case IPI_CPU_STOP:
-		irq_enter();
 		local_cpu_stop();
-		irq_exit();
 		break;
 
 	case IPI_CPU_CRASH_STOP:
 		if (IS_ENABLED(CONFIG_KEXEC_CORE)) {
-			irq_enter();
-			ipi_cpu_crash_stop(cpu, regs);
+			ipi_cpu_crash_stop(cpu, get_irq_regs());
 
 			unreachable();
 		}
@@ -938,17 +943,13 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 
 #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST
 	case IPI_TIMER:
-		irq_enter();
 		tick_receive_broadcast();
-		irq_exit();
 		break;
 #endif
 
 #ifdef CONFIG_IRQ_WORK
 	case IPI_IRQ_WORK:
-		irq_enter();
 		irq_work_run();
-		irq_exit();
 		break;
 #endif
 
@@ -967,9 +968,76 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 
 	if ((unsigned)ipinr < NR_IPI)
 		trace_ipi_exit_rcuidle(ipi_types[ipinr]);
+}
+
+/* Legacy version, should go away once all irqchips have been converted */
+void handle_IPI(int ipinr, struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	irq_enter();
+	do_handle_IPI(ipinr);
+	irq_exit();
+
 	set_irq_regs(old_regs);
 }
 
+static irqreturn_t ipi_handler(int irq, void *data)
+{
+	do_handle_IPI(irq - ipi_irq_base);
+	return IRQ_HANDLED;
+}
+
+static void ipi_send(const struct cpumask *target, unsigned int ipi)
+{
+	__ipi_send_mask(ipi_desc[ipi], target);
+}
+
+static void ipi_setup(int cpu)
+{
+	if (ipi_irq_base) {
+		int i;
+
+		for (i = 0; i < nr_ipi; i++)
+			enable_percpu_irq(ipi_irq_base + i, 0);
+	}
+}
+
+static void ipi_teardown(int cpu)
+{
+	if (ipi_irq_base) {
+		int i;
+
+		for (i = 0; i < nr_ipi; i++)
+			disable_percpu_irq(ipi_irq_base + i);
+	}
+}
+
+void __init set_smp_ipi_range(int ipi_base, int n)
+{
+	int i;
+
+	WARN_ON(n < NR_IPI);
+	nr_ipi = min(n, NR_IPI);
+
+	for (i = 0; i < nr_ipi; i++) {
+		int err;
+
+		err = request_percpu_irq(ipi_base + i, ipi_handler,
+					 "IPI", &irq_stat);
+		WARN_ON(err);
+
+		ipi_desc[i] = irq_to_desc(ipi_base + i);
+		irq_set_status_flags(ipi_base + i, IRQ_NO_ACCOUNTING);
+	}
+
+	ipi_irq_base = ipi_base;
+	__smp_cross_call = ipi_send;
+
+	/* Setup the boot CPU immediately */
+	ipi_setup(smp_processor_id());
+}
+
 void smp_send_reschedule(int cpu)
 {
 	smp_cross_call(cpumask_of(cpu), IPI_RESCHEDULE);
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 03/11] arm64: Allow IPIs to be handled as normal interrupts

[Valentin Schneider]

On 19/05/20 17:17, Marc Zyngier wrote:
> In order to deal with IPIs as normal interrupts, let's add
> a new way to register them with the architecture code.
>
> set_smp_ipi_range() takes a range of interrupts, and allows
> the arch code to request them as if the were normal interrupts.
                                      ^^^
                                  s/the/they/

> A standard handler is then called by the core IRQ code to deal
> with the IPI.
>
> This means that we don't need to call irq_enter/irq_exit, and
> that we don't need to deal with set_irq_regs either. So let's
> move the dispatcher into its own function, and leave handle_IPI()
> as a compatibility function.
>
> On the sending side, let's make use of ipi_send_mask, which
> already exists for this purpose.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/Kconfig           |  1 +
>  arch/arm64/include/asm/smp.h |  5 ++
>  arch/arm64/kernel/smp.c      | 92 +++++++++++++++++++++++++++++++-----
>  3 files changed, 86 insertions(+), 12 deletions(-)
>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index 061f60fe452f..93ba0025e7b9 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -247,6 +254,8 @@ asmlinkage notrace void secondary_start_kernel(void)
>        */
>       notify_cpu_starting(cpu);
>
> +	ipi_setup(cpu);
> +
>       store_cpu_topology(cpu);
>       numa_add_cpu(cpu);
>
> @@ -374,6 +383,8 @@ void cpu_die(void)
>
>       local_daif_mask();
>
> +	ipi_teardown(cpu);
> +

Would it make sense to move it up to say __cpu_disable()? I'm thinking it
would make sense to bunch this up with the toggling of the cpu_online_mask
bit, and FWIW it'd match with the comment atop the cpuhp callsite.

Once the CPU is set as offline, all it has left to do is to go die in
do_idle(), so AFAICT we can do that IPI teardown anywhere inbetween.

>       /* Tell __cpu_die() that this CPU is now safe to dispose of */
>       (void)cpu_report_death();
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 04/11] ARM: Allow IPIs to be handled as normal interrupts

[Marc Zyngier]

In order to deal with IPIs as normal interrupts, let's add
a new way to register them with the architecture code.

set_smp_ipi_range() takes a range of interrupts, and allows
the arch code to request them as if the were normal interrupts.
A standard handler is then called by the core IRQ code to deal
with the IPI.

This means that we don't need to call irq_enter/irq_exit, and
that we don't need to deal with set_irq_regs either. So let's
move the dispatcher into its own function, and leave handle_IPI()
as a compatibility function.

On the sending side, let's make use of ipi_send_mask, which
already exists for this purpose.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm/Kconfig           |  1 +
 arch/arm/include/asm/smp.h |  5 ++
 arch/arm/kernel/smp.c      | 97 ++++++++++++++++++++++++++++++++------
 3 files changed, 88 insertions(+), 15 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index c77c93c485a0..0caaba9bf880 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -48,6 +48,7 @@ config ARM
 	select GENERIC_ARCH_TOPOLOGY if ARM_CPU_TOPOLOGY
 	select GENERIC_ATOMIC64 if CPU_V7M || CPU_V6 || !CPU_32v6K || !AEABI
 	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
+	select GENERIC_IRQ_IPI if SMP
 	select GENERIC_CPU_AUTOPROBE
 	select GENERIC_EARLY_IOREMAP
 	select GENERIC_IDLE_POLL_SETUP
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index a91f21e3c5b5..0e29730295ca 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -45,6 +45,11 @@ extern void smp_init_cpus(void);
  */
 extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
 
+/*
+ * Register IPI interrupts with the arch SMP code
+ */
+extern void set_smp_ipi_range(int ipi_base, int nr_ipi);
+
 /*
  * Called from platform specific assembly code, this is the
  * secondary CPU entry point.
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 46e1be9e57a8..618641978a5b 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -79,10 +79,19 @@ enum ipi_msg_type {
 	 */
 };
 
+static int ipi_irq_base;
+static int nr_ipi = NR_IPI;
+static struct irq_desc *ipi_desc[NR_IPI];
+
+static void ipi_setup(int cpu);
+static void ipi_teardown(int cpu);
+
 static DECLARE_COMPLETION(cpu_running);
 
 static struct smp_operations smp_ops __ro_after_init;
 
+static void ipi_setup(int cpu);
+
 void __init smp_set_ops(const struct smp_operations *ops)
 {
 	if (ops)
@@ -308,6 +317,8 @@ void arch_cpu_idle_dead(void)
 
 	local_irq_disable();
 
+	ipi_teardown(cpu);
+
 	/*
 	 * Flush the data out of the L1 cache for this CPU.  This must be
 	 * before the completion to ensure that data is safely written out
@@ -424,6 +435,8 @@ asmlinkage void secondary_start_kernel(void)
 
 	notify_cpu_starting(cpu);
 
+	ipi_setup(cpu);
+
 	calibrate_delay();
 
 	smp_store_cpu_info(cpu);
@@ -629,10 +642,9 @@ asmlinkage void __exception_irq_entry do_IPI(int ipinr, struct pt_regs *regs)
 	handle_IPI(ipinr, regs);
 }
 
-void handle_IPI(int ipinr, struct pt_regs *regs)
+static void do_handle_IPI(int ipinr)
 {
 	unsigned int cpu = smp_processor_id();
-	struct pt_regs *old_regs = set_irq_regs(regs);
 
 	if ((unsigned)ipinr < NR_IPI) {
 		trace_ipi_entry_rcuidle(ipi_types[ipinr]);
@@ -645,9 +657,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 
 #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST
 	case IPI_TIMER:
-		irq_enter();
 		tick_receive_broadcast();
-		irq_exit();
 		break;
 #endif
 
@@ -656,36 +666,26 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 		break;
 
 	case IPI_CALL_FUNC:
-		irq_enter();
 		generic_smp_call_function_interrupt();
-		irq_exit();
 		break;
 
 	case IPI_CPU_STOP:
-		irq_enter();
 		ipi_cpu_stop(cpu);
-		irq_exit();
 		break;
 
 #ifdef CONFIG_IRQ_WORK
 	case IPI_IRQ_WORK:
-		irq_enter();
 		irq_work_run();
-		irq_exit();
 		break;
 #endif
 
 	case IPI_COMPLETION:
-		irq_enter();
 		ipi_complete(cpu);
-		irq_exit();
 		break;
 
 	case IPI_CPU_BACKTRACE:
 		printk_nmi_enter();
-		irq_enter();
-		nmi_cpu_backtrace(regs);
-		irq_exit();
+		nmi_cpu_backtrace(get_irq_regs());
 		printk_nmi_exit();
 		break;
 
@@ -697,9 +697,76 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
 
 	if ((unsigned)ipinr < NR_IPI)
 		trace_ipi_exit_rcuidle(ipi_types[ipinr]);
+}
+
+/* Legacy version, should go away once all irqchips have been converted */
+void handle_IPI(int ipinr, struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	irq_enter();
+	do_handle_IPI(ipinr);
+	irq_exit();
+
 	set_irq_regs(old_regs);
 }
 
+static irqreturn_t ipi_handler(int irq, void *data)
+{
+	do_handle_IPI(irq - ipi_irq_base);
+	return IRQ_HANDLED;
+}
+
+static void ipi_send(const struct cpumask *target, unsigned int ipi)
+{
+	__ipi_send_mask(ipi_desc[ipi], target);
+}
+
+static void ipi_setup(int cpu)
+{
+	if (ipi_irq_base) {
+		int i;
+
+		for (i = 0; i < nr_ipi; i++)
+			enable_percpu_irq(ipi_irq_base + i, 0);
+	}
+}
+
+static void ipi_teardown(int cpu)
+{
+	if (ipi_irq_base) {
+		int i;
+
+		for (i = 0; i < nr_ipi; i++)
+			disable_percpu_irq(ipi_irq_base + i);
+	}
+}
+
+void __init set_smp_ipi_range(int ipi_base, int n)
+{
+	int i;
+
+	WARN_ON(n < NR_IPI);
+	nr_ipi = min(n, NR_IPI);
+
+	for (i = 0; i < nr_ipi; i++) {
+		int err;
+
+		err = request_percpu_irq(ipi_base + i, ipi_handler,
+					 "IPI", &irq_stat);
+		WARN_ON(err);
+
+		ipi_desc[i] = irq_to_desc(ipi_base + i);
+		irq_set_status_flags(ipi_base + i, IRQ_NO_ACCOUNTING);
+	}
+
+	ipi_irq_base = ipi_base;
+	set_smp_cross_call(ipi_send);
+
+	/* Setup the boot CPU immediately */
+	ipi_setup(smp_processor_id());
+}
+
 void smp_send_reschedule(int cpu)
 {
 	smp_cross_call(cpumask_of(cpu), IPI_RESCHEDULE);
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 04/11] ARM: Allow IPIs to be handled as normal interrupts

[Russell King - ARM Linux admin]

On Tue, May 19, 2020 at 05:17:48PM +0100, Marc Zyngier wrote:
> In order to deal with IPIs as normal interrupts, let's add
> a new way to register them with the architecture code.
> 
> set_smp_ipi_range() takes a range of interrupts, and allows
> the arch code to request them as if the were normal interrupts.
> A standard handler is then called by the core IRQ code to deal
> with the IPI.
> 
> This means that we don't need to call irq_enter/irq_exit, and
> that we don't need to deal with set_irq_regs either. So let's
> move the dispatcher into its own function, and leave handle_IPI()
> as a compatibility function.
> 
> On the sending side, let's make use of ipi_send_mask, which
> already exists for this purpose.

You say nothing about the nesting of irq_enter() and irq_exit()
for scheduler_ipi().

Given that lockdep introduced the requirement that hard IRQs can't
be nested, are we sure that calling irq_exit() twice is safe?

Looking at irqtime_account_irq(), it seems that will cause double-
accounting of in-interrupt time, since we will increment
irq_start_time by just over twice the the period spent handling
the IPI.

I think the rest of irq_exit() should be safe, but still, this
behaviour should be documented at the very least, if not avoided.

> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm/Kconfig           |  1 +
>  arch/arm/include/asm/smp.h |  5 ++
>  arch/arm/kernel/smp.c      | 97 ++++++++++++++++++++++++++++++++------
>  3 files changed, 88 insertions(+), 15 deletions(-)
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index c77c93c485a0..0caaba9bf880 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -48,6 +48,7 @@ config ARM
>  	select GENERIC_ARCH_TOPOLOGY if ARM_CPU_TOPOLOGY
>  	select GENERIC_ATOMIC64 if CPU_V7M || CPU_V6 || !CPU_32v6K || !AEABI
>  	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
> +	select GENERIC_IRQ_IPI if SMP
>  	select GENERIC_CPU_AUTOPROBE
>  	select GENERIC_EARLY_IOREMAP
>  	select GENERIC_IDLE_POLL_SETUP
> diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
> index a91f21e3c5b5..0e29730295ca 100644
> --- a/arch/arm/include/asm/smp.h
> +++ b/arch/arm/include/asm/smp.h
> @@ -45,6 +45,11 @@ extern void smp_init_cpus(void);
>   */
>  extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
>  
> +/*
> + * Register IPI interrupts with the arch SMP code
> + */
> +extern void set_smp_ipi_range(int ipi_base, int nr_ipi);
> +
>  /*
>   * Called from platform specific assembly code, this is the
>   * secondary CPU entry point.
> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> index 46e1be9e57a8..618641978a5b 100644
> --- a/arch/arm/kernel/smp.c
> +++ b/arch/arm/kernel/smp.c
> @@ -79,10 +79,19 @@ enum ipi_msg_type {
>  	 */
>  };
>  
> +static int ipi_irq_base;
> +static int nr_ipi = NR_IPI;
> +static struct irq_desc *ipi_desc[NR_IPI];
> +
> +static void ipi_setup(int cpu);
> +static void ipi_teardown(int cpu);
> +
>  static DECLARE_COMPLETION(cpu_running);
>  
>  static struct smp_operations smp_ops __ro_after_init;
>  
> +static void ipi_setup(int cpu);
> +
>  void __init smp_set_ops(const struct smp_operations *ops)
>  {
>  	if (ops)
> @@ -308,6 +317,8 @@ void arch_cpu_idle_dead(void)
>  
>  	local_irq_disable();
>  
> +	ipi_teardown(cpu);
> +
>  	/*
>  	 * Flush the data out of the L1 cache for this CPU.  This must be
>  	 * before the completion to ensure that data is safely written out
> @@ -424,6 +435,8 @@ asmlinkage void secondary_start_kernel(void)
>  
>  	notify_cpu_starting(cpu);
>  
> +	ipi_setup(cpu);
> +
>  	calibrate_delay();
>  
>  	smp_store_cpu_info(cpu);
> @@ -629,10 +642,9 @@ asmlinkage void __exception_irq_entry do_IPI(int ipinr, struct pt_regs *regs)
>  	handle_IPI(ipinr, regs);
>  }
>  
> -void handle_IPI(int ipinr, struct pt_regs *regs)
> +static void do_handle_IPI(int ipinr)
>  {
>  	unsigned int cpu = smp_processor_id();
> -	struct pt_regs *old_regs = set_irq_regs(regs);
>  
>  	if ((unsigned)ipinr < NR_IPI) {
>  		trace_ipi_entry_rcuidle(ipi_types[ipinr]);
> @@ -645,9 +657,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
>  
>  #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST
>  	case IPI_TIMER:
> -		irq_enter();
>  		tick_receive_broadcast();
> -		irq_exit();
>  		break;
>  #endif
>  
> @@ -656,36 +666,26 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
>  		break;
>  
>  	case IPI_CALL_FUNC:
> -		irq_enter();
>  		generic_smp_call_function_interrupt();
> -		irq_exit();
>  		break;
>  
>  	case IPI_CPU_STOP:
> -		irq_enter();
>  		ipi_cpu_stop(cpu);
> -		irq_exit();
>  		break;
>  
>  #ifdef CONFIG_IRQ_WORK
>  	case IPI_IRQ_WORK:
> -		irq_enter();
>  		irq_work_run();
> -		irq_exit();
>  		break;
>  #endif
>  
>  	case IPI_COMPLETION:
> -		irq_enter();
>  		ipi_complete(cpu);
> -		irq_exit();
>  		break;
>  
>  	case IPI_CPU_BACKTRACE:
>  		printk_nmi_enter();
> -		irq_enter();
> -		nmi_cpu_backtrace(regs);
> -		irq_exit();
> +		nmi_cpu_backtrace(get_irq_regs());
>  		printk_nmi_exit();
>  		break;
>  
> @@ -697,9 +697,76 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
>  
>  	if ((unsigned)ipinr < NR_IPI)
>  		trace_ipi_exit_rcuidle(ipi_types[ipinr]);
> +}
> +
> +/* Legacy version, should go away once all irqchips have been converted */
> +void handle_IPI(int ipinr, struct pt_regs *regs)
> +{
> +	struct pt_regs *old_regs = set_irq_regs(regs);
> +
> +	irq_enter();
> +	do_handle_IPI(ipinr);
> +	irq_exit();
> +
>  	set_irq_regs(old_regs);
>  }
>  
> +static irqreturn_t ipi_handler(int irq, void *data)
> +{
> +	do_handle_IPI(irq - ipi_irq_base);
> +	return IRQ_HANDLED;
> +}
> +
> +static void ipi_send(const struct cpumask *target, unsigned int ipi)
> +{
> +	__ipi_send_mask(ipi_desc[ipi], target);
> +}
> +
> +static void ipi_setup(int cpu)
> +{
> +	if (ipi_irq_base) {
> +		int i;
> +
> +		for (i = 0; i < nr_ipi; i++)
> +			enable_percpu_irq(ipi_irq_base + i, 0);
> +	}
> +}
> +
> +static void ipi_teardown(int cpu)
> +{
> +	if (ipi_irq_base) {
> +		int i;
> +
> +		for (i = 0; i < nr_ipi; i++)
> +			disable_percpu_irq(ipi_irq_base + i);
> +	}
> +}
> +
> +void __init set_smp_ipi_range(int ipi_base, int n)
> +{
> +	int i;
> +
> +	WARN_ON(n < NR_IPI);
> +	nr_ipi = min(n, NR_IPI);
> +
> +	for (i = 0; i < nr_ipi; i++) {
> +		int err;
> +
> +		err = request_percpu_irq(ipi_base + i, ipi_handler,
> +					 "IPI", &irq_stat);
> +		WARN_ON(err);
> +
> +		ipi_desc[i] = irq_to_desc(ipi_base + i);
> +		irq_set_status_flags(ipi_base + i, IRQ_NO_ACCOUNTING);
> +	}
> +
> +	ipi_irq_base = ipi_base;
> +	set_smp_cross_call(ipi_send);
> +
> +	/* Setup the boot CPU immediately */
> +	ipi_setup(smp_processor_id());
> +}
> +
>  void smp_send_reschedule(int cpu)
>  {
>  	smp_cross_call(cpumask_of(cpu), IPI_RESCHEDULE);
> -- 
> 2.26.2
> 
> 

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 04/11] ARM: Allow IPIs to be handled as normal interrupts

[Valentin Schneider]

On 19/05/20 23:24, Russell King - ARM Linux admin wrote:
> On Tue, May 19, 2020 at 05:17:48PM +0100, Marc Zyngier wrote:
>> In order to deal with IPIs as normal interrupts, let's add
>> a new way to register them with the architecture code.
>>
>> set_smp_ipi_range() takes a range of interrupts, and allows
>> the arch code to request them as if the were normal interrupts.
>> A standard handler is then called by the core IRQ code to deal
>> with the IPI.
>>
>> This means that we don't need to call irq_enter/irq_exit, and
>> that we don't need to deal with set_irq_regs either. So let's
>> move the dispatcher into its own function, and leave handle_IPI()
>> as a compatibility function.
>>
>> On the sending side, let's make use of ipi_send_mask, which
>> already exists for this purpose.
>
> You say nothing about the nesting of irq_enter() and irq_exit()
> for scheduler_ipi().
>
> Given that lockdep introduced the requirement that hard IRQs can't
> be nested, are we sure that calling irq_exit() twice is safe?
>
> Looking at irqtime_account_irq(), it seems that will cause double-
> accounting of in-interrupt time, since we will increment
> irq_start_time by just over twice the the period spent handling
> the IPI.
>
> I think the rest of irq_exit() should be safe, but still, this
> behaviour should be documented at the very least, if not avoided.
>

x86 does the same (though IIUC only when tracing reschedule IPI's), and
MIPS has the same issue as it also uses generic IRQ IPI's - so although
it's not ideal, I think we can live with it.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 04/11] ARM: Allow IPIs to be handled as normal interrupts

[Russell King - ARM Linux admin]

On Thu, May 21, 2020 at 03:03:49PM +0100, Valentin Schneider wrote:
> 
> On 19/05/20 23:24, Russell King - ARM Linux admin wrote:
> > On Tue, May 19, 2020 at 05:17:48PM +0100, Marc Zyngier wrote:
> >> In order to deal with IPIs as normal interrupts, let's add
> >> a new way to register them with the architecture code.
> >>
> >> set_smp_ipi_range() takes a range of interrupts, and allows
> >> the arch code to request them as if the were normal interrupts.
> >> A standard handler is then called by the core IRQ code to deal
> >> with the IPI.
> >>
> >> This means that we don't need to call irq_enter/irq_exit, and
> >> that we don't need to deal with set_irq_regs either. So let's
> >> move the dispatcher into its own function, and leave handle_IPI()
> >> as a compatibility function.
> >>
> >> On the sending side, let's make use of ipi_send_mask, which
> >> already exists for this purpose.
> >
> > You say nothing about the nesting of irq_enter() and irq_exit()
> > for scheduler_ipi().
> >
> > Given that lockdep introduced the requirement that hard IRQs can't
> > be nested, are we sure that calling irq_exit() twice is safe?
> >
> > Looking at irqtime_account_irq(), it seems that will cause double-
> > accounting of in-interrupt time, since we will increment
> > irq_start_time by just over twice the the period spent handling
> > the IPI.
> >
> > I think the rest of irq_exit() should be safe, but still, this
> > behaviour should be documented at the very least, if not avoided.
> >
> 
> x86 does the same (though IIUC only when tracing reschedule IPI's),

Right, so when the system is operating normally, then the accounting is
correct.  When the reschedule path is being explicitly traced, then
the accounting will be doubled for it.

What's being proposed for ARM is to always have this mis-accounting,
where no mis-accounting was present before - and some of us (me) /do/
enable IRQ accounting in our kernels as standard. So, you can take
this as a kernel regression report from a user.

> and MIPS has the same issue as it also uses generic IRQ IPI's - so
> although it's not ideal, I think we can live with it.

Yes, but is there anyone who cares about this for MIPS?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 04/11] ARM: Allow IPIs to be handled as normal interrupts

[Valentin Schneider]

On 21/05/20 16:12, Russell King - ARM Linux admin wrote:
> On Thu, May 21, 2020 at 03:03:49PM +0100, Valentin Schneider wrote:
>>
>> On 19/05/20 23:24, Russell King - ARM Linux admin wrote:
>> > On Tue, May 19, 2020 at 05:17:48PM +0100, Marc Zyngier wrote:
>> >> In order to deal with IPIs as normal interrupts, let's add
>> >> a new way to register them with the architecture code.
>> >>
>> >> set_smp_ipi_range() takes a range of interrupts, and allows
>> >> the arch code to request them as if the were normal interrupts.
>> >> A standard handler is then called by the core IRQ code to deal
>> >> with the IPI.
>> >>
>> >> This means that we don't need to call irq_enter/irq_exit, and
>> >> that we don't need to deal with set_irq_regs either. So let's
>> >> move the dispatcher into its own function, and leave handle_IPI()
>> >> as a compatibility function.
>> >>
>> >> On the sending side, let's make use of ipi_send_mask, which
>> >> already exists for this purpose.
>> >
>> > You say nothing about the nesting of irq_enter() and irq_exit()
>> > for scheduler_ipi().
>> >
>> > Given that lockdep introduced the requirement that hard IRQs can't
>> > be nested, are we sure that calling irq_exit() twice is safe?
>> >
>> > Looking at irqtime_account_irq(), it seems that will cause double-
>> > accounting of in-interrupt time, since we will increment
>> > irq_start_time by just over twice the the period spent handling
>> > the IPI.
>> >
>> > I think the rest of irq_exit() should be safe, but still, this
>> > behaviour should be documented at the very least, if not avoided.
>> >
>>
>> x86 does the same (though IIUC only when tracing reschedule IPI's),
>
> Right, so when the system is operating normally, then the accounting is
> correct.  When the reschedule path is being explicitly traced, then
> the accounting will be doubled for it.
>

Right, it's true that they are only affected when tracing.


That said, AFAICT the accounting nests correctly. Consider:

  irq_enter() @t0
    irq_enter() @t1
    ...
    irq_exit() @t2
  irq_exit() @t3

Entering irqtime_account_irq() at time t, we get something like:

  delta = t - irq_start_time;
  irq_start_time = t;

  if (hardirq_count())
          total += delta;

Since we go through the accounting on both irq_enter() and irq_exit(), we'd
have something like:

  irq_enter() @t0
    irq_start_time = t0

  irq_enter() @t1
    delta = t1 - t0
    irq_start_time = t1
    total += t1 - t0

  irq_exit() @t2
    delta = t2 - t1
    irq_start_time = t2
    total += t2 - t1

  irq_exit() @t3
    delta = t3 - t2
    irq_start_time = t3
    total += t3 - t2


So at the end we have incremented the total by

  t1-t0 + t2-t1 + t3-t2 = t3 - t0

IOW the duration of the outermost pair (... Unless I goofed up).

> What's being proposed for ARM is to always have this mis-accounting,
> where no mis-accounting was present before - and some of us (me) /do/
> enable IRQ accounting in our kernels as standard. So, you can take
> this as a kernel regression report from a user.
>
>> and MIPS has the same issue as it also uses generic IRQ IPI's - so
>> although it's not ideal, I think we can live with it.
>
> Yes, but is there anyone who cares about this for MIPS?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 05/11] irqchip/gic-v3: Describe the SGI range

[Marc Zyngier]

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index d7006ef18a0d..23d7c87da407 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -112,6 +112,7 @@ static DEFINE_PER_CPU(bool, has_rss);
 #define DEFAULT_PMR_VALUE	0xf0
 
 enum gic_intid_range {
+	SGI_RANGE,
 	PPI_RANGE,
 	SPI_RANGE,
 	EPPI_RANGE,
@@ -123,6 +124,8 @@ enum gic_intid_range {
 static enum gic_intid_range __get_intid_range(irq_hw_number_t hwirq)
 {
 	switch (hwirq) {
+	case 0 ... 15:
+		return SGI_RANGE;
 	case 16 ... 31:
 		return PPI_RANGE;
 	case 32 ... 1019:
@@ -148,15 +151,22 @@ static inline unsigned int gic_irq(struct irq_data *d)
 	return d->hwirq;
 }
 
-static inline int gic_irq_in_rdist(struct irq_data *d)
+static inline bool gic_irq_in_rdist(struct irq_data *d)
 {
-	enum gic_intid_range range = get_intid_range(d);
-	return range == PPI_RANGE || range == EPPI_RANGE;
+	switch (get_intid_range(d)) {
+	case SGI_RANGE:
+	case PPI_RANGE:
+	case EPPI_RANGE:
+		return true;
+	default:
+		return false;
+	}
 }
 
 static inline void __iomem *gic_dist_base(struct irq_data *d)
 {
 	switch (get_intid_range(d)) {
+	case SGI_RANGE:
 	case PPI_RANGE:
 	case EPPI_RANGE:
 		/* SGI+PPI -> SGI_base for this CPU */
@@ -253,6 +263,7 @@ static void gic_enable_redist(bool enable)
 static u32 convert_offset_index(struct irq_data *d, u32 offset, u32 *index)
 {
 	switch (get_intid_range(d)) {
+	case SGI_RANGE:
 	case PPI_RANGE:
 	case SPI_RANGE:
 		*index = d->hwirq;
@@ -1277,6 +1288,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 		chip = &gic_eoimode1_chip;
 
 	switch (__get_intid_range(hw)) {
+	case SGI_RANGE:
 	case PPI_RANGE:
 	case EPPI_RANGE:
 		irq_set_percpu_devid(irq);
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 06/11] irqchip/gic-v3: Configure SGIs as standard interrupts

[Marc Zyngier]

Change the way we deal with GICv3 SGIs by turning them into proper
IRQs, and calling into the arch code to register the interrupt range
instead of a callback.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3.c | 91 +++++++++++++++++++++---------------
 1 file changed, 53 insertions(+), 38 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 23d7c87da407..d57289057b75 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -36,6 +36,9 @@
 #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996	(1ULL << 0)
 #define FLAGS_WORKAROUND_CAVIUM_ERRATUM_38539	(1ULL << 1)
 
+#define GIC_IRQ_TYPE_PARTITION	(GIC_IRQ_TYPE_LPI + 1)
+#define GIC_IRQ_TYPE_SGI	(GIC_IRQ_TYPE_LPI + 2)
+
 struct redist_region {
 	void __iomem		*redist_base;
 	phys_addr_t		phys_base;
@@ -657,38 +660,14 @@ static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs
 	if ((irqnr >= 1020 && irqnr <= 1023))
 		return;
 
-	/* Treat anything but SGIs in a uniform way */
-	if (likely(irqnr > 15)) {
-		int err;
-
-		if (static_branch_likely(&supports_deactivate_key))
-			gic_write_eoir(irqnr);
-		else
-			isb();
-
-		err = handle_domain_irq(gic_data.domain, irqnr, regs);
-		if (err) {
-			WARN_ONCE(true, "Unexpected interrupt received!\n");
-			gic_deactivate_unhandled(irqnr);
-		}
-		return;
-	}
-	if (irqnr < 16) {
+	if (static_branch_likely(&supports_deactivate_key))
 		gic_write_eoir(irqnr);
-		if (static_branch_likely(&supports_deactivate_key))
-			gic_write_dir(irqnr);
-#ifdef CONFIG_SMP
-		/*
-		 * Unlike GICv2, we don't need an smp_rmb() here.
-		 * The control dependency from gic_read_iar to
-		 * the ISB in gic_write_eoir is enough to ensure
-		 * that any shared data read by handle_IPI will
-		 * be read after the ACK.
-		 */
-		handle_IPI(irqnr, regs);
-#else
-		WARN_ONCE(true, "Unexpected SGI received!\n");
-#endif
+	else
+		isb();
+
+	if (handle_domain_irq(gic_data.domain, irqnr, regs)) {
+		WARN_ONCE(true, "Unexpected interrupt received!\n");
+		gic_deactivate_unhandled(irqnr);
 	}
 }
 
@@ -1136,11 +1115,11 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq)
 	gic_write_sgi1r(val);
 }
 
-static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
+static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask)
 {
 	int cpu;
 
-	if (WARN_ON(irq >= 16))
+	if (WARN_ON(d->hwirq >= 16))
 		return;
 
 	/*
@@ -1154,7 +1133,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
 		u16 tlist;
 
 		tlist = gic_compute_target_list(&cpu, mask, cluster_id);
-		gic_send_sgi(cluster_id, tlist, irq);
+		gic_send_sgi(cluster_id, tlist, d->hwirq);
 	}
 
 	/* Force the above writes to ICC_SGI1R_EL1 to be executed */
@@ -1163,10 +1142,36 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
 
 static void gic_smp_init(void)
 {
-	set_smp_cross_call(gic_raise_softirq);
+	struct irq_fwspec sgi_fwspec = {
+		.fwnode		= gic_data.fwnode,
+	};
+	int base_sgi;
+
 	cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
 				  "irqchip/arm/gicv3:starting",
 				  gic_starting_cpu, NULL);
+
+	if (is_of_node(gic_data.fwnode)) {
+		/* DT */
+		sgi_fwspec.param_count = 3;
+		sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
+		sgi_fwspec.param[1] = 0;
+		sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
+	} else {
+		/* ACPI */
+		sgi_fwspec.param_count = 2;
+		sgi_fwspec.param[0] = 0;
+		sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
+	}
+
+	/* Register all 8 non-secure SGIs */
+	base_sgi = __irq_domain_alloc_irqs(gic_data.domain, -1, 8,
+					   NUMA_NO_NODE, &sgi_fwspec,
+					   false, NULL);
+	if (WARN_ON(base_sgi <= 0))
+		return;
+
+	set_smp_ipi_range(base_sgi, 8);
 }
 
 static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
@@ -1215,6 +1220,7 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
 }
 #else
 #define gic_set_affinity	NULL
+#define gic_ipi_send_mask	NULL
 #define gic_smp_init()		do { } while(0)
 #endif
 
@@ -1257,6 +1263,7 @@ static struct irq_chip gic_chip = {
 	.irq_set_irqchip_state	= gic_irq_set_irqchip_state,
 	.irq_nmi_setup		= gic_irq_nmi_setup,
 	.irq_nmi_teardown	= gic_irq_nmi_teardown,
+	.ipi_send_mask		= gic_ipi_send_mask,
 	.flags			= IRQCHIP_SET_TYPE_MASKED |
 				  IRQCHIP_SKIP_SET_WAKE |
 				  IRQCHIP_MASK_ON_SUSPEND,
@@ -1289,6 +1296,13 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 
 	switch (__get_intid_range(hw)) {
 	case SGI_RANGE:
+		irq_set_percpu_devid(irq);
+		irq_domain_set_info(d, irq, hw, chip, d->host_data,
+				    handle_percpu_devid_fasteoi_ipi,
+				    NULL, NULL);
+		irq_set_status_flags(irq, IRQ_NOAUTOEN);
+		break;
+
 	case PPI_RANGE:
 	case EPPI_RANGE:
 		irq_set_percpu_devid(irq);
@@ -1319,8 +1333,6 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 	return 0;
 }
 
-#define GIC_IRQ_TYPE_PARTITION	(GIC_IRQ_TYPE_LPI + 1)
-
 static int gic_irq_domain_translate(struct irq_domain *d,
 				    struct irq_fwspec *fwspec,
 				    unsigned long *hwirq,
@@ -1353,6 +1365,9 @@ static int gic_irq_domain_translate(struct irq_domain *d,
 			else
 				*hwirq += 16;
 			break;
+		case GIC_IRQ_TYPE_SGI:
+			*hwirq = fwspec->param[1];
+			break;
 		default:
 			return -EINVAL;
 		}
@@ -1657,9 +1672,9 @@ static int __init gic_init_bases(void __iomem *dist_base,
 
 	gic_update_rdist_properties();
 
-	gic_smp_init();
 	gic_dist_init();
 	gic_cpu_init();
+	gic_smp_init();
 	gic_cpu_pm_init();
 
 	if (gic_dist_supports_lpis()) {
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 06/11] irqchip/gic-v3: Configure SGIs as standard interrupts

[Sumit Garg]

Hi Marc,

On Tue, 19 May 2020 at 21:48, Marc Zyngier <maz@kernel.org> wrote:
>
> Change the way we deal with GICv3 SGIs by turning them into proper
> IRQs, and calling into the arch code to register the interrupt range
> instead of a callback.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  drivers/irqchip/irq-gic-v3.c | 91 +++++++++++++++++++++---------------
>  1 file changed, 53 insertions(+), 38 deletions(-)
>
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 23d7c87da407..d57289057b75 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -36,6 +36,9 @@
>  #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996    (1ULL << 0)
>  #define FLAGS_WORKAROUND_CAVIUM_ERRATUM_38539  (1ULL << 1)
>
> +#define GIC_IRQ_TYPE_PARTITION (GIC_IRQ_TYPE_LPI + 1)
> +#define GIC_IRQ_TYPE_SGI       (GIC_IRQ_TYPE_LPI + 2)
> +
>  struct redist_region {
>         void __iomem            *redist_base;
>         phys_addr_t             phys_base;
> @@ -657,38 +660,14 @@ static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs
>         if ((irqnr >= 1020 && irqnr <= 1023))
>                 return;
>
> -       /* Treat anything but SGIs in a uniform way */
> -       if (likely(irqnr > 15)) {
> -               int err;
> -
> -               if (static_branch_likely(&supports_deactivate_key))
> -                       gic_write_eoir(irqnr);
> -               else
> -                       isb();
> -
> -               err = handle_domain_irq(gic_data.domain, irqnr, regs);
> -               if (err) {
> -                       WARN_ONCE(true, "Unexpected interrupt received!\n");
> -                       gic_deactivate_unhandled(irqnr);
> -               }
> -               return;
> -       }
> -       if (irqnr < 16) {
> +       if (static_branch_likely(&supports_deactivate_key))
>                 gic_write_eoir(irqnr);
> -               if (static_branch_likely(&supports_deactivate_key))
> -                       gic_write_dir(irqnr);
> -#ifdef CONFIG_SMP
> -               /*
> -                * Unlike GICv2, we don't need an smp_rmb() here.
> -                * The control dependency from gic_read_iar to
> -                * the ISB in gic_write_eoir is enough to ensure
> -                * that any shared data read by handle_IPI will
> -                * be read after the ACK.
> -                */
> -               handle_IPI(irqnr, regs);
> -#else
> -               WARN_ONCE(true, "Unexpected SGI received!\n");
> -#endif
> +       else
> +               isb();
> +
> +       if (handle_domain_irq(gic_data.domain, irqnr, regs)) {
> +               WARN_ONCE(true, "Unexpected interrupt received!\n");
> +               gic_deactivate_unhandled(irqnr);
>         }
>  }
>
> @@ -1136,11 +1115,11 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq)
>         gic_write_sgi1r(val);
>  }
>
> -static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
> +static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask)
>  {
>         int cpu;
>
> -       if (WARN_ON(irq >= 16))
> +       if (WARN_ON(d->hwirq >= 16))
>                 return;
>
>         /*
> @@ -1154,7 +1133,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
>                 u16 tlist;
>
>                 tlist = gic_compute_target_list(&cpu, mask, cluster_id);
> -               gic_send_sgi(cluster_id, tlist, irq);
> +               gic_send_sgi(cluster_id, tlist, d->hwirq);
>         }
>
>         /* Force the above writes to ICC_SGI1R_EL1 to be executed */
> @@ -1163,10 +1142,36 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
>
>  static void gic_smp_init(void)
>  {
> -       set_smp_cross_call(gic_raise_softirq);
> +       struct irq_fwspec sgi_fwspec = {
> +               .fwnode         = gic_data.fwnode,
> +       };
> +       int base_sgi;
> +
>         cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
>                                   "irqchip/arm/gicv3:starting",
>                                   gic_starting_cpu, NULL);
> +
> +       if (is_of_node(gic_data.fwnode)) {
> +               /* DT */
> +               sgi_fwspec.param_count = 3;
> +               sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
> +               sgi_fwspec.param[1] = 0;
> +               sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
> +       } else {
> +               /* ACPI */
> +               sgi_fwspec.param_count = 2;
> +               sgi_fwspec.param[0] = 0;
> +               sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
> +       }
> +
> +       /* Register all 8 non-secure SGIs */
> +       base_sgi = __irq_domain_alloc_irqs(gic_data.domain, -1, 8,
> +                                          NUMA_NO_NODE, &sgi_fwspec,
> +                                          false, NULL);
> +       if (WARN_ON(base_sgi <= 0))
> +               return;
> +
> +       set_smp_ipi_range(base_sgi, 8);
>  }
>
>  static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
> @@ -1215,6 +1220,7 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
>  }
>  #else
>  #define gic_set_affinity       NULL
> +#define gic_ipi_send_mask      NULL
>  #define gic_smp_init()         do { } while(0)
>  #endif
>
> @@ -1257,6 +1263,7 @@ static struct irq_chip gic_chip = {
>         .irq_set_irqchip_state  = gic_irq_set_irqchip_state,
>         .irq_nmi_setup          = gic_irq_nmi_setup,
>         .irq_nmi_teardown       = gic_irq_nmi_teardown,
> +       .ipi_send_mask          = gic_ipi_send_mask,
>         .flags                  = IRQCHIP_SET_TYPE_MASKED |
>                                   IRQCHIP_SKIP_SET_WAKE |
>                                   IRQCHIP_MASK_ON_SUSPEND,

It looks like you missed to update "struct irq_chip gic_eoimode1_chip"
with similar change as follows:

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 2a09634..ceef63b 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1291,6 +1291,7 @@ static struct irq_chip gic_eoimode1_chip = {
        .irq_set_vcpu_affinity  = gic_irq_set_vcpu_affinity,
        .irq_nmi_setup          = gic_irq_nmi_setup,
        .irq_nmi_teardown       = gic_irq_nmi_teardown,
+       .ipi_send_mask          = gic_ipi_send_mask,
        .flags                  = IRQCHIP_SET_TYPE_MASKED |
                                  IRQCHIP_SKIP_SET_WAKE |
                                  IRQCHIP_MASK_ON_SUSPEND,

After incorporating this change, your patch-set works fine on my
Developerbox machine.

-Sumit

> @@ -1289,6 +1296,13 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>
>         switch (__get_intid_range(hw)) {
>         case SGI_RANGE:
> +               irq_set_percpu_devid(irq);
> +               irq_domain_set_info(d, irq, hw, chip, d->host_data,
> +                                   handle_percpu_devid_fasteoi_ipi,
> +                                   NULL, NULL);
> +               irq_set_status_flags(irq, IRQ_NOAUTOEN);
> +               break;
> +
>         case PPI_RANGE:
>         case EPPI_RANGE:
>                 irq_set_percpu_devid(irq);
> @@ -1319,8 +1333,6 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>         return 0;
>  }
>
> -#define GIC_IRQ_TYPE_PARTITION (GIC_IRQ_TYPE_LPI + 1)
> -
>  static int gic_irq_domain_translate(struct irq_domain *d,
>                                     struct irq_fwspec *fwspec,
>                                     unsigned long *hwirq,
> @@ -1353,6 +1365,9 @@ static int gic_irq_domain_translate(struct irq_domain *d,
>                         else
>                                 *hwirq += 16;
>                         break;
> +               case GIC_IRQ_TYPE_SGI:
> +                       *hwirq = fwspec->param[1];
> +                       break;
>                 default:
>                         return -EINVAL;
>                 }
> @@ -1657,9 +1672,9 @@ static int __init gic_init_bases(void __iomem *dist_base,
>
>         gic_update_rdist_properties();
>
> -       gic_smp_init();
>         gic_dist_init();
>         gic_cpu_init();
> +       gic_smp_init();
>         gic_cpu_pm_init();
>
>         if (gic_dist_supports_lpis()) {
> --
> 2.26.2
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 06/11] irqchip/gic-v3: Configure SGIs as standard interrupts

[Marc Zyngier]

Hi Sumit,

On 2020-05-20 10:52, Sumit Garg wrote:
> Hi Marc,
> 
> On Tue, 19 May 2020 at 21:48, Marc Zyngier <maz@kernel.org> wrote:
>> 
>> Change the way we deal with GICv3 SGIs by turning them into proper
>> IRQs, and calling into the arch code to register the interrupt range
>> instead of a callback.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  drivers/irqchip/irq-gic-v3.c | 91 
>> +++++++++++++++++++++---------------
>>  1 file changed, 53 insertions(+), 38 deletions(-)
>> 
>> diff --git a/drivers/irqchip/irq-gic-v3.c 
>> b/drivers/irqchip/irq-gic-v3.c
>> index 23d7c87da407..d57289057b75 100644
>> --- a/drivers/irqchip/irq-gic-v3.c
>> +++ b/drivers/irqchip/irq-gic-v3.c
>> @@ -36,6 +36,9 @@
>>  #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996    (1ULL << 0)
>>  #define FLAGS_WORKAROUND_CAVIUM_ERRATUM_38539  (1ULL << 1)
>> 
>> +#define GIC_IRQ_TYPE_PARTITION (GIC_IRQ_TYPE_LPI + 1)
>> +#define GIC_IRQ_TYPE_SGI       (GIC_IRQ_TYPE_LPI + 2)
>> +
>>  struct redist_region {
>>         void __iomem            *redist_base;
>>         phys_addr_t             phys_base;
>> @@ -657,38 +660,14 @@ static asmlinkage void __exception_irq_entry 
>> gic_handle_irq(struct pt_regs *regs
>>         if ((irqnr >= 1020 && irqnr <= 1023))
>>                 return;
>> 
>> -       /* Treat anything but SGIs in a uniform way */
>> -       if (likely(irqnr > 15)) {
>> -               int err;
>> -
>> -               if (static_branch_likely(&supports_deactivate_key))
>> -                       gic_write_eoir(irqnr);
>> -               else
>> -                       isb();
>> -
>> -               err = handle_domain_irq(gic_data.domain, irqnr, regs);
>> -               if (err) {
>> -                       WARN_ONCE(true, "Unexpected interrupt 
>> received!\n");
>> -                       gic_deactivate_unhandled(irqnr);
>> -               }
>> -               return;
>> -       }
>> -       if (irqnr < 16) {
>> +       if (static_branch_likely(&supports_deactivate_key))
>>                 gic_write_eoir(irqnr);
>> -               if (static_branch_likely(&supports_deactivate_key))
>> -                       gic_write_dir(irqnr);
>> -#ifdef CONFIG_SMP
>> -               /*
>> -                * Unlike GICv2, we don't need an smp_rmb() here.
>> -                * The control dependency from gic_read_iar to
>> -                * the ISB in gic_write_eoir is enough to ensure
>> -                * that any shared data read by handle_IPI will
>> -                * be read after the ACK.
>> -                */
>> -               handle_IPI(irqnr, regs);
>> -#else
>> -               WARN_ONCE(true, "Unexpected SGI received!\n");
>> -#endif
>> +       else
>> +               isb();
>> +
>> +       if (handle_domain_irq(gic_data.domain, irqnr, regs)) {
>> +               WARN_ONCE(true, "Unexpected interrupt received!\n");
>> +               gic_deactivate_unhandled(irqnr);
>>         }
>>  }
>> 
>> @@ -1136,11 +1115,11 @@ static void gic_send_sgi(u64 cluster_id, u16 
>> tlist, unsigned int irq)
>>         gic_write_sgi1r(val);
>>  }
>> 
>> -static void gic_raise_softirq(const struct cpumask *mask, unsigned 
>> int irq)
>> +static void gic_ipi_send_mask(struct irq_data *d, const struct 
>> cpumask *mask)
>>  {
>>         int cpu;
>> 
>> -       if (WARN_ON(irq >= 16))
>> +       if (WARN_ON(d->hwirq >= 16))
>>                 return;
>> 
>>         /*
>> @@ -1154,7 +1133,7 @@ static void gic_raise_softirq(const struct 
>> cpumask *mask, unsigned int irq)
>>                 u16 tlist;
>> 
>>                 tlist = gic_compute_target_list(&cpu, mask, 
>> cluster_id);
>> -               gic_send_sgi(cluster_id, tlist, irq);
>> +               gic_send_sgi(cluster_id, tlist, d->hwirq);
>>         }
>> 
>>         /* Force the above writes to ICC_SGI1R_EL1 to be executed */
>> @@ -1163,10 +1142,36 @@ static void gic_raise_softirq(const struct 
>> cpumask *mask, unsigned int irq)
>> 
>>  static void gic_smp_init(void)
>>  {
>> -       set_smp_cross_call(gic_raise_softirq);
>> +       struct irq_fwspec sgi_fwspec = {
>> +               .fwnode         = gic_data.fwnode,
>> +       };
>> +       int base_sgi;
>> +
>>         cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
>>                                   "irqchip/arm/gicv3:starting",
>>                                   gic_starting_cpu, NULL);
>> +
>> +       if (is_of_node(gic_data.fwnode)) {
>> +               /* DT */
>> +               sgi_fwspec.param_count = 3;
>> +               sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
>> +               sgi_fwspec.param[1] = 0;
>> +               sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
>> +       } else {
>> +               /* ACPI */
>> +               sgi_fwspec.param_count = 2;
>> +               sgi_fwspec.param[0] = 0;
>> +               sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
>> +       }
>> +
>> +       /* Register all 8 non-secure SGIs */
>> +       base_sgi = __irq_domain_alloc_irqs(gic_data.domain, -1, 8,
>> +                                          NUMA_NO_NODE, &sgi_fwspec,
>> +                                          false, NULL);
>> +       if (WARN_ON(base_sgi <= 0))
>> +               return;
>> +
>> +       set_smp_ipi_range(base_sgi, 8);
>>  }
>> 
>>  static int gic_set_affinity(struct irq_data *d, const struct cpumask 
>> *mask_val,
>> @@ -1215,6 +1220,7 @@ static int gic_set_affinity(struct irq_data *d, 
>> const struct cpumask *mask_val,
>>  }
>>  #else
>>  #define gic_set_affinity       NULL
>> +#define gic_ipi_send_mask      NULL
>>  #define gic_smp_init()         do { } while(0)
>>  #endif
>> 
>> @@ -1257,6 +1263,7 @@ static struct irq_chip gic_chip = {
>>         .irq_set_irqchip_state  = gic_irq_set_irqchip_state,
>>         .irq_nmi_setup          = gic_irq_nmi_setup,
>>         .irq_nmi_teardown       = gic_irq_nmi_teardown,
>> +       .ipi_send_mask          = gic_ipi_send_mask,
>>         .flags                  = IRQCHIP_SET_TYPE_MASKED |
>>                                   IRQCHIP_SKIP_SET_WAKE |
>>                                   IRQCHIP_MASK_ON_SUSPEND,
> 
> It looks like you missed to update "struct irq_chip gic_eoimode1_chip"
> with similar change as follows:
> 
> diff --git a/drivers/irqchip/irq-gic-v3.c 
> b/drivers/irqchip/irq-gic-v3.c
> index 2a09634..ceef63b 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -1291,6 +1291,7 @@ static struct irq_chip gic_eoimode1_chip = {
>         .irq_set_vcpu_affinity  = gic_irq_set_vcpu_affinity,
>         .irq_nmi_setup          = gic_irq_nmi_setup,
>         .irq_nmi_teardown       = gic_irq_nmi_teardown,
> +       .ipi_send_mask          = gic_ipi_send_mask,
>         .flags                  = IRQCHIP_SET_TYPE_MASKED |
>                                   IRQCHIP_SKIP_SET_WAKE |
>                                   IRQCHIP_MASK_ON_SUSPEND,
> 
> After incorporating this change, your patch-set works fine on my
> Developerbox machine.

Huh, well spotted. As I said, it has only been tested as guests,
hence not hitting this path. Time to throw it at the bigger stuff.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 06/11] irqchip/gic-v3: Configure SGIs as standard interrupts

[Valentin Schneider]

On 19/05/20 17:17, Marc Zyngier wrote:
> Change the way we deal with GICv3 SGIs by turning them into proper
> IRQs, and calling into the arch code to register the interrupt range
> instead of a callback.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  drivers/irqchip/irq-gic-v3.c | 91 +++++++++++++++++++++---------------
>  1 file changed, 53 insertions(+), 38 deletions(-)
>
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 23d7c87da407..d57289057b75 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -1163,10 +1142,36 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
>
>  static void gic_smp_init(void)
>  {
> -	set_smp_cross_call(gic_raise_softirq);
> +	struct irq_fwspec sgi_fwspec = {
> +		.fwnode		= gic_data.fwnode,
> +	};
> +	int base_sgi;
> +
>       cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
>                                 "irqchip/arm/gicv3:starting",
>                                 gic_starting_cpu, NULL);
> +
> +	if (is_of_node(gic_data.fwnode)) {
> +		/* DT */
> +		sgi_fwspec.param_count = 3;
> +		sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
> +		sgi_fwspec.param[1] = 0;
> +		sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
> +	} else {
> +		/* ACPI */
> +		sgi_fwspec.param_count = 2;
> +		sgi_fwspec.param[0] = 0;
> +		sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
> +	}
> +
> +	/* Register all 8 non-secure SGIs */
> +	base_sgi = __irq_domain_alloc_irqs(gic_data.domain, -1, 8,
> +					   NUMA_NO_NODE, &sgi_fwspec,
> +					   false, NULL);

So IIUC using irq_reserve_ipi() would require us to have a separate IPI
domain, so instead here we can use a fwspec + the 'regular' GIC domain.

One thing I see is that by not going through irq_reserve_ipi(), we don't set
data->common->ipi_offset. I think this is all kzalloc'd, and we want an
offset of 0 so it all works out, but this feels somewhat fragile.

> +	if (WARN_ON(base_sgi <= 0))
> +		return;
> +
> +	set_smp_ipi_range(base_sgi, 8);
>  }
>
>  static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
> @@ -1289,6 +1296,13 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>
>       switch (__get_intid_range(hw)) {
>       case SGI_RANGE:
> +		irq_set_percpu_devid(irq);
> +		irq_domain_set_info(d, irq, hw, chip, d->host_data,
> +				    handle_percpu_devid_fasteoi_ipi,
> +				    NULL, NULL);
> +		irq_set_status_flags(irq, IRQ_NOAUTOEN);

FWIW IRQ_NOAUTOEN is already set by irq_set_percpu_devid_flags(), so that's
not required. I know we do that for (E)PPIs, I think I already have a small
patch stashed somewhere regarding that.

> +		break;
> +
>       case PPI_RANGE:
>       case EPPI_RANGE:
>               irq_set_percpu_devid(irq);

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 06/11] irqchip/gic-v3: Configure SGIs as standard interrupts

[Marc Zyngier]

Hi Valentin,

On Thu, 21 May 2020 15:04:54 +0100
Valentin Schneider <valentin.schneider@arm.com> wrote:

> On 19/05/20 17:17, Marc Zyngier wrote:
> > Change the way we deal with GICv3 SGIs by turning them into proper
> > IRQs, and calling into the arch code to register the interrupt range
> > instead of a callback.
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  drivers/irqchip/irq-gic-v3.c | 91 +++++++++++++++++++++---------------
> >  1 file changed, 53 insertions(+), 38 deletions(-)
> >
> > diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> > index 23d7c87da407..d57289057b75 100644
> > --- a/drivers/irqchip/irq-gic-v3.c
> > +++ b/drivers/irqchip/irq-gic-v3.c
> > @@ -1163,10 +1142,36 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
> >
> >  static void gic_smp_init(void)
> >  {
> > -	set_smp_cross_call(gic_raise_softirq);
> > +	struct irq_fwspec sgi_fwspec = {
> > +		.fwnode		= gic_data.fwnode,
> > +	};
> > +	int base_sgi;
> > +
> >       cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
> >                                 "irqchip/arm/gicv3:starting",
> >                                 gic_starting_cpu, NULL);
> > +
> > +	if (is_of_node(gic_data.fwnode)) {
> > +		/* DT */
> > +		sgi_fwspec.param_count = 3;
> > +		sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
> > +		sgi_fwspec.param[1] = 0;
> > +		sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
> > +	} else {
> > +		/* ACPI */
> > +		sgi_fwspec.param_count = 2;
> > +		sgi_fwspec.param[0] = 0;
> > +		sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
> > +	}
> > +
> > +	/* Register all 8 non-secure SGIs */
> > +	base_sgi = __irq_domain_alloc_irqs(gic_data.domain, -1, 8,
> > +					   NUMA_NO_NODE, &sgi_fwspec,
> > +					   false, NULL);  
> 
> So IIUC using irq_reserve_ipi() would require us to have a separate IPI
> domain, so instead here we can use a fwspec + the 'regular' GIC domain.

Indeed. Using an IPI domain wouldn't bring much. But the major point
against the current state of the IPI domain is that it sucks a bit for
our use case. We want interrupts to be contiguous in the Linux IRQ
space, and the IPI allocator prevents this.

But maybe I should just bite the bullet and hack that as well.

> One thing I see is that by not going through irq_reserve_ipi(), we don't set
> data->common->ipi_offset. I think this is all kzalloc'd, and we want an
> offset of 0 so it all works out, but this feels somewhat fragile.

So far, nothing is using this field on the limited piece of code we
use. But I agree, not the nicest behaviour.

> > +	if (WARN_ON(base_sgi <= 0))
> > +		return;
> > +
> > +	set_smp_ipi_range(base_sgi, 8);
> >  }
> >
> >  static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
> > @@ -1289,6 +1296,13 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
> >
> >       switch (__get_intid_range(hw)) {
> >       case SGI_RANGE:
> > +		irq_set_percpu_devid(irq);
> > +		irq_domain_set_info(d, irq, hw, chip, d->host_data,
> > +				    handle_percpu_devid_fasteoi_ipi,
> > +				    NULL, NULL);
> > +		irq_set_status_flags(irq, IRQ_NOAUTOEN);  
> 
> FWIW IRQ_NOAUTOEN is already set by irq_set_percpu_devid_flags(), so that's
> not required. I know we do that for (E)PPIs, I think I already have a small
> patch stashed somewhere regarding that.

Already merged! ;-)

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 07/11] irqchip/gic: Refactor SMP configuration

[Marc Zyngier]

As we are about to change quite a lot of the SMP support code,
let's start by moving it around so that it minimizes the amount
of #ifdefery.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic.c | 92 ++++++++++++++++++++-------------------
 1 file changed, 48 insertions(+), 44 deletions(-)

diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 30ab623343d3..e289810e794c 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -325,36 +325,6 @@ static int gic_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
 	return 0;
 }
 
-#ifdef CONFIG_SMP
-static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
-			    bool force)
-{
-	void __iomem *reg = gic_dist_base(d) + GIC_DIST_TARGET + (gic_irq(d) & ~3);
-	unsigned int cpu, shift = (gic_irq(d) % 4) * 8;
-	u32 val, mask, bit;
-	unsigned long flags;
-
-	if (!force)
-		cpu = cpumask_any_and(mask_val, cpu_online_mask);
-	else
-		cpu = cpumask_first(mask_val);
-
-	if (cpu >= NR_GIC_CPU_IF || cpu >= nr_cpu_ids)
-		return -EINVAL;
-
-	gic_lock_irqsave(flags);
-	mask = 0xff << shift;
-	bit = gic_cpu_map[cpu] << shift;
-	val = readl_relaxed(reg) & ~mask;
-	writel_relaxed(val | bit, reg);
-	gic_unlock_irqrestore(flags);
-
-	irq_data_update_effective_affinity(d, cpumask_of(cpu));
-
-	return IRQ_SET_MASK_OK_DONE;
-}
-#endif
-
 static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 {
 	u32 irqstat, irqnr;
@@ -803,6 +773,34 @@ static int gic_pm_init(struct gic_chip_data *gic)
 #endif
 
 #ifdef CONFIG_SMP
+static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
+			    bool force)
+{
+	void __iomem *reg = gic_dist_base(d) + GIC_DIST_TARGET + (gic_irq(d) & ~3);
+	unsigned int cpu, shift = (gic_irq(d) % 4) * 8;
+	u32 val, mask, bit;
+	unsigned long flags;
+
+	if (!force)
+		cpu = cpumask_any_and(mask_val, cpu_online_mask);
+	else
+		cpu = cpumask_first(mask_val);
+
+	if (cpu >= NR_GIC_CPU_IF || cpu >= nr_cpu_ids)
+		return -EINVAL;
+
+	gic_lock_irqsave(flags);
+	mask = 0xff << shift;
+	bit = gic_cpu_map[cpu] << shift;
+	val = readl_relaxed(reg) & ~mask;
+	writel_relaxed(val | bit, reg);
+	gic_unlock_irqrestore(flags);
+
+	irq_data_update_effective_affinity(d, cpumask_of(cpu));
+
+	return IRQ_SET_MASK_OK_DONE;
+}
+
 static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
 {
 	int cpu;
@@ -832,6 +830,23 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
 
 	gic_unlock_irqrestore(flags);
 }
+
+static int gic_starting_cpu(unsigned int cpu)
+{
+	gic_cpu_init(&gic_data[0]);
+	return 0;
+}
+
+static __init void gic_smp_init(void)
+{
+	set_smp_cross_call(gic_raise_softirq);
+	cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
+				  "irqchip/arm/gic:starting",
+				  gic_starting_cpu, NULL);
+}
+#else
+#define gic_smp_init()		do { } while(0)
+#define gic_set_affinity	NULL
 #endif
 
 #ifdef CONFIG_BL_SWITCHER
@@ -1036,12 +1051,6 @@ static int gic_irq_domain_translate(struct irq_domain *d,
 	return -EINVAL;
 }
 
-static int gic_starting_cpu(unsigned int cpu)
-{
-	gic_cpu_init(&gic_data[0]);
-	return 0;
-}
-
 static int gic_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
 				unsigned int nr_irqs, void *arg)
 {
@@ -1088,10 +1097,8 @@ static void gic_init_chip(struct gic_chip_data *gic, struct device *dev,
 		gic->chip.irq_set_vcpu_affinity = gic_irq_set_vcpu_affinity;
 	}
 
-#ifdef CONFIG_SMP
 	if (gic == &gic_data[0])
 		gic->chip.irq_set_affinity = gic_set_affinity;
-#endif
 }
 
 static int gic_init_bases(struct gic_chip_data *gic,
@@ -1208,12 +1215,7 @@ static int __init __gic_init_bases(struct gic_chip_data *gic,
 		 */
 		for (i = 0; i < NR_GIC_CPU_IF; i++)
 			gic_cpu_map[i] = 0xff;
-#ifdef CONFIG_SMP
-		set_smp_cross_call(gic_raise_softirq);
-#endif
-		cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
-					  "irqchip/arm/gic:starting",
-					  gic_starting_cpu, NULL);
+
 		set_handle_irq(gic_handle_irq);
 		if (static_branch_likely(&supports_deactivate_key))
 			pr_info("GIC: Using split EOI/Deactivate mode\n");
@@ -1230,6 +1232,8 @@ static int __init __gic_init_bases(struct gic_chip_data *gic,
 	ret = gic_init_bases(gic, handle);
 	if (ret)
 		kfree(name);
+	else if (gic == &gic_data[0])
+		gic_smp_init();
 
 	return ret;
 }
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[Marc Zyngier]

Change the way we deal with GIC SGIs by turning them into proper
IRQs, and calling into the arch code to register the interrupt range
instead of a callback.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic.c | 113 +++++++++++++++++++++++++-------------
 1 file changed, 74 insertions(+), 39 deletions(-)

diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index e289810e794c..8d146ac28c74 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -47,6 +47,8 @@
 
 #include "irq-gic-common.h"
 
+#define GIC_IRQ_TYPE_SGI	0x561
+
 #ifdef CONFIG_ARM64
 #include <asm/cpufeature.h>
 
@@ -335,31 +337,22 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 		irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
 		irqnr = irqstat & GICC_IAR_INT_ID_MASK;
 
-		if (likely(irqnr > 15 && irqnr < 1020)) {
-			if (static_branch_likely(&supports_deactivate_key))
-				writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
-			isb();
-			handle_domain_irq(gic->domain, irqnr, regs);
-			continue;
-		}
-		if (irqnr < 16) {
+		if (unlikely(irqnr >= 1020))
+			break;
+
+		if (static_branch_likely(&supports_deactivate_key))
 			writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
-			if (static_branch_likely(&supports_deactivate_key))
-				writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE);
-#ifdef CONFIG_SMP
-			/*
-			 * Ensure any shared data written by the CPU sending
-			 * the IPI is read after we've read the ACK register
-			 * on the GIC.
-			 *
-			 * Pairs with the write barrier in gic_raise_softirq
-			 */
+		isb();
+
+		/*
+		 * Ensure any shared data written by the CPU sending the IPI
+		 * is read after we've read the ACK register on the GIC.
+		 *
+		 * Pairs with the write barrier in gic_ipi_send_mask
+		 */
+		if (irqnr <= 15)
 			smp_rmb();
-			handle_IPI(irqnr, regs);
-#endif
-			continue;
-		}
-		break;
+		handle_domain_irq(gic->domain, irqnr, regs);
 	} while (1);
 }
 
@@ -801,14 +794,14 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
 	return IRQ_SET_MASK_OK_DONE;
 }
 
-static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
+static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask)
 {
 	int cpu;
 	unsigned long flags, map = 0;
 
 	if (unlikely(nr_cpu_ids == 1)) {
 		/* Only one CPU? let's do a self-IPI... */
-		writel_relaxed(2 << 24 | irq,
+		writel_relaxed(2 << 24 | d->hwirq,
 			       gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
 		return;
 	}
@@ -826,7 +819,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
 	dmb(ishst);
 
 	/* this always happens on GIC0 */
-	writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+	writel_relaxed(map << 16 | d->hwirq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
 
 	gic_unlock_irqrestore(flags);
 }
@@ -839,14 +832,40 @@ static int gic_starting_cpu(unsigned int cpu)
 
 static __init void gic_smp_init(void)
 {
-	set_smp_cross_call(gic_raise_softirq);
+	struct irq_fwspec sgi_fwspec = {
+		.fwnode		= gic_data[0].domain->fwnode,
+	};
+	int base_sgi;
+
 	cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_GIC_STARTING,
 				  "irqchip/arm/gic:starting",
 				  gic_starting_cpu, NULL);
+
+	if (is_of_node(gic_data[0].domain->fwnode)) {
+		/* DT */
+		sgi_fwspec.param_count = 3;
+		sgi_fwspec.param[0] = GIC_IRQ_TYPE_SGI;
+		sgi_fwspec.param[1] = 0;
+		sgi_fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
+	} else {
+		/* ACPI */
+		sgi_fwspec.param_count = 2;
+		sgi_fwspec.param[0] = 0;
+		sgi_fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
+	}
+
+	base_sgi = __irq_domain_alloc_irqs(gic_data[0].domain, -1, 8,
+					   NUMA_NO_NODE, &sgi_fwspec,
+					   false, NULL);
+	if (WARN_ON(base_sgi <= 0))
+		return;
+
+	set_smp_ipi_range(base_sgi, 8);
 }
 #else
 #define gic_smp_init()		do { } while(0)
 #define gic_set_affinity	NULL
+#define gic_ipi_send_mask	NULL
 #endif
 
 #ifdef CONFIG_BL_SWITCHER
@@ -993,16 +1012,26 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 {
 	struct gic_chip_data *gic = d->host_data;
 
-	if (hw < 32) {
+	switch (hw) {
+	case 0 ... 15:
+		irq_set_percpu_devid(irq);
+		irq_domain_set_info(d, irq, hw, &gic->chip, d->host_data,
+				    handle_percpu_devid_fasteoi_ipi,
+				    NULL, NULL);
+		irq_set_status_flags(irq, IRQ_NOAUTOEN);
+		break;
+	case 16 ... 31:
 		irq_set_percpu_devid(irq);
 		irq_domain_set_info(d, irq, hw, &gic->chip, d->host_data,
 				    handle_percpu_devid_irq, NULL, NULL);
 		irq_set_status_flags(irq, IRQ_NOAUTOEN);
-	} else {
+		break;
+	default:
 		irq_domain_set_info(d, irq, hw, &gic->chip, d->host_data,
 				    handle_fasteoi_irq, NULL, NULL);
 		irq_set_probe(irq);
 		irqd_set_single_target(irq_desc_get_irq_data(irq_to_desc(irq)));
+		break;
 	}
 	return 0;
 }
@@ -1020,15 +1049,19 @@ static int gic_irq_domain_translate(struct irq_domain *d,
 		if (fwspec->param_count < 3)
 			return -EINVAL;
 
-		/* Get the interrupt number and add 16 to skip over SGIs */
-		*hwirq = fwspec->param[1] + 16;
-
-		/*
-		 * For SPIs, we need to add 16 more to get the GIC irq
-		 * ID number
-		 */
-		if (!fwspec->param[0])
-			*hwirq += 16;
+		switch (fwspec->param[0]) {
+		case 0:			/* SPI */
+			*hwirq = fwspec->param[1] + 32;
+			break;
+		case 1:			/* PPI */
+			*hwirq = fwspec->param[1] + 16;
+			break;
+		case GIC_IRQ_TYPE_SGI:	/* SGI, can't be set via DT */
+			*hwirq = fwspec->param[1];
+			break;
+		default:
+			return -EINVAL;
+		}
 
 		*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
 
@@ -1097,8 +1130,10 @@ static void gic_init_chip(struct gic_chip_data *gic, struct device *dev,
 		gic->chip.irq_set_vcpu_affinity = gic_irq_set_vcpu_affinity;
 	}
 
-	if (gic == &gic_data[0])
+	if (gic == &gic_data[0]) {
 		gic->chip.irq_set_affinity = gic_set_affinity;
+		gic->chip.ipi_send_mask = gic_ipi_send_mask;
+	}
 }
 
 static int gic_init_bases(struct gic_chip_data *gic,
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[dann frazier]

On Tue, May 19, 2020 at 05:17:52PM +0100, Marc Zyngier wrote:
> Change the way we deal with GIC SGIs by turning them into proper
> IRQs, and calling into the arch code to register the interrupt range
> instead of a callback.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

hey Marc,

  I bisected a boot failure on our Gigabyte R120-T33 systems (ThunderX
CN88XX) down to this commit, but only when running in ACPI mode. See below:


EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
[    0.000000] Linux version 5.11.0-13-generic (buildd@bos02-arm64-067) (gcc (Ubuntu 10.2.1-23ubuntu1) 10.2.1 20210312, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #14-Ubuntu SMP Fri Mar 19 16:57:35 UTC 2021 (Ubuntu 5.11.0-13.14-generic 5.11.7)
[    0.000000] Machine model: Cavium ThunderX CN88XX board
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi: ESRT=0xffce0ff18 SMBIOS 3.0=0xfffb0000 ACPI 2.0=0xffc870000 MEMRESERVE=0xffa9b8e98 
[    0.000000] secureboot: Secure boot disabled
[    0.000000] esrt: Reserving ESRT space from 0x0000000ffce0ff18 to 0x0000000ffce0ff50.
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x0000000FFC870000 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000FFC870028 00008C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000FFC8700B8 000114 (v06 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000FFC8701D0 00220B (v02 CAVIUM THUNDERX 00000001 INTL 20130517)
[    0.000000] ACPI: SPMI 0x0000000FFC8723E0 000041 (v05 ALASKA A M I    00000000 AMI. 00000000)
[    0.000000] ACPI: FIDT 0x0000000FFC872428 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: APIC 0x0000000FFC8724C8 000F68 (v03 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: DBG2 0x0000000FFC873430 000067 (v01 CAVIUM CN88XDBG 00000000 INTL 20150619)
[    0.000000] ACPI: GTDT 0x0000000FFC873498 0000E0 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: IORT 0x0000000FFC873578 0013D4 (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: MCFG 0x0000000FFC874950 00006C (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SSDT 0x0000000FFC8749C0 00089C (v02 CAVIUM NETWORK  00000001 INTL 20150619)
[    0.000000] ACPI: OEM1 0x0000000FFC875260 0001E8 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SLIT 0x0000000FFC875448 000030 (v01 CAVIUM TEMPLATE 00000001 INTL 20150619)
[    0.000000] ACPI: SPCR 0x0000000FFC875478 000050 (v02 A M I  APTIO V  01072009 AMI. 0005000B)
[    0.000000] ACPI: BGRT 0x0000000FFC8754C8 000038 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0x87e024000000,115200
[    0.000000] efi_bgrt: Ignoring BGRT: Incorrect BMP magic number 0x3707 (expected 0x4d42)
[    0.000000] ACPI: NUMA: Failed to initialise from firmware
[    0.000000] NUMA: Faking a node at [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000] NUMA: NODE_DATA [mem 0xff67d40c0-0xff67d8fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000500000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x0000000fff0fffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000500000-0x0000000000dfffff]
[    0.000000]   node   0: [mem 0x0000000000e00000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000010000000-0x00000000102fffff]
[    0.000000]   node   0: [mem 0x0000000010300000-0x00000000fff9ffff]
[    0.000000]   node   0: [mem 0x00000000fffa0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x0000000fee69ffff]
[    0.000000]   node   0: [mem 0x0000000fee6a0000-0x0000000fee6bffff]
[    0.000000]   node   0: [mem 0x0000000fee6c0000-0x0000000ffa98ffff]
[    0.000000]   node   0: [mem 0x0000000ffa990000-0x0000000ffa9affff]
[    0.000000]   node   0: [mem 0x0000000ffa9b0000-0x0000000ffac1ffff]
[    0.000000]   node   0: [mem 0x0000000ffac20000-0x0000000ffb09ffff]
[    0.000000]   node   0: [mem 0x0000000ffb0a0000-0x0000000ffc8affff]
[    0.000000]   node   0: [mem 0x0000000ffc8b0000-0x0000000ffc90ffff]
[    0.000000]   node   0: [mem 0x0000000ffc910000-0x0000000ffca2ffff]
[    0.000000]   node   0: [mem 0x0000000ffca30000-0x0000000ffca3ffff]
[    0.000000]   node   0: [mem 0x0000000ffca40000-0x0000000ffcdbffff]
[    0.000000]   node   0: [mem 0x0000000ffcdc0000-0x0000000ffd12ffff]
[    0.000000]   node   0: [mem 0x0000000ffd130000-0x0000000ffecbffff]
[    0.000000]   node   0: [mem 0x0000000ffecc0000-0x0000000ffed1ffff]
[    0.000000]   node   0: [mem 0x0000000ffed20000-0x0000000fff0fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000]   Normal zone: 256 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x00000000fd000000
[    0.000000] psci: probing for conduit method from ACPI.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] ACPI: SRAT not present
[    0.000000] percpu: Embedded 33 pages/cpu s97176 r8192 d29800 u135168
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Software prefetching using PRFM
[    0.000000] CPU features: detected: Cavium erratum 27456
[    0.000000] CPU features: detected: Cavium erratum 30115
[    0.000000] CPU features: kernel page table isolation forced OFF by ARM64_WORKAROUND_CAVIUM_27456
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16510032
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=(http,10.229.32.21:5248)/images/ubuntu/arm64/ga-21.04/hirsute/stable/boot-kernel nomodeset ro root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-21.04/hirsute/stable/squashfs ip=::::seidel:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{datasource_list: [MAAS]}end_cc cloud-config-url=http://10.229.32.21:5248/MAAS/metadata/latest/by-id/sb3xy8/?op=get_preseed apparmor=0 log_host=10.229.32.21 log_port=5247 --- acpi=force BOOTIF=01-1c:1b:0d:0d:52:d7
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 192512 bytes
[    0.000000] printk: log_buf_len min size: 262144 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 255416(97%)
[    0.000000] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f9000000-0x00000000fd000000] (64MB)
[    0.000000] Memory: 65534308K/67088384K available (16064K kernel code, 3570K rwdata, 11876K rodata, 7680K init, 1152K bss, 1521308K reserved, 32768K cma-reserved)
[    0.000000] random: get_random_u64 called from kmem_cache_open+0x34/0x280 with crng_init=0
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=48, Nodes=1
[    0.000000] ftrace: allocating 54554 entries in 214 pages
[    0.000000] ftrace: allocated 214 pages with 5 groups
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=48.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=48
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GIC: enabling workaround for GICv3: Cavium erratum 38539
[    0.000000] GICv3: 128 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000801080000000
[    0.000000] ACPI: SRAT not present
[    0.000000] ITS [mem 0x801000020000-0x80100003ffff]
[    0.000000] ITS@0x0000801000020000: allocated 2097152 Devices @102000000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000100280000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000100290000
[    0.000000] arch_timer: Failed to initialize memory-mapped timer.
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000000] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000164] Console: colour dummy device 80x25
[    0.000255] ACPI: Core revision 20201113
[    0.000481] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.000492] pid_max: default: 49152 minimum: 384
[    0.000609] LSM: Security Framework initializing
[    0.000627] Yama: becoming mindful.
[    0.000801] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000866] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.001701] ACPI PPTT: No PPTT table found, CPU and cache topology may be inaccurate
[    0.002306] rcu: Hierarchical SRCU implementation.
[    0.003082] Platform MSI: ITS@0x801000020000 domain created
[    0.003097] PCI/MSI: ITS@0x801000020000 domain created
[    0.003107] fsl-mc MSI: ITS@0x801000020000 domain created
[    0.003128] Remapping and enabling EFI services.
[    0.005245] smp: Bringing up secondary CPUs ...
[    0.005739] Detected VIPT I-cache on CPU1
[    0.005754] GICv3: CPU1: found redistributor 1 region 0:0x0000801080020000
[    0.005781] GICv3: CPU1: using allocated LPI pending table @0x00000001002a0000
[    0.005803] CPU1: Booted secondary processor 0x0000000001 [0x431f0a11]
[    0.006364] Detected VIPT I-cache on CPU2
[    0.006374] GICv3: CPU2: found redistributor 2 region 0:0x0000801080040000
[    0.006399] GICv3: CPU2: using allocated LPI pending table @0x00000001002b0000
[    0.006418] CPU2: Booted secondary processor 0x0000000002 [0x431f0a11]
[    0.006962] Detected VIPT I-cache on CPU3
[    0.006973] GICv3: CPU3: found redistributor 3 region 0:0x0000801080060000
[    0.006997] GICv3: CPU3: using allocated LPI pending table @0x00000001002c0000
[    0.007017] CPU3: Booted secondary processor 0x0000000003 [0x431f0a11]
[    0.008363] Detected VIPT I-cache on CPU4
[    0.008374] GICv3: CPU4: found redistributor 4 region 0:0x0000801080080000
[    0.008399] GICv3: CPU4: using allocated LPI pending table @0x00000001002d0000
[    0.008419] CPU4: Booted secondary processor 0x0000000004 [0x431f0a11]
[    0.008968] Detected VIPT I-cache on CPU5
[    0.008979] GICv3: CPU5: found redistributor 5 region 0:0x00008010800a0000
[    0.009004] GICv3: CPU5: using allocated LPI pending table @0x00000001002e0000
[    0.009025] CPU5: Booted secondary processor 0x0000000005 [0x431f0a11]
[    0.009566] Detected VIPT I-cache on CPU6
[    0.009577] GICv3: CPU6: found redistributor 6 region 0:0x00008010800c0000
[    0.009602] GICv3: CPU6: using allocated LPI pending table @0x00000001002f0000
[    0.009623] CPU6: Booted secondary processor 0x0000000006 [0x431f0a11]
[    0.010169] Detected VIPT I-cache on CPU7
[    0.010180] GICv3: CPU7: found redistributor 7 region 0:0x00008010800e0000
[    0.010206] GICv3: CPU7: using allocated LPI pending table @0x0000000100300000
[    0.010227] CPU7: Booted secondary processor 0x0000000007 [0x431f0a11]
[    0.010771] Detected VIPT I-cache on CPU8
[    0.010783] GICv3: CPU8: found redistributor 8 region 0:0x0000801080100000
[    0.010809] GICv3: CPU8: using allocated LPI pending table @0x0000000100310000
[    0.010830] CPU8: Booted secondary processor 0x0000000008 [0x431f0a11]
[    0.011382] Detected VIPT I-cache on CPU9
[    0.011394] GICv3: CPU9: found redistributor 9 region 0:0x0000801080120000
[    0.011419] GICv3: CPU9: using allocated LPI pending table @0x0000000100320000
[    0.011CPU10: using allocated LPI pending table @0x0000000100330000
[    0.012048] CPU10: Booted secondary processor 0x000000000a [0x431f0a11]
[    0.012613] Detected VIPT I-cache on CPU11
[    0.012626] GICv3: CPU11: found redistributor b region 0:0x0000801080160000
[    0.012651] GICv3: CPU11: using allocated LPI pending table @0x0000000100340000
[    0.012673] CPU11: Booted secondary processor 0x000000000b [0x431f0a11]
[    0.013215] Detected VIPT I-cache on CPU12
[    0.013227] GICv3: CPU12: found redistributor c region 0:0x0000801080180000
[    0.013253] GICv3: CPU12: using allocated LPI pending table @0x0000000100350000
[    0.013275] CPU12: Booted secondary processor 0x000000000c [0x431f0a11]
[    0.013837] Detected VIPT I-cache on CPU13
[    0.013850] GICv3: CPU13: found redistributor d region 0:0x00008010801a0000
[    0.013877] GICv3: CPU13: using allocated LPI pending table @0x0000000100360000
[    0.013899] CPU13: Booted secondary processor 0x000000000d [0x431f0a11]
[    0.014444] Detected VIPT I-cache on CPU14
[    0.014457] GICv3: CPU14: found redistributor e region 0:0x00008010801c0000
[    0.014483] GICv3: CPU14: using allocated LPI pending table @0x0000000100370000
[    0.014506] CPU14: Booted secondary processor 0x000000000e [0x431f0a11]
[    0.015064] Detected VIPT I-cache on CPU15
[    0.015077] GICv3: CPU15: found redistributor f region 0:0x00008010801e0000
[    0.015107] GICv3: CPU15: using allocated LPI pending table @0x0000000100380000
[    0.015130] CPU15: Booted secondary processor 0x000000000f [0x431f0a11]
[    0.015679] Detected VIPT I-cache on CPU16
[    0.015692] GICv3: CPU16: found redistributor 100 region 0:0x0000801080200000
[    0.015719] GICv3: CPU16: using allocated LPI pending table @0x0000000100390000
[    0.015741] CPU16: Booted secondary processor 0x0000000100 [0x431f0a11]
[    0.016300] Detected VIPT I-cache on CPU17
[    0.016314] GICv3: CPU17: found redistributor 101 region 0:0x0000801080220000
[    0.016341] GICv3: CPU17: using allocated LPI pending table @0x00000001003a0000
[    0.016364] CPU17: Booted secondary processor 0x0000000101 [0x431f0a11]
[    0.016932] Detected VIPT I-cache on CPU18
[    0.016945] GICv3: CPU18: found redistributor 102 region 0:0x0000801080240000
[    0.016973] GICv3: CPU18: using allocated LPI pending table @0x00000001003b0000
[    0.016996] CPU18: Booted secondary processor 0x0000000102 [0x431f0a11]
[    0.017548] Detected VIPT I-cache on CPU19
[    0.017563] GICv3: CPU19: found redistributor 103 region 0:0x0000801080260000
[    0.017591] GICv3: CPU19: using allocated LPI pending table @0x00000001003c0000
[    0.017614] CPU19: Booted secondary processor 0x0000000103 [0x431f0a11]
[    0.018172] Detected VIPT I-cache on CPU20
[    0.018186] GICv3: CPU20: found redistributor 104 region 0:0x0000801080280000
[    0.018213] GICv3: CPU20: using allocated LPI pending table @0x00000001003d0000
[    0.018237] CPU20: Booted secondary processor 0x0000000104 [0x431f0a11]
[    0.018783] Detected VIPT I-cache on CPU21
[    0.018797] GICv3: CPU21: found redistributor 105 region 0:0x00008010802a0000
[    0.018825] GICv3: CPU21: using allocated LPI pending table @0x00000001003e0000
[    0.018849] CPU21: Booted secondary processor 0x0000000105 [0x431f0a11]
[    0.019408] Detected VIPT I-cache on CPU22
[    0.019422] GICv3: CPU22: found redistributor 106 region 0:0x00008010802c0000
[    0.019450] GICv3: CPU22: using allocated LPI pending table @0x00000001003f0000
[    0.019474] CPU22: Booted secondary processor 0x0000000106 [0x431f0a11]
[    0.020034] Detected VIPT I-cache on CPU23
[    0.020049] GICv3: CPU23: found redistributor 107 region 0:0x00008010802e0000
[    0.020077] GICv3: CPU23: using allocated LPI pending table @0x0000000100400000
[    0.020102] CPU23: Booted secondary processor 0x0000000107 [0x431f0a11]
[    0.020675] Detected VIPT I-cache on CPU24
[    0.020690] GICv3: CPU24: found redistributor 108 region 0:0x0000801080300000
[    0.020717] GICv3: CPU24: using allocated LPI pending table @0x0000000100410000
[    0.020743] CPU24: Booted secondary processor 0x0000000108 [0x431f0a11]
[    0.021312] Detected VIPT I-cache on CPU25
[    0.021327] GICv3: CPU25: found redistributor 109 region 0:0x0000801080320000
[    0.021356] GICv3: CPU25: using allocated LPI pending table @0x0000000100420000
[    0.021381] CPU25: Booted secondary processor 0x0000000109 [0x431f0a11]
[    0.021936] Detected VIPT I-cache on CPU26
[    0.021951] GICv3: CPU26: found redistributor 10a region 0:0x0000801080340000
[    0.021980] GICv3: CPU26: using allocated LPI pending table @0x0000000100430000
[    0.022005] CPU26: Booted secondary processor 0x000000010a [0x431f0a11]
[    0.022571] Detected VIPT I-cache on CPU27
[    0.022587] GICv3: CPU27: found redistributor 10b region 0:0x0000801080360000
[    0.022615] GICv3: CPU27: using allocated LPI pending table @0x0000000100440000
[    0.022641] CPU27: Booted secondary processor 0x000000010b [0x431f0a11]
[    0.023198] Detected VIPT I-cache on CPU28
[    0.023213] GICv3: CPU28: found redistributor 10c region 0:0x0000801080380000
[    0.023242] GICv3: CPU28: using allocated LPI pending table @0x0000000100450000
[    0.023268] CPU28: Booted secondary processor 0x000000010c [0x431f0a11]
[    0.023833] Detected VIPT I-cache on CPU29
[    0.023848] GICv3: CPU29: found redistributor 10d region 0:0x00008010803a0000
[    0.023878] GICv3: CPU29: using allocated LPI pending table @0x0000000100460000
[    0.023905] CPU29: Booted secondary processor 0x000000010d [0x431f0a11]
[    0.024490] Detected VIPT I-cache on CPU30
[    0.024505] GICv3: CPU30: found redistributor 10e region 0:0x00008010803c0000
[    0.024534] GICv3: CPU30: using allocated LPI pending table @0x0000000100470000
[    0.024561] CPU30: Booted secondary processor 0x000000010e [0x431f0a11]
[    0.025127] Detected VIPT I-cache on CPU31
[    0.025143] GICv3: CPU31: found redistributor 10f region 0:0x00008010803e0000
[    0.025173] GICv3: CPU31: using allocated LPI pending table @0x0000000100480000
[    0.025200] CPU31: Booted secondary processor 0x000000010f [0x431f0a11]
[    0.025766] Detected VIPT I-cache on CPU32
[    0.025781] GICv3: CPU32: found redistributor 200 region 0:0x0000801080400000
[    0.025811] GICv3: CPU32: using allocated LPI pending table @0x0000000100490000
[    0.025838] CPU32: Booted secondary processor 0x0000000200 [0x431f0a11]
[    0.026417] Detected VIPT I-cache on CPU33
[    0.026434] GICv3: CPU33: found redistributor 201 region 0:0x0000801080420000
[    0.026463] GICv3: CPU33: using allocated LPI pending table @0x00000001004a0000
[    0.026490] CPU33: Booted secondary processor 0x0000000201 [0x431f0a11]
[    0.027062] Detected VIPT I-cache on CPU34
[    0.027078] GICv3: CPU34: found redistributor 202 region 0:0x0000801080440000
[    0.027108] GICv3: CPU34: using allocated LPI pending table @0x00000001004b0000
[    0.027135] CPU34: Booted secondary processor 0x0000000202 [0x431f0a11]
[    0.027705] Detected VIPT I-cache on CPU35
[    0.027722] GICv3: CPU35: found redistributor 203 region 0:0x0000801080460000
[    0.027752] GICv3: CPU35: using allocated LPI pending table @0x00000001004c0000
[    0.027780] CPU35: Booted secondary processor 0x0000000203 [0x431f0a11]
[    0.028354] Detected VIPT I-cache on CPU36
[    0.028371] GICv3: CPU36: found redistributor 204 region 0:0x0000801080480000
[    0.028401] GICv3: CPU36: using allocated LPI pending table @0x00000001004d0000
[    0.028430] CPU36: Booted secondary processor 0x0000000204 [0x431f0a11]
[    0.028997] Detected VIPT I-cache on CPU37
[    0.029014] GICv3: CPU37: found redistributor 205 region 0:0x00008010804a0000
[    0.029044] GICv3: CPU37: using allocated LPI pending table @0x00000001004e0000
[    0.029073] CPU37: Booted secondary processor 0x0000000205 [0x431f0a11]
[    0.029658] Detected VIPT I-cache on CPU38
[    0.029675] GICv3: CPU38: found redistributor 206 region 0:0x00008010804c0000
[    0.029705] GICv3: CPU38: using allocated LPI pending table @0x00000001004f0000
[    0.029734] CPU38: Booted secondary processor 0x0000000206 [0x431f0a11]
[    0.030298] Detected VIPT I-cache on CPU39
[    0.030315] GICv3: CPU39: found redistributor 207 region 0:0x00008010804e0000
[    0.030346] GICv3: CPU39: using allocated LPI pending table @0x0000000100500000
[    0.030375] CPU39: Booted secondary processor 0x0000000207 [0x431f0a11]
[    0.030951] Detected VIPT I-cache on CPU40
[    0.030969] GICv3: CPU40: found redistributor 208 region 0:0x0000801080500000
[    0.030999] GICv3: CPU40: using allocated LPI pending table @0x0000000100510000
[    0.031029] CPU40: Booted secondary processor 0x0000000208 [0x431f0a11]
[    0.031605] Detected VIPT I-cache on CPU41
[    0.031623] GICv3: CPU41: found redistributor 209 region 0:0x0000801080520000
[    0.031654] GICv3: CPU41: using allocated LPI pending table @0x0000000100520000
[    0.031683] CPU41: Booted secondary processor 0x0000000209 [0x431f0a11]
[    0.032260] Detected VIPT I-cache on CPU42
[    0.032278] GICv3: CPU42: found redistributor 20a region 0:0x0000801080540000
[    0.032308] GICv3: CPU42: using allocated LPI pending table @0x0000000100530000
[    0.032338] CPU42: Booted secondary processor 0x000000020a [0x431f0a11]
[    0.032913] Detected VIPT I-cache on CPU43
[    0.032931] GICv3: CPU43: found redistributor 20b region 0:0x0000801080560000
[    0.032962] GICv3: CPU43: using allocated LPI pending table @0x0000000100540000
[    0.032992] CPU43: Booted secondary processor 0x000000020b [0x431f0a11]
[    0.033571] Detected VIPT I-cache on CPU44
[    0.033589] GICv3: CPU44: found redistributor 20c region 0:0x0000801080580000
[    0.033620] GICv3: CPU44: using allocated LPI pending table @0x0000000100550000
[    0.033650] CPU44: Booted secondary processor 0x000000020c [0x431f0a11]
[    0.034235] Detected VIPT I-cache on CPU45
[    0.034254] GICv3: CPU45: found redistributor 20d region 0:0x00008010805a0000
[    0.034285] GICv3: CPU45: using allocated LPI pending table @0x0000000100560000
[    0.034316] CPU45: Booted secondary processor 0x000000020d [0x431f0a11]
[    0.034893] Detected VIPT I-cache on CPU46
[    0.034912] GICv3: CPU46: found redistributor 20e region 0:0x00008010805c0000
[    0.034944] GICv3: CPU46: using allocated LPI pending table @0x0000000100570000
[    0.034974] CPU46: Booted secondary processor 0x000000020e [0x431f0a11]
[    0.035553] Detected VIPT I-cache on CPU47
[    0.035573] GICv3: CPU47: found redistributor 20f region 0:0x00008010805e0000
[    0.035604] GICv3: CPU47: using allocated LPI pending table @0x0000000100580000
[    0.035635] CPU47: Booted secondary processor 0x000000020f [0x431f0a11]
[    0.035745] smp: Brought up 1 node, 48 CPUs
[    0.036015] SMP: Total of 48 processors activated.
[    0.036022] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.036027] CPU features: detected: CRC32 instructions
[    0.036265] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.044965] CPU: All CPU(s) started at EL2
[    0.045123] alternatives: patching kernel code
[    0.058571] devtmpfs: initialized
[    0.061951] Registered cp15_barrier emulation handler
[    0.061976] Registered setend emulation handler
[    0.061984] KASLR disabled due to lack of seed
[    0.062154] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.062224] futex hash table entries: 16384 (order: 8, 1048576 bytes, linear)
[    0.063592] pinctrl core: initialized pinctrl subsystem
[    0.064122] SMBIOS 3.0.0 present.
[    0.064136] DMI: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    0.064654] NET: Registered protocol family 16
[    0.068677] DMA: preallocated 8192 KiB GFP_KERNEL pool for atomic allocations
[    0.069902] DMA: preallocated 8192 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.072709] DMA: preallocated 8192 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.072766] audit: initializing netlink subsys (disabled)
[    0.072987] audit: type=2000 audit(0.072:1): state=initialized audit_enabled=0 res=1
[    0.073545] thermal_sys: Registered thermal governor 'fair_share'
[    0.073551] thermal_sys: Registered thermal governor 'bang_bang'
[    0.073556] thermal_sys: Registered thermal governor 'step_wise'
[    0.073561] thermal_sys: Registered thermal governor 'user_space'
[    0.073565] thermal_sys: Registered thermal governor 'power_allocator'
[    0.073753] cpuidle: using governor ladder
[    0.074029] cpuidle: using governor menu
[    0.074319] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.075056] ASID allocator initialised with 65536 entries
[    0.075356] ACPI: bus type PCI registered
[    0.075364] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.075614] Serial: AMBA PL011 UART driver
[    0.084001] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.084009] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.084015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.084020] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.089191] ACPI: Added _OSI(Module Device)
[    0.089199] ACPI: Added _OSI(Processor Device)
[    0.089203] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.089208] ACPI: Added _OSI(Processor Aggregator Device)
[    0.089214] ACPI: Added _OSI(Linux-Dell-Video)
[    0.089219] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.089224] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.093110] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.095000] ACPI: Interpreter enabled
[    0.095006] ACPI: Using GIC for interrupt routing
[    0.095035] ACPI: MCFG table detected, 4 entries
[    0.118740] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1f])
[    0.118758] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.118904] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    0.119032] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability]
[    0.119502] acpi PNP0A08:00: ECAM area [mem 0x848000000000-0x848001ffffff] reserved by CAVA02C:00
[    0.119521] acpi PNP0A08:00: ECAM at [mem 0x848000000000-0x848001ffffff] for [bus 00-1f]
[    0.119665] PCI host bridge to bus 0000:00
[    0.119673] pci_bus 0000:00: root bus resource [mem 0x801000000000-0x807fffffffff window]
[    0.119681] pci_bus 0000:00: root bus resource [mem 0x838000000000-0x841fffffffff window]
[    0.119687] pci_bus 0000:00: root bus resource [mem 0x846000000000-0x847fffffffff window]
[    0.119694] pci_bus 0000:00: root bus resource [mem 0x868000000000-0x87e023ffffff window]
[    0.119700] pci_bus 0000:00: root bus resource [mem 0x87e026000000-0x87e0bfffffff window]
[    0.119707] pci_bus 0000:00: root bus resource [mem 0x87e0c6000000-0x87ffffffffff window]
[    0.119714] pci_bus 0000:00: root bus resource [bus 00-1f]
[    0.119743] pci 0000:00:01.0: [177d:a002] type 01 class 0x060400
[    0.119915] pci 0000:00:09.0: [177d:a018] type 00 class 0x120000
[    0.119952] pci 0000:00:09.0: BAR 0: [mem 0x87e040000000-0x87e0400fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    0.119962] pci 0000:00:09.0: VF BAR 0: [mem 0x840000800000-0x8400008fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.153606] pci 0000:00:09.0: VF(n) BAR0 space: [mem 0x840000800000-0x8400008fffff 64bit] (contains BAR0 for 1 VFs)
[    1.153757] pci 0000:00:10.0: [177d:a01b] type 00 class 0x0c0330
[    1.153798] pci 0000:00:10.0: BAR 0: [mem 0x868000000000-0x8680001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153808] pci 0000:00:10.0: BAR 4: [mem 0x868000200000-0x8680002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153936] pci 0000:00:11.0: [177d:a01b] type 00 class 0x0c0330
[    1.153972] pci 0000:00:11.0: BAR 0: [mem 0x869000000000-0x8690001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153982] pci 0000:00:11.0: BAR 4: [mem 0x869000200000-0x8690002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154109] pci 0000:00:14.0: [177d:a002] type 01 class 0x060400
[    1.154312] pci 0000:01:00.0: [177d:a001] type 00 class 0x088000
[    1.154349] pci 0000:01:00.0: BAR 0: [mem 0x87e0fc000000-0x87e0fc0fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154358] pci 0000:01:00.0: BAR 4: [mem 0x87e0fcf00000-0x87e0fcffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154484] pci 0000:01:00.1: [177d:a00e] type 00 class 0x088000
[    1.154522] pci 0000:01:00.1: BAR 0: [mem 0x87e006000000-0x87e0067fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154531] pci 0000:01:00.1: BAR 4: [mem 0x87e006f00000-0x87e006ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154651] pci 0000:01:01.3: [177d:a02b] type 00 class 0x0c8000
[    1.154687] pci 0000:01:01.3: BAR 0: [mem 0x87e005000000-0x87e0057fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154805] pci 0000:01:01.4: [177d:a010] type 00 class 0x058000
[    1.154841] pci 0000:01:01.4: BAR 0: [mem 0x87e009000000-0x87e0097fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154850] pci 0000:01:01.4: BAR 4: [mem 0x87e009f00000-0x87e009ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154971] pci 0000:01:06.0: [177d:a02e] type 00 class 0x058000
[    1.155012] pci 0000:01:06.0: BAR 0: [mem 0x87e050000000-0x87e0507fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155021] pci 0000:01:06.0: BAR 4: [mem 0x87e050f00000-0x87e050ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155149] pci 0000:01:06.1: [177d:a02e] type 00 class 0x058000
[    1.155186] pci 0000:01:06.1: BAR 0: [mem 0x87e051000000-0x87e0517fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155196] pci 0000:01:06.1: BAR 4: [mem 0x87e051f00000-0x87e051ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155316] pci 0000:01:06.2: [177d:a02e] type 00 class 0x058000
[    1.155354] pci 0000:01:06.2: BAR 0: [mem 0x87e052000000-0x87e0527fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155363] pci 0000:01:06.2: BAR 4: [mem 0x87e052f00000-0x87e052ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155485] pci 0000:01:06.3: [177d:a02e] type 00 class 0x058000
[    1.155522] pci 0000:01:06.3: BAR 0: [mem 0x87e053000000-0x87e0537fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155531] pci 0000:01:06.3: BAR 4: [mem 0x87e053f00000-0x87e053ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155651] pci 0000:01:06.4: [177d:a02e] type 00 class 0x058000
[    1.155687] pci 0000:01:06.4: BAR 0: [mem 0x87e054000000-0x87e0547fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155697] pci 0000:01:06.4: BAR 4: [mem 0x87e054f00000-0x87e054ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155817] pci 0000:01:06.5: [177d:a02e] type 00 class 0x058000
[    1.155854] pci 0000:01:06.5: BAR 0: [mem 0x87e055000000-0x87e0557fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155863] pci 0000:01:06.5: BAR 4: [mem 0x87e055f00000-0x87e055ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155986] pci 0000:01:06.6: [177d:a02e] type 00 class 0x058000
[    1.156023] pci 0000:01:06.6: BAR 0: [mem 0x87e056000000-0x87e0567fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156032] pci 0000:01:06.6: BAR 4: [mem 0x87e056f00000-0x87e056ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156152] pci 0000:01:06.7: [177d:a02e] type 00 class 0x058000
[    1.156188] pci 0000:01:06.7: BAR 0: [mem 0x87e057000000-0x87e0577fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156198] pci 0000:01:06.7: BAR 4: [mem 0x87e057f00000-0x87e057ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156318] pci 0000:01:07.0: [177d:a02f] type 00 class 0x058000
[    1.156358] pci 0000:01:07.0: BAR 0: [mem 0x87e058000000-0x87e0587fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156367] pci 0000:01:07.0: BAR 4: [mem 0x87e058f00000-0x87e058ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156496] pci 0000:01:07.1: [177d:a02f] type 00 class 0x058000
[    1.156534] pci 0000:01:07.1: BAR 0: [mem 0x87e059000000-0x87e0597fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156543] pci 0000:01:07.1: BAR 4: [mem 0x87e059f00000-0x87e059ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156664] pci 0000:01:07.2: [177d:a02f] type 00 class 0x058000
[    1.156701] pci 0000:01:07.2: BAR 0: [mem 0x87e05a000000-0x87e05a7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156710] pci 0000:01:07.2: BAR 4: [mem 0x87e05af00000-0x87e05affffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156831] pci 0000:01:07.3: [177d:a02f] type 00 class 0x058000
[    1.156868] pci 0000:01:07.3: BAR 0: [mem 0x87e05b000000-0x87e05b7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156877] pci 0000:01:07.3: BAR 4: [mem 0x87e05bf00000-0x87e05bffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157000] pci 0000:01:07.4: [177d:a030] type 00 class 0x058000
[    1.157037] pci 0000:01:07.4: BAR 0: [mem 0x87e05c000000-0x87e05c7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157046] pci 0000:01:07.4: BAR 4: [mem 0x87e05cf00000-0x87e05cffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157166] pci 0000:01:07.5: [177d:a030] type 00 class 0x058000
[    1.157203] pci 0000:01:07.5: BAR 0: [mem 0x87e05d000000-0x87e05d7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157212] pci 0000:01:07.5: BAR 4: [mem 0x87e05df00000-0x87e05dffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157341] pci 0000:01:07.6: [177d:a030] type 00 class 0x058000
[    1.157378] pci 0000:01:07.6: BAR 0: [mem 0x87e05e000000-0x87e05e7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157388] pci 0000:01:07.6: BAR 4: [mem 0x87e05ef00000-0x87e05effffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157509] pci 0000:01:07.7: [177d:a030] type 00 class 0x058000
[    1.157546] pci 0000:01:07.7: BAR 0: [mem 0x87e05f000000-0x87e05f7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157555] pci 0000:01:07.7: BAR 4: [mem 0x87e05ff00000-0x87e05fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157688] pci 0000:01:09.2: [177d:a012] type 00 class 0x0c8000
[    1.157730] pci 0000:01:09.2: BAR 0: [mem 0x87e0d2000000-0x87e0d27fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157739] pci 0000:01:09.2: BAR 4: [mem 0x87e0d2f00000-0x87e0d2ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157866] pci 0000:01:09.4: [177d:a012] type 00 class 0x0c8000
[    1.157903] pci 0000:01:09.4: BAR 0: [mem 0x87e0d4000000-0x87e0d47fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157912] pci 0000:01:09.4: BAR 4: [mem 0x87e0d4f00000-0x87e0d4ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158042] pci 0000:01:0a.0: [177d:a022] type 00 class 0x058000
[    1.158079] pci 0000:01:0a.0: BAR 0: [mem 0x87e088000000-0x87e0887fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158088] pci 0000:01:0a.0: BAR 4: [mem 0x87e088f00000-0x87e088ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158211] pci 0000:01:0a.1: [177d:a022] type 00 class 0x058000
[    1.158248] pci 0000:01:0a.1: BAR 0: [mem 0x87e089000000-0x87e0897fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158257] pci 0000:01:0a.1: BAR 4: [mem 0x87e089f00000-0x87e089ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158380] pci 0000:01:10.0: [177d:a026] type 00 class 0x028000
[    1.158416] pci 0000:01:10.0: BAR 0: [mem 0x87e0e0000000-0x87e0e03fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158425] pci 0000:01:10.0: BAR 4: [mem 0x87e0e0400000-0x87e0e07fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158548] pci 0000:01:10.1: [177d:a026] type 00 class 0x028000
[    1.158584] pci 0000:01:10.1: BAR 0: [mem 0x87e0e1000000-0x87e0e13fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158593] pci 0000:01:10.1: BAR 4: [mem 0x87e0e1400000-0x87e0e17fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158704] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.158871] pci 0000:02:00.0: [177d:a01d] type 00 class 0x010400
[    1.158908] pci 0000:02:00.0: BAR 0: [mem 0x870000000000-0x8700007fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158917] pci 0000:02:00.0: BAR 4: [mem 0x870000f00000-0x870000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.159031] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.159109] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.159123] pci 0000:00:14.0: PCI bridge to [bus 02]
[    1.159131] pci_bus 0000:00: resource 4 [mem 0x801000000000-0x807fffffffff window]
[    1.159138] pci_bus 0000:00: resource 5 [mem 0x838000000000-0x841fffffffff window]
[    1.159144] pci_bus 0000:00: resource 6 [mem 0x846000000000-0x847fffffffff window]
[    1.159150] pci_bus 0000:00: resource 7 [mem 0x868000000000-0x87e023ffffff window]
[    1.159157] pci_bus 0000:00: resource 8 [mem 0x87e026000000-0x87e0bfffffff window]
[    1.159163] pci_bus 0000:00: resource 9 [mem 0x87e0c6000000-0x87ffffffffff window]
[    1.159327] ACPI: PCI Root Bridge [PCI1] (domain 0001 [bus 00-1f])
[    1.159340] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.159482] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.159611] acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
[    1.160089] acpi PNP0A08:01: ECAM area [mem 0x849000000000-0x849001ffffff] reserved by CAVA02C:01
[    1.160105] acpi PNP0A08:01: ECAM at [mem 0x849000000000-0x849001ffffff] for [bus 00-1f]
[    1.160233] PCI host bridge to bus 0001:00
[    1.160241] pci_bus 0001:00: root bus resource [mem 0x810000000000-0x817fffffffff window]
[    1.160249] pci_bus 0001:00: root bus resource [bus 00-1f]
[    1.160277] pci 0001:00:08.0: [177d:a01c] type 00 class 0x010601
[    1.160310] pci 0001:00:08.0: BAR 0: [mem 0x814000000000-0x8140001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160319] pci 0001:00:08.0: BAR 4: [mem 0x814000200000-0x8140002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160447] pci 0001:00:09.0: [177d:a01c] type 00 class 0x010601
[    1.160483] pci 0001:00:09.0: BAR 0: [mem 0x815000000000-0x8150001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160492] pci 0001:00:09.0: BAR 4: [mem 0x815000200000-0x8150002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160621] pci 0001:00:0a.0: [177d:a01c] type 00 class 0x010601
[    1.160660] pci 0001:00:0a.0: BAR 0: [mem 0x816000000000-0x8160001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160669] pci 0001:00:0a.0: BAR 4: [mem 0x816000200000-0x8160002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160797] pci 0001:00:0b.0: [177d:a01c] type 00 class 0x010601
[    1.160833] pci 0001:00:0b.0: BAR 0: [mem 0x817000000000-0x8170001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160842] pci 0001:00:0b.0: BAR 4: [mem 0x817000200000-0x8170002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160976] pci_bus 0001:00: resource 4 [mem 0x810000000000-0x817fffffffff window]
[    1.161088] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-1f])
[    1.161101] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.161242] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.161371] acpi PNP0A08:02: _OSC: OS now controls [PCIeCapability]
[    1.161879] acpi PNP0A08:02: ECAM area [mem 0x84a000000000-0x84a001ffffff] reserved by CAVA02C:02
[    1.161895] acpi PNP0A08:02: ECAM at [mem 0x84a000000000-0x84a001ffffff] for [bus 00-1f]
[    1.162028] PCI host bridge to bus 0002:00
[    1.162035] pci_bus 0002:00: root bus resource [mem 0x842000000000-0x843fffffffff window]
[    1.162043] pci_bus 0002:00: root bus resource [bus 00-1f]
[    1.162067] pci 0002:00:02.0: [177d:a002] type 01 class 0x060400
[    1.162214] pci 0002:00:03.0: [177d:a01f] type 00 class 0x028000
[    1.162251] pci 0002:00:03.0: BAR 0: [mem 0x842000000000-0x84200000ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162260] pci 0002:00:03.0: BAR 2: [mem 0x842040000000-0x84207fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162269] pci 0002:00:03.0: BAR 4: [mem 0x842000f00000-0x842000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162462] pci 0002:01:00.0: [177d:a01e] type 00 class 0x020000
[    1.162499] pci 0002:01:00.0: BAR 0: [mem 0x843000000000-0x84303fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162509] pci 0002:01:00.0: BAR 4: [mem 0x843060000000-0x8430600fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162517] pci 0002:01:00.0: VF BAR 0: [mem 0x8430a0000000-0x8430a01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.162526] pci 0002:01:00.0: VF BAR 4: [mem 0x8430e0000000-0x8430e01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    2.178227] pci 0002:01:00.0: VF(n) BAR0 space: [mem 0x8430a0000000-0x8430afffffff 64bit] (contains BAR0 for 128 VFs)
[    2.178242] pci 0002:01:00.0: VF(n) BAR4 space: [mem 0x8430e0000000-0x8430efffffff 64bit] (contains BAR4 for 128 VFs)
[    2.178455] pci 0002:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.178481] pci 0002:00:02.0: PCI bridge to [bus 01]
[    2.178490] pci_bus 0002:00: resource 4 [mem 0x842000000000-0x843fffffffff window]
[    2.178603] ACPI: PCI Root Bridge [PCI3] (domain 0003 [bus 00-1f])
[    2.178616] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.178758] acpi PNP0A08:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.178887] acpi PNP0A08:03: _OSC: OS now controls [PCIeCapability]
[    2.179408] acpi PNP0A08:03: ECAM area [mem 0x84b000000000-0x84b001ffffff] reserved by CAVA02C:03
[    2.179425] acpi PNP0A08:03: ECAM at [mem 0x84b000000000-0x84b001ffffff] for [bus 00-1f]
[    2.179557] PCI host bridge to bus 0003:00
[    2.179564] pci_bus 0003:00: root bus resource [mem 0x818000000000-0x81ffffffffff window]
[    2.179572] pci_bus 0003:00: root bus resource [bus 00-1f]
[    2.179607] pci_bus 0003:00: resource 4 [mem 0x818000000000-0x81ffffffffff window]
[    2.179717] ACPI: PCI Root Bridge [PEM0] (domain 0004 [bus 1f-57])
[    2.179730] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.179870] acpi PNP0A08:04: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.180000] acpi PNP0A08:04: _OSC: OS now controls [PCIeCapability]
[    2.180009] acpi PNP0A08:04: MCFG quirk: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57] with thunder_pem_ecam_ops
[    2.180563] acpi PNP0A08:04: ECAM area [mem 0x88001f000000-0x880057ffffff] reserved by CAVA02B:00
[    2.181159] acpi PNP0A08:04: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57]
[    2.181195] Remapped I/O 0x0000883000000000 to [io  0x0000-0xffff window]
[    2.181331] PCI host bridge to bus 0004:1f
[    2.181339] pci_bus 0004:1f: root bus resource [io  0x0000-0xffff window]
[    2.181347] pci_bus 0004:1f: root bus resource [mem 0x881010000000-0x881fffffffff window] (bus address [0x10000000-0xfffffffff])
[    2.181355] pci_bus 0004:1f: root bus resource [mem 0x882000000000-0x882fffffffff pref window] (bus address [0x1000000000-0x1fffffffff])
[    2.181362] pci_bus 0004:1f: root bus resource [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.181369] pci_bus 0004:1f: root bus resource [bus 1f-57]
[    2.181395] pci 0004:1f:00.0: [177d:a100] type 01 class 0x060400
[    2.181411] pci 0004:1f:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
[    2.181421] pci 0004:1f:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.181436] pci 0004:1f:00.0: enabling Extended Tags
[    2.181461] pci 0004:1f:00.0: BAR 0: [mem 0x87e0c0f00000-0x87e0c0ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.181635] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.181732] pci 0004:20:00.0: [1a03:1150] type 01 class 0x060400
[    2.181783] pci 0004:20:00.0: Upstream bridge's Max Payload Size set to 128 (was 256, max 256)
[    2.181792] pci 0004:20:00.0: Max Payload Size set to 128 (was 128, max 128)
[    2.181801] pci 0004:20:00.0: enabling Extended Tags
[    2.181888] pci 0004:20:00.0: supports D1 D2
[    2.181894] pci 0004:20:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.182063] pci 0004:20:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.182130] pci_bus 0004:21: extended config space not accessible
[    2.182191] pci 0004:21:00.0: [1a03:2000] type 00 class 0x030000
[    2.182211] pci 0004:21:00.0: reg 0x10: [mem 0x881010000000-0x881010ffffff]
[    2.182235] pci 0004:21:00.0: reg 0x14: [mem 0x881011000000-0x88101101ffff]
[    2.182247] pci 0004:21:00.0: reg 0x18: [io  0x0000-0x007f]
[    2.182292] pci 0004:21:00.0: BAR 0: assigned to efifb
[    2.182341] pci 0004:21:00.0: supports D1 D2
[    2.182347] pci 0004:21:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.182542] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.182560] pci 0004:1f:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.182569] pci 0004:1f:00.0: BAR 6: assigned [mem 0x881011800000-0x88101180ffff pref]
[    2.182576] pci 0004:1f:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.182585] pci 0004:20:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.182592] pci 0004:20:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.182600] pci 0004:21:00.0: BAR 0: assigned [mem 0x881010000000-0x881010ffffff]
[    2.182609] pci 0004:21:00.0: BAR 1: assigned [mem 0x881011000000-0x88101101ffff]
[    2.182618] pci 0004:21:00.0: BAR 2: assigned [io  0x1000-0x107f]
[    2.182626] pci 0004:20:00.0: PCIbridge to [bus 20-21]
[    2.182653] pci 0004:1f:00.0:   bridge window [io  0x1000-0x1fff]
[    2.182660] pci 0004:1f:00.0:   bridge window [mem 0x881010000000-0x8810117fffff]
[    2.182668] pci_bus 0004:1f: resource 4 [io  0x0000-0xffff window]
[    2.182674] pci_bus 0004:1f: resource 5 [mem 0x881010000000-0x881fffffffff window]
[    2.182680] pci_bus 0004:1f: resource 6 [mem 0x882000000000-0x882fffffffff pref window]
[    2.182687] pci_bus 0004:1f: resource 7 [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.182693] pci_bus 0004:20: resource 0 [io  0x1000-0x1fff]
[    2.182699] pci_bus 0004:20: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.182706] pci_bus 0004:21: resource 0 [io  0x1000-0x1fff]
[    2.182712] pci_bus 0004:21: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.182830] ACPI: PCI Interrupt Link [LN0A] (IRQs *48)
[    2.182882] ACPI: PCI Interrupt Link [LN0B] (IRQs *49)
[    2.182929] ACPI: PCI Interrupt Link [LN0C] (IRQs *50)
[    2.182976] ACPI: PCI Interrupt Link [LN0D] (IRQs *51)
[    2.183270] ARMH0011:00: ttyAMA0 at MMIO 0x87e024000000 (irq = 21, base_baud = 0) is a SBSA
[    6.410105] printk: console [ttyAMA0] enabled
[    6.417695] ARMH0011:01: ttyAMA1 at MMIO 0x87e025000000 (irq = 22, base_baud = 0) is a SBSA
[    6.432839] iommu: Default domain type: Translated 
[    6.438451] SCSI subsystem initialized
[    6.442472] pci 0004:21:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    6.450830] pci 0004:21:00.0: vgaarb: bridge control possible
[    6.456570] pci 0004:21:00.0: vgaarb: setting as boot device (VGA legacy resources not available)
[    6.465432] vgaarb: loaded
[    6.468188] ACPI: bus type USB registered
[    6.472245] usbcore: registered new interface driver usbfs
[    6.477756] usbcore: registered new interface driver hub
[    6.483112] usbcore: registered new device driver usb
[    6.488250] pps_core: LinuxPPS API ver. 1 registered
[    6.493212] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    6.502346] PTP clock support registered
[    6.506465] EDAC MC: Ver: 3.0.0
[    6.510054] Registered efivars operations
[    6.515854] NetLabel: Initializing
[    6.519257] NetLabel:  domain hash size = 128
[    6.523605] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    6.529294] NetLabel:  unlabeled traffic allowed by default
[    6.535499] clocksource: Switched to clocksource arch_sys_counter
[    6.581508] VFS: Disk quotas dquot_6.6.0
[    6.585491] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    6.592632] pnp: PnP ACPI init
[    6.596310] system 00:00: [mem 0x848000000000-0x848001ffffff] could not be reserved
[    6.604162] system 00:01: [mem 0x849000000000-0x849001ffffff] could not be reserved
[    6.612012] system 00:02: [mem 0x84a000000000-0x84a001ffffff] could not be reserved
[    6.619856] system 00:03: [mem 0x84b000000000-0x84b001ffffff] could not be reserved
[    6.627713] system 00:04: [mem 0x87e0c0000000-0x87e0c0ffffff] could not be reserved
[    6.635374] system 00:04: [mem 0x88001f000000-0x880057ffffff] could not be reserved
[    6.645259] pnp: PnP ACPI: found 5 devices
[    6.655213] NET: Registered protocol family 2
[    6.660206] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    6.669442] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    6.679300] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    6.687069] TCP: Hash tables configured (established 524288 bind 65536)
[    6.694018] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, linear)
[    6.702227] UDP hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.709784] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.717920] NET: Registered protocol family 1
[    6.722293] NET: Registered protocol family 44
[    6.726930] PCI: CLS 0 bytes, default 64
[    6.730959] Trying to unpack rootfs image as initramfs...
[    8.239745] Freeing initrd memory: 106132K
[    8.247176] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available
[    8.255210] kvm [1]: IPA Size Limit: 48 bits
[    8.261136] kvm [1]: GICv3: no GICV resource entry
[    8.265933] kvm [1]: disabling GICv2 emulation
[    8.270367] kvm [1]: GICv3 sysreg trapping enabled ([G0G1], reduced performance)
[    8.277767] kvm [1]: GIC system register CPU interface enabled
[    8.284738] kvm [1]: vgic interrupt IRQ9
[    8.289892] kvm [1]: Hyp mode initialized successfully
[    8.297815] Initialise system trusted keyrings
[    8.302285] Key type blacklist registered
[    8.306483] workingset: timestamp_bits=40 max_order=24 bucket_order=0
[    8.317158] zbud: loaded
[    8.320884] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    8.327253] fuse: init (API version 7.33)
[    8.331648] integrity: Platform Keyring initialized
[    8.359114] Key type asymmetric registered
[    8.363210] Asymmetric key parser 'x509' registered
[    8.368099] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)
[    8.375627] io scheduler mq-deadline registered
[    8.383525] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    8.391174] efifb: probing for efifb
[    8.394782] efifb: No BGRT, not showing boot graphics
[    8.399826] efifb: framebuffer at 0x881010000000, using 3072k, total 3072k
[    8.406692] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    8.412690] efifb: scrolling: redraw
[    8.416256] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    8.421952] fbcon: Deferring console take-over
[    8.426403] fb0: EFI VGA frame buffer device
[    8.431351] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[    8.439771] ACPI: Power Button [PWRB]
[    8.444895] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address.
[    8.455456] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
[    8.464493] Mem abort info:
[    8.467274]   ESR = 0x96000004
[    8.470317]   EC = 0x25: DABT (current EL), IL = 32 bits
[    8.475618]   SET = 0, FnV = 0
[    8.478661]   EA = 0, S1PTW = 0
[    8.481790] Data abort info:
[    8.484659]   ISV = 0, ISS = 0x00000004
[    8.488482]   CM = 0, WnR = 0
[    8.491439] [0000000000000028] user address but active_mm is swapper
[    8.497782] Internal error: Oops: 96000004 [#1] SMP
[    8.502649] Modules linked in:
[    8.505698] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.11.0-13-generic #14-Ubuntu
[    8.513256] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    8.520119] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
[    8.526115] pc : __ipi_send_mask+0x60/0x114
[    8.530296] lr : smp_cross_call+0x3c/0xdc
[    8.534299] sp : ffff80001279bc90
[    8.537602] x29: ffff80001279bc90 x28: 000000000000000a 
[    8.542907] x27: ffff8000114eb2d8 x26: ffff8000122e5108 
[    8.548211] x25: 0000000000000001 x24: 0000000000000000 
[    8.553515] x23: ffff8000122ee000 x22: ffff8000123f8448 
[    8.558820] x21: ffff800010fd9a28 x20: ffff800010fd9a28 
[    8.564124] x19: ffff000100157c60 x18: 0000000000000020 
[    8.569428] x17: 0000000001f80e3b x16: 000000005628f43e 
[    8.574732] x15: ffffffffffffffff x14: 0000000000000000 
[    8.580036] x13: 003d090000000000 x12: 00003d0900000000 
[    8.585341] x11: 0000000000000000 x10: 00003d0900000000 
[    8.590645] x9 : ffff800010028c40 x8 : 0000000000000001 
[    8.595949] x7 : 00000000ffffffff x6 : ffff000ff61d0318 
[    8.601253] x5 : ffff80001161f2a0 x4 : ffff80001161f370 
[    8.606557] x3 : ffff800010fd99e8 x2 : 0000000000000000 
[    8.611861] x1 : ffff800010fd9a28 x0 : 0000000000000000 
[    8.617166] Call trace:
[    8.619602]  __ipi_send_mask+0x60/0x114
[    8.623428]  smp_cross_call+0x3c/0xdc
[    8.627081]  smp_send_reschedule+0x3c/0x50
[    8.631169]  resched_curr+0x5c/0xb0
[    8.634649]  check_preempt_curr+0x58/0x90
[    8.638649]  ttwu_do_wakeup+0x2c/0x1a0
[    8.642389]  ttwu_do_activate+0x7c/0x114
[    8.646301]  try_to_wake_up+0x2cc/0x5b0
[    8.650128]  wake_up_process+0x24/0x30
[    8.653867]  swake_up_one+0x48/0x9c
[    8.657346]  rcu_gp_kthread_wake+0x68/0x8c
[    8.661433]  rcu_accelerate_cbs_unlocked+0xb4/0xf0
[    8.666214]  rcu_core+0x208/0x230
[    8.669520]  rcu_core_si+0x1c/0x30
[    8.672911]  __do_softirq+0x128/0x3a4
[    8.676563]  irq_exit+0xc4/0xec
[    8.679695]  __handle_domain_irq+0x8c/0xec
[    8.683783]  gic_handle_irq+0x84/0xfc
[    8.687435]  el1_irq+0xc0/0x180
[    8.690566]  ww_mutex_lock+0x320/0x540
[    8.694306]  kobject_uevent_env+0x324/0x4c4
[    8.698481]  kobject_uevent+0x14/0x20
[    8.702133]  driver_register+0xc0/0x140
[    8.705961]  __platform_driver_register+0x34/0x40
[    8.710656]  scpsys_drv_init+0x28/0x34
[    8.714398]  do_one_initcall+0x50/0x290
[    8.718224]  do_initcalls+0x104/0x144
[    8.721881]  kernel_init_freeable+0x174/0x1c0
[    8.726228]  kernel_init+0x20/0x134
[    8.729708]  ret_from_fork+0x10/0x18
[    8.733278] Code: a90363f7 aa0103f5 b0010db7 f9401260 (b9402800) 
[    8.739394] ---[ end trace 4c5c8d7f98c860a1 ]---
[    8.744001] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[    8.750883] SMP: stopping secondary CPUs
[    9.798801] SMP: failed to stop secondary CPUs 2,8
[    9.803583] Kernel Offset: disabled
[    9.807059] CPU features: 0x00040002,69101108
[    9.811405] Memory Limit: none
[    9.814467] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[dann frazier]

On Tue, Apr 20, 2021 at 02:37:10PM -0600, dann frazier wrote:
> On Tue, May 19, 2020 at 05:17:52PM +0100, Marc Zyngier wrote:
> > Change the way we deal with GIC SGIs by turning them into proper
> > IRQs, and calling into the arch code to register the interrupt range
> > instead of a callback.
> > 
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> 
> hey Marc,
> 
>   I bisected a boot failure on our Gigabyte R120-T33 systems (ThunderX
> CN88XX) down to this commit, but only when running in ACPI mode. See below:
> 
> 
> EFI stub: Booting Linux Kernel...
> EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services and installing virtual address map...
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
> [    0.000000] Linux version 5.11.0-13-generic (buildd@bos02-arm64-067) (gcc (Ubuntu 10.2.1-23ubuntu1) 10.2.1 20210312, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #14-Ubuntu SMP Fri Mar 19 16:57:35 UTC 2021 (Ubuntu 5.11.0-13.14-generic 5.11.7)

Sorry, realized I posted a log from an Ubuntu kernel. Here's an
upstream one:

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
[    0.000000] Linux version 5.12.0-rc8 (ubuntu@seidel) (gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #19 SMP Tue Apr 20 21:09:04 UTC 2021
[    0.000000] Machine model: Cavium ThunderX CN88XX board
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi: ESRT=0xffce0ff18 SMBIOS 3.0=0xfffb0000 ACPI 2.0=0xffec60000 MEMRESERVE=0xffc889d98 
[    0.000000] esrt: Reserving ESRT space from 0x0000000ffce0ff18 to 0x0000000ffce0ff50.
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x0000000FFEC60000 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000FFEC60028 00008C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000FFEC600B8 000114 (v06 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000FFEC601D0 00220B (v02 CAVIUM THUNDERX 00000001 INTL 20130517)
[    0.000000] ACPI: SPMI 0x0000000FFEC623E0 000041 (v05 ALASKA A M I    00000000 AMI. 00000000)
[    0.000000] ACPI: FIDT 0x0000000FFEC62428 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: APIC 0x0000000FFEC624C8 000F68 (v03 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: DBG2 0x0000000FFEC63430 000067 (v01 CAVIUM CN88XDBG 00000000 INTL 20150619)
[    0.000000] ACPI: GTDT 0x0000000FFEC63498 0000E0 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: IORT 0x0000000FFEC63578 0013D4 (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: MCFG 0x0000000FFEC64950 00006C (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SSDT 0x0000000FFEC649C0 00089C (v02 CAVIUM NETWORK  00000001 INTL 20150619)
[    0.000000] ACPI: OEM1 0x0000000FFEC65260 0001E8 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SLIT 0x0000000FFEC65448 000030 (v01 CAVIUM TEMPLATE 00000001 INTL 20150619)
[    0.000000] ACPI: SPCR 0x0000000FFEC65478 000050 (v02 A M I  APTIO V  01072009 AMI. 0005000B)
[    0.000000] ACPI: BGRT 0x0000000FFEC654C8 000038 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0x87e024000000,115200
[    0.000000] efi_bgrt: Ignoring BGRT: Incorrect BMP magic number 0x3707 (expected 0x4d42)
[    0.000000] NUMA: Failed to initialise from firmware
[    0.000000] NUMA: Faking a node at [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000] NUMA: NODE_DATA [mem 0xff677c480-0xff6780fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000500000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x0000000fff0fffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000500000-0x0000000000dfffff]
[    0.000000]   node   0: [mem 0x0000000000e00000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000010000000-0x00000000102fffff]
[    0.000000]   node   0: [mem 0x0000000010300000-0x00000000fff9ffff]
[    0.000000]   node   0: [mem 0x00000000fffa0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x0000000fee6affff]
[    0.000000]   node   0: [mem 0x0000000fee6b0000-0x0000000fee6cffff]
[    0.000000]   node   0: [mem 0x0000000fee6d0000-0x0000000ffac2ffff]
[    0.000000]   node   0: [mem 0x0000000ffac30000-0x0000000ffb0affff]
[    0.000000]   node   0: [mem 0x0000000ffb0b0000-0x0000000ffc8bffff]
[    0.000000]   node   0: [mem 0x0000000ffc8c0000-0x0000000ffc91ffff]
[    0.000000]   node   0: [mem 0x0000000ffc920000-0x0000000ffca3ffff]
[    0.000000]   node   0: [mem 0x0000000ffca40000-0x0000000ffca4ffff]
[    0.000000]   node   0: [mem 0x0000000ffca50000-0x0000000ffcdbffff]
[    0.000000]   node   0: [mem 0x0000000ffcdc0000-0x0000000ffd12ffff]
[    0.000000]   node   0: [mem 0x0000000ffd130000-0x0000000ffec2ffff]
[    0.000000]   node   0: [mem 0x0000000ffec30000-0x0000000ffec5ffff]
[    0.000000]   node   0: [mem 0x0000000ffec60000-0x0000000ffec6ffff]
[    0.000000]   node   0: [mem 0x0000000ffec70000-0x0000000ffed1ffff]
[    0.000000]   node   0: [mem 0x0000000ffed20000-0x0000000fff0fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000]   Normal zone: 256 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x00000000fdc00000
[    0.000000] psci: probing for conduit method from ACPI.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] ACPI: SRAT not present
[    0.000000] percpu: Embedded 33 pages/cpu s98072 r8192 d28904 u135168
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Software prefetching using PRFM
[    0.000000] CPU features: detected: Cavium erratum 27456
[    0.000000] CPU features: detected: Cavium erratum 30115
[    0.000000] CPU features: kernel page table isolation forced OFF by ARM64_WORKAROUND_CAVIUM_27456
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16510032
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-dannf root=UUID=76e81d96-78e4-4432-a490-bb852f6111e8 ro sysrq_always_enabled acpi=force
[    0.000000] sysrq: sysrq always enabled.
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 192512 bytes
[    0.000000] printk: log_buf_len min size: 262144 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 255872(97%)
[    0.000000] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f9c00000-0x00000000fdc00000] (64MB)
[    0.000000] Memory: 65572072K/67088384K available (15616K kernel code, 3626K rwdata, 11808K rodata, 7616K init, 1157K bss, 1483544K reserved, 32768K cma-reserved)
[    0.000000] random: get_random_u64 called from __kmem_cache_create+0x38/0x5b4 with crng_init=0
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=48, Nodes=1
[    0.000000] ftrace: allocating 49728 entries in 195 pages
[    0.000000] ftrace: allocated 195 pages with 4 groups
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=48.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=48
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GIC: enabling workaround for GICv3: Cavium erratum 38539
[    0.000000] GICv3: 128 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000801080000000
[    0.000000] ACPI: SRAT not present
[    0.000000] ITS [mem 0x801000020000-0x80100003ffff]
[    0.000000] ITS@0x0000801000020000: Devices Table too large, reduce ids 21->19
[    0.000000] ITS@0x0000801000020000: allocated 524288 Devices @100800000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000100270000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000100280000
[    0.000000] arch_timer: Failed to initialize memory-mapped timer.
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000000] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000149] Console: colour dummy device 80x25
[    0.000230] ACPI: Core revision 20210105
[    0.000451] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.000462] pid_max: default: 49152 minimum: 384
[    0.000575] LSM: Security Framework initializing
[    0.000590] Yama: becoming mindful.
[    0.000648] AppArmor: AppArmor initialized
[    0.000801] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000866] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.001656] ACPI PPTT: No PPTT table found, CPU and cache topology may be inaccurate
[    0.002242] rcu: Hierarchical SRCU implementation.
[    0.002993] Platform MSI: ITS@0x801000020000 domain created
[    0.003007] PCI/MSI: ITS@0x801000020000 domain created
[    0.003019] fsl-mc MSI: ITS@0x801000020000 domain created
[    0.003036] Remapping and enabling EFI services.
[    0.005145] smp: Bringing up secondary CPUs ...
[    0.005629] Detected VIPT I-cache on CPU1
[    0.005644] GICv3: CPU1: found redistributor 1 region 0:0x0000801080020000
[    0.005669] GICv3: CPU1: using allocated LPI pending table @0x0000000100290000
[    0.005690] CPU1: Booted secondary processor 0x0000000001 [0x431f0a11]
[    0.006245] Detected VIPT I-cache on CPU2
[    0.006255] GICv3: CPU2: found redistributor 2 region 0:0x0000801080040000
[    0.006279] GICv3: CPU2: using allocated LPI pending table @0x00000001002a0000
[    0.006298] CPU2: Booted secondary processor 0x0000000002 [0x431f0a11]
[    0.006840] Detected VIPT I-cache on CPU3
[    0.006851] GICv3: CPU3: found redistributor 3 region 0:0x0000801080060000
[    0.006876] GICv3: CPU3: using allocated LPI pending table @0x00000001002b0000
[    0.006896] CPU3: Booted secondary processor 0x0000000003 [0x431f0a11]
[    0.007435] Detected VIPT I-cache on CPU4
[    0.007446] GICv3: CPU4: found redistributor 4 region 0:0x0000801080080000
[    0.007470] GICv3: CPU4: using allocated LPI pending table @0x00000001002c0000
[    0.007490] CPU4: Booted secondary processor 0x0000000004 [0x431f0a11]
[    0.008034] Detected VIPT I-cache on CPU5
[    0.008045] GICv3: CPU5: found redistributor 5 region 0:0x00008010800a0000
[    0.008069] GICv3: CPU5: using allocated LPI pending table @0x00000001002d0000
[    0.008089] CPU5: Booted secondary processor 0x0000000005 [0x431f0a11]
[    0.008626] Detected VIPT I-cache on CPU6
[    0.008637] GICv3: CPU6: found redistributor 6 region 0:0x00008010800c0000
[    0.008664] GICv3: CPU6: using allocated LPI pending table @0x00000001002e0000
[    0.008685] CPU6: Booted secondary processor 0x0000000006 [0x431f0a11]
[    0.009228] Detected VIPT I-cache on CPU7
[    0.009239] GICv3: CPU7: found redistributor 7 region 0:0x00008010800e0000
[    0.009265] GICv3: CPU7: using allocated LPI pending table @0x00000001002f0000
[    0.009286] CPU7: Booted secondary processor 0x0000000007 [0x431f0a11]
[    0.009825] Detected VIPT I-cache on CPU8
[    0.009837] GICv3: CPU8: found redistributor 8 region 0:0x0000801080100000
[    0.009862] GICv3: CPU8: using allocated LPI pending table @0x0000000100300000
[    0.009883] CPU8: Booted secondary processor 0x0000000008 [0x431f0a11]
[    0.010436] Detected VIPT I-cache on CPU9
[    0.010447] GICv3: CPU9: found redistributor 9 region 0:0x0000801080120000
[    0.010473] GICv3: CPU9: using allocated LPI pending table @0x0000000100310000
[    0.010494] CPU9: Booted secondary processor 0x0000000009 [0x431f0a11]
[    0.011040] Detected VIPT I-cache on CPU10
[    0.011052] GICv3: CPU10: found redistributor a region 0:0x0000801080140000
[    0.011078] GICv3: CPU10: using allocated LPI pending table @0x0000000100320000
[    0.011099] CPU10: Booted secondary processor 0x000000000a [0x431f0a11]
[    0.011647] Detected VIPT I-cache on CPU11
[    0.011659] GICv3: CPU11: found redistributor b region 0:0x0000801080160000
[    0.011686] GICv3: CPU11: using allocated LPI pending table @0x0000000100330000
[    0.011707] CPU11: Booted secondary processor 0x000000000b [0x431f0a11]
[    0.012266] Detected VIPT I-cache on CPU12
[    0.012278] GICv3: CPU12: found redistributor c region 0:0x0000801080180000
[    0.012304] GICv3: CPU12: using allocated LPI pending table @0x0000000100340000
[    0.012326] CPU12: Booted secondary processor 0x000000000c [0x431f0a11]
[    0.012889] Detected VIPT I-cache on CPU13
[    0.012901] GICv3: CPU13: found redistributor d region 0:0x00008010801a0000
[    0.012928] GICv3: CPU13: using allocated LPI pending table @0x0000000100350000
[    0.012950] CPU13: Booted secondary processor 0x000000000d [0x431f0a11]
[    0.013498] Detected VIPT I-cache on CPU14
[    0.013511] GICv3: CPU14: found redistributor e region 0:0x00008010801c0000
[    0.013537] GICv3: CPU14: using allocated LPI pending table @0x0000000100360000
[    0.013560] CPU14: Booted secondary processor 0x000000000e [0x431f0a11]
[    0.014110] Detected VIPT I-cache on CPU15
[    0.014124] GICv3: CPU15: found redistributor f region 0:0x00008010801e0000
[    0.014152] GICv3: CPU15: using allocated LPI pending table @0x0000000100370000
[    0.014175] CPU15: Booted secondary processor 0x000000000f [0x431f0a11]
[    0.014719] Detected VIPT I-cache on CPU16
[    0.014732] GICv3: CPU16: found redistributor 100 region 0:0x0000801080200000
[    0.014759] GICv3: CPU16: using allocated LPI pending table @0x0000000100380000
[    0.014782] CPU16: Booted secondary processor 0x0000000100 [0x431f0a11]
[    0.015323] Detected VIPT I-cache on CPU17
[    0.015337] GICv3: CPU17: found redistributor 101 region 0:0x0000801080220000
[    0.015363] GICv3: CPU17: using allocated LPI pending table @0x0000000100390000
[    0.015386] CPU17: Booted secondary processor 0x0000000101 [0x431f0a11]
[    0.015955] Detected VIPT I-cache on CPU18
[    0.015969] GICv3: CPU18: found redistributor 102 region 0:0x0000801080240000
[    0.015996] GICv3: CPU18: using allocated LPI pending table @0x00000001003a0000
[    0.016019] CPU18: Booted secondary processor 0x0000000102 [0x431f0a11]
[    0.016582] Detected VIPT I-cache on CPU19
[    0.016595] GICv3: CPU19: found redistributor 103 region 0:0x0000801080260000
[    0.016623] GICv3: CPU19: using allocated LPI pending table @0x00000001003b0000
[    0.016647] CPU19: Booted secondary processor 0x0000000103 [0x431f0a11]
[   0.017266] CPU20: Booted secondary processor 0x0000000104 [0x431f0a11]
[    0.017811] Detected VIPT I-cache on CPU21
[    0.017826] GICv3: CPU21: found redistributor 105 region 0:0x00008010802a0000
[    0.017853] GICv3: CPU21: using allocated LPI pending table @0x00000001003d0000
[    0.017877] CPU21: Booted secondary processor 0x0000000105 [0x431f0a11]
[    0.018439] Detected VIPT I-cache on CPU22
[    0.018453] GICv3: CPU22: found redistributor 106 region 0:0x00008010802c0000
[    0.018481] GICv3: CPU22: using allocated LPI pending table @0x00000001003e0000
[    0.018505] CPU22: Booted secondary processor 0x0000000106 [0x431f0a11]
[    0.019068] Detected VIPT I-cache on CPU23
[    0.019083] GICv3: CPU23: found redistributor 107 region 0:0x00008010802e0000
[    0.019111] GICv3: CPU23: using allocated LPI pending table @0x00000001003f0000
[    0.019135] CPU23: Booted secondary processor 0x0000000107 [0x431f0a11]
[    0.019684] Detected VIPT I-cache on CPU24
[    0.019698] GICv3: CPU24: found redistributor 108 region 0:0x0000801080300000
[    0.019726] GICv3: CPU24: using allocated LPI pending table @0x0000000100c00000
[    0.019751] CPU24: Booted secondary processor 0x0000000108 [0x431f0a11]
[    0.020328] Detected VIPT I-cache on CPU25
[    0.020343] GICv3: CPU25: found redistributor 109 region 0:0x0000801080320000
[    0.020372] GICv3: CPU25: using allocated LPI pending table @0x0000000100c10000
[    0.020396] CPU25: Booted secondary processor 0x0000000109 [0x431f0a11]
[    0.020948] Detected VIPT I-cache on CPU26
[    0.020963] GICv3: CPU26: found redistributor 10a region 0:0x0000801080340000
[    0.020992] GICv3: CPU26: using allocated LPI pending table @0x0000000100c20000
[    0.021017] CPU26: Booted secondary processor 0x000000010a [0x431f0a11]
[    0.021585] Detected VIPT I-cache on CPU27
[    0.021601] GICv3: CPU27: found redistributor 10b region 0:0x0000801080360000
[    0.021629] GICv3: CPU27: using allocated LPI pending table @0x0000000100c30000
[    0.021655] CPU27: Booted secondary processor 0x000000010b [0x431f0a11]
[    0.022205] Detected VIPT I-cache on CPU28
[    0.022220] GICv3: CPU28: found redistributor 10c region 0:0x0000801080380000
[    0.022248] GICv3: CPU28: using allocated LPI pending table @0x0000000100c40000
[    0.022274] CPU28: Booted secondary processor 0x000000010c [0x431f0a11]
[    0.022835] Detected VIPT I-cache on CPU29
[    0.022851] GICv3: CPU29: found redistributor 10d region 0:0x00008010803a0000
[    0.022880] GICv3: CPU29: using allocated LPI pending table @0x0000000100c50000
[    0.022906] CPU29: Booted secondary processor 0x000000010d [0x431f0a11]
[    0.023469] Detected VIPT I-cache on CPU30
[    0.023484] GICv3: CPU30: found redistributor 10e region 0:0x00008010803c0000
[    0.023513] GICv3: CPU30: using allocated LPI pending table @0x0000000100c60000
[    0.023539] CPU30: Booted secondary processor 0x000000010e [0x431f0a11]
[    0.024102] Detected VIPT I-cache on CPU31
[    0.024118] GICv3: CPU31: found redistributor 10f region 0:0x00008010803e0000
[    0.024147] GICv3: CPU31: using allocated LPI pending table @0x0000000100c70000
[    0.024174] CPU31: Booted secondary processor 0x000000010f [0x431f0a11]
[    0.024748] Detected VIPT I-cache on CPU32
[    0.024764] GICv3: CPU32: found redistributor 200 region 0:0x0000801080400000
[    0.024793] GICv3: CPU32: using allocated LPI pending table @0x0000000100c80000
[    0.024820] CPU32: Booted secondary processor 0x0000000200 [0x431f0a11]
[    0.025388] Detected VIPT I-cache on CPU33
[    0.025405] GICv3: CPU33: found redistributor 201 region 0:0x0000801080420000
[    0.025434] GICv3: CPU33: using allocated LPI pending table @0x0000000100c90000
[    0.025461] CPU33: Booted secondary processor 0x0000000201 [0x431f0a11]
[    0.026028] Detected VIPT I-cache on CPU34
[    0.026044] GICv3: CPU34: found redistributor 202 region 0:0x0000801080440000
[    0.026074] GICv3: CPU34: using allocated LPI pending table @0x0000000100ca0000
[    0.026101] CPU34: Booted secondary processor 0x0000000202 [0x431f0a11]
[    0.026670] Detected VIPT I-cache on CPU35
[    0.026687] GICv3: CPU35: found redistributor 203 region 0:0x0000801080460000
[    0.026716] GICv3: CPU35: using allocated LPI pending table @0x0000000100cb0000
[    0.026744] CPU35: Booted secondary processor 0x0000000203 [0x431f0a11]
[    0.027305] Detected VIPT I-cache on CPU36
[    0.027322] GICv3: CPU36: found redistributor 204 region 0:0x0000801080480000
[    0.027352] GICv3: CPU36: using allocated LPI pending table @0x0000000100cc0000
[    0.027380] CPU36: Booted secondary processor 0x0000000204 [0x431f0a11]
[    0.027942] Detected VIPT I-cache on CPU37
[    0.027959] GICv3: CPU37: found redistributor 205 region 0:0x00008010804a0000
[    0.027989] GICv3: CPU37: using allocated LPI pending table @0x0000000100cd0000
[    0.028018] CPU37: Booted secondary processor 0x0000000205 [0x431f0a11]
[    0.028597] Detected VIPT I-cache on CPU38
[    0.028614] GICv3: CPU38: found redistributor 206 region 0:0x00008010804c0000
[    0.028644] GICv3: CPU38: using allocated LPI pending table @0x0000000100ce0000
[    0.028672] CPU38: Booted secondary processor 0x0000000206 [0x431f0a11]
[    0.029233] Detected VIPT I-cache on CPU39
[    0.029251] GICv3: CPU39: found redistributor 207 region 0:0x00008010804e0000
[    0.029282] GICv3: CPU39: using allocated LPI pending table @0x0000000100cf0000
[    0.029310] CPU39: Booted secondary processor 0x0000000207 [0x431f0a11]
[    0.029885] Detected VIPT I-cache on CPU40
[    0.029902] GICv3: CPU40: found redistributor 208 region 0:0x0000801080500000
[    0.029932] GICv3: CPU40: using allocated LPI pending table @0x0000000100d00000
[    0.029961] CPU40: Booted secondary processor 0x0000000208 [0x431f0a11]
[    0.030544] Detected VIPT I-cache on CPU41
[    0.030562] GICv3: CPU41: found redistributor 209 region 0:0x0000801080520000
[    0.030592] GICv3: CPU41: using allocated LPI pending table @0x0000000100d10000
[    0.030622] CPU41: Booted secondary processor 0x0000000209 [0x431f0a11]
[    0.031188] Detected VIPT I-cache on CPU42
[    0.031205] GICv3: CPU42: found redistributor 20a region 0:0x0000801080540000
[    0.031236] GICv3: CPU42: using allocated LPI pending table @0x0000000100d20000
[    0.031266] CPU42: Booted secondary processor 0x000000020a [0x431f0a11]
[    0.031845] Detected VIPT I-cache on CPU43
[    0.031863] GICv3: CPU43: found redistributor 20b region 0:0x0000801080560000
[    0.031896] GICv3: CPU43: using allocated LPI pending table @0x0000000100d30000
[    0.031925] CPU43: Booted secondary processor 0x000000020b [0x431f0a11]
[    0.032520] Detected VIPT I-cache on CPU44
[    0.032538] GICv3: CPU44: found redistributor 20c region 0:0x0000801080580000
[    0.032569] GICv3: CPU44: using allocated LPI pending table @0x0000000100d40000
[    0.032599] CPU44: Booted secondary processor 0x000000020c [0x431f0a11]
[    0.033181] Detected VIPT I-cache on CPU45
[    0.033200] GICv3: CPU45: found redistributor 20d region 0:0x00008010805a0000
[    0.033231] GICv3: CPU45: using allocated LPI pending table @0x0000000100d50000
[    0.033261] CPU45: Booted secondary processor 0x000000020d [0x431f0a11]
[    0.033833] Detected VIPT I-cache on CPU46
[    0.033852] GICv3: CPU46: found redistributor 20e region 0:0x00008010805c0000
[    0.033883] GICv3: CPU46: using allocated LPI pending table @0x0000000100d60000
[    0.033913] CPU46: Booted secondary processor 0x000000020e [0x431f0a11]
[    0.035252] Detected VIPT I-cache on CPU47
[    0.035271] GICv3: CPU47: found redistributor 20f region 0:0x00008010805e0000
[    0.035303] GICv3: CPU47: using allocated LPI pending table @0x0000000100d70000
[    0.035334] CPU47: Booted secondary processor 0x000000020f [0x431f0a11]
[    0.035443] smp: Brought up 1 node, 48 CPUs
[    0.035708] SMP: Total of 48 processors activated.
[    0.035715] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.035720] CPU features: detected: CRC32 instructions
[    0.035948] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.044445] CPU: All CPU(s) started at EL2
[    0.044603] alternatives: patching kernel code
[    0.058479] devtmpfs: initialized
[    0.078107] Registered cp15_barrier emulation handler
[    0.078132] Registered setend emulation handler
[    0.078139] KASLR disabled due to lack of seed
[    0.078309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.078378] futex hash table entries: 16384 (order: 8, 1048576 bytes, linear)
[    0.079795] pinctrl core: initialized pinctrl subsystem
[    0.080358] SMBIOS 3.0.0 present.
[    0.080372] DMI: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    0.080861] NET: Registered protocol family 16
[    0.082637] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.083107] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.083586] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.083620] audit: initializing netlink subsys (disabled)
[    0.083779] audit: type=2000 audit(0.080:1): state=initialized audit_enabled=0 res=1
[    0.084319] thermal_sys: Registered thermal governor 'fair_share'
[    0.084325] thermal_sys: Registered thermal governor 'bang_bang'
[    0.084330] thermal_sys: Registered thermal governor 'step_wise'
[    0.084335] thermal_sys: Registered thermal governor 'user_space'
[    0.084339] thermal_sys: Registered thermal governor 'power_allocator'
[    0.084521] cpuidle: using governor ladder
[    0.084784] cpuidle: using governor menu
[    0.084931] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.085512] ASID allocator initialised with 65536 entries
[    0.085808] ACPI: bus type PCI registered
[    0.085815] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.086052] Serial: AMBA PL011 UART driver
[    0.096807] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.096815] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.096821] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.096826] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.101895] ACPI: Added _OSI(Module Device)
[    0.101903] ACPI: Added _OSI(Processor Device)
[    0.101907] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.101912] ACPI: Added _OSI(Processor Aggregator Device)
[    0.101917] ACPI: Added _OSI(Linux-Dell-Video)
[    0.101923] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.101928] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.105849] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.107742] ACPI: Interpreter enabled
[    0.107749] ACPI: Using GIC for interrupt routing
[    0.107778] ACPI: MCFG table detected, 4 entries
[    0.131437] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1f])
[    0.131454] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.131603] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    0.131732] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability]
[    0.132192] acpi PNP0A08:00: ECAM area [mem 0x848000000000-0x848001ffffff] reserved by CAVA02C:00
[    0.132210] acpi PNP0A08:00: ECAM at [mem 0x848000000000-0x848001ffffff] for [bus 00-1f]
[    0.132366] PCI host bridge to bus 0000:00
[    0.132374] pci_bus 0000:00: root bus resource [mem 0x801000000000-0x807fffffffff window]
[    0.132381] pci_bus 0000:00: root bus resource [mem 0x838000000000-0x841fffffffff window]
[    0.132388] pci_bus 0000:00: root bus resource [mem 0x846000000000-0x847fffffffff window]
[    0.132394] pci_bus 0000:00: root bus resource [mem 0x868000000000-0x87e023ffffff window]
[    0.132400] pci_bus 0000:00: root bus resource [mem 0x87e026000000-0x87e0bfffffff window]
[    0.132407] pci_bus 0000:00: root bus resource [mem 0x87e0c6000000-0x87ffffffffff window]
[    0.132414] pci_bus 0000:00: root bus resource [bus 00-1f]
[    0.132444] pci 0000:00:01.0: [177d:a002] type 01 class 0x060400
[    0.132615] pci 0000:00:09.0: [177d:a018] type 00 class 0x120000
[    0.132651] pci 0000:00:09.0: BAR 0: [mem 0x87e040000000-0x87e0400fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    0.132660] pci 0000:00:09.0: VF BAR 0: [mem 0x840000800000-0x8400008fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.153436] pci 0000:00:09.0: VF(n) BAR0 space: [mem 0x840000800000-0x8400008fffff 64bit] (contains BAR0 for 1 VFs)
[    1.153589] pci 0000:00:10.0: [177d:a01b] type 00 class 0x0c0330
[    1.153625] pci 0000:00:10.0: BAR 0: [mem 0x868000000000-0x8680001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153634] pci 0000:00:10.0: BAR 4: [mem 0x868000200000-0x8680002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153761] pci 0000:00:11.0: [177d:a01b] type 00 class 0x0c0330
[    1.153797] pci 0000:00:11.0: BAR 0: [mem 0x869000000000-0x8690001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153806] pci 0000:00:11.0: BAR 4: [mem 0x869000200000-0x8690002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153931] pci 0000:00:14.0: [177d:a002] type 01 class 0x060400
[    1.154132] pci 0000:01:00.0: [177d:a001] type 00 class 0x088000
[    1.154167] pci 0000:01:00.0: BAR 0: [mem 0x87e0fc000000-0x87e0fc0fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154177] pci 0000:01:00.0: BAR 4: [mem 0x87e0fcf00000-0x87e0fcffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154302] pci 0000:01:00.1: [177d:a00e] type 00 class 0x088000
[    1.154338] pci 0000:01:00.1: BAR 0: [mem 0x87e006000000-0x87e0067fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154347] pci 0000:01:00.1: BAR 4: [mem 0x87e006f00000-0x87e006ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154467] pci 0000:01:01.3: [177d:a02b] type 00 class 0x0c8000
[    1.154501] pci 0000:01:01.3: BAR 0: [mem 0x87e005000000-0x87e0057fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154619] pci 0000:01:01.4: [177d:a010] type 00 class 0x058000
[    1.154659] pci 0000:01:01.4: BAR 0: [mem 0x87e009000000-0x87e0097fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154668] pci 0000:01:01.4: BAR 4: [mem 0x87e009f00000-0x87e009ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154790] pci 0000:01:06.0: [177d:a02e] type 00 class 0x058000
[    1.154826] pci 0000:01:06.0: BAR 0: [mem 0x87e050000000-0x87e0507fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154836] pci 0000:01:06.0: BAR 4: [mem 0x87e050f00000-0x87e050ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154961] pci 0000:01:06.1: [177d:a02e] type 00 class 0x058000
[    1.154997] pci 0000:01:06.1: BAR 0: [mem 0x87e051000000-0x87e0517fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155006] pci 0000:01:06.1: BAR 4: [mem 0x87e051f00000-0x87e051ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155127] pci 0000:01:06.2: [177d:a02e] type 00 class 0x058000
[    1.155163] pci 0000:01:06.2: BAR 0: [mem 0x87e052000000-0x87e0527fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155172] pci 0000:01:06.2: BAR 4: [mem 0x87e052f00000-0x87e052ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155291] pci 0000:01:06.3: [177d:a02e] type 00 class 0x058000
[    1.155326] pci 0000:01:06.3: BAR 0: [mem 0x87e053000000-0x87e0537fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155335] pci 0000:01:06.3: BAR 4: [mem 0x87e053f00000-0x87e053ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155455] pci 0000:01:06.4: [177d:a02e] type 00 class 0x058000
[    1.155491] pci 0000:01:06.4: BAR 0: [mem 0x87e054000000-0x87e0547fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155500] pci 0000:01:06.4: BAR 4: [mem 0x87e054f00000-0x87e054ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155621] pci 0000:01:06.5: [177d:a02e] type 00 class 0x058000
[    1.155657] pci 0000:01:06.5: BAR 0: [mem 0x87e055000000-0x87e0557fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155666] pci 0000:01:06.5: BAR 4: [mem 0x87e055f00000-0x87e055ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155785] pci 0000:01:06.6: [177d:a02e] type 00 class 0x058000
[    1.155821] pci 0000:01:06.6: BAR 0: [mem 0x87e056000000-0x87e0567fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155830] pci 0000:01:06.6: BAR 4: [mem 0x87e056f00000-0x87e056ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155950] pci 0000:01:06.7: [177d:a02e] type 00 class 0x058000
[    1.155989] pci 0000:01:06.7: BAR 0: [mem 0x87e057000000-0x87e0577fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155998] pci 0000:01:06.7: BAR 4: [mem 0x87e057f00000-0x87e057ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156120] pci 0000:01:07.0: [177d:a02f] type 00 class 0x058000
[    1.156156] pci 0000:01:07.0: BAR 0: [mem 0x87e058000000-0x87e0587fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156165] pci 0000:01:07.0: BAR 4: [mem 0x87e058f00000-0x87e058ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156291] pci 0000:01:07.1: [177d:a02f] type 00 class 0x058000
[    1.156327] pci 0000:01:07.1: BAR 0: [mem 0x87e059000000-0x87e0597fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156336] pci 0000:01:07.1: BAR 4: [mem 0x87e059f00000-0x87e059ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156456] pci 0000:01:07.2: [177d:a02f] type 00 class 0x058000
[    1.156492] pci 0000:01:07.2: BAR 0: [mem 0x87e05a000000-0x87e05a7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156501] pci 0000:01:07.2: BAR 4: [mem 0x87e05af00000-0x87e05affffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156623] pci 0000:01:07.3: [177d:a02f] type 00 class 0x058000
[    1.156659] pci 0000:01:07.3: BAR 0: [mem 0x87e05b000000-0x87e05b7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156668] pci 0000:01:07.3: BAR 4: [mem 0x87e05bf00000-0x87e05bffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156787] pci 0000:01:07.4: [177d:a030] type 00 class 0x058000
[    1.156822] pci 0000:01:07.4: BAR 0: [mem 0x87e05c000000-0x87e05c7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156831] pci 0000:01:07.4: BAR 4: [mem 0x87e05cf00000-0x87e05cffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156951] pci 0000:01:07.5: [177d:a030] type 00 class 0x058000
[    1.156986] pci 0000:01:07.5: BAR 0: [mem 0x87e05d000000-0x87e05d7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156995] pci 0000:01:07.5: BAR 4: [mem 0x87e05df00000-0x87e05dffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157118] pci 0000:01:07.6: [177d:a030] type 00 class 0x058000
[    1.157153] pci 0000:01:07.6: BAR 0: [mem 0x87e05e000000-0x87e05e7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157162] pci 0000:01:07.6: BAR 4: [mem 0x87e05ef00000-0x87e05effffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157286] pci 0000:01:07.7: [177d:a030] type 00 class 0x058000
[    1.157334] pci 0000:01:07.7: BAR 0: [mem 0x87e05f000000-0x87e05f7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157343] pci 0000:01:07.7: BAR 4: [mem 0x87e05ff00000-0x87e05fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157476] pci 0000:01:09.2: [177d:a012] type 00 class 0x0c8000
[    1.157513] pci 0000:01:09.2: BAR 0: [mem 0x87e0d2000000-0x87e0d27fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157522] pci 0000:01:09.2: BAR 4: [mem 0x87e0d2f00000-0x87e0d2ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157649] pci 0000:01:09.4: [177d:a012] type 00 class 0x0c8000
[    1.157685] pci 0000:01:09.4: BAR 0: [mem 0x87e0d4000000-0x87e0d47fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157694] pci 0000:01:09.4: BAR 4: [mem 0x87e0d4f00000-0x87e0d4ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157815] pci 0000:01:0a.0: [177d:a022] type 00 class 0x058000
[    1.157851] pci 0000:01:0a.0: BAR 0: [mem 0x87e088000000-0x87e0887fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157860] pci 0000:01:0a.0: BAR 4: [mem 0x87e088f00000-0x87e088ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157983] pci 0000:01:0a.1: [177d:a022] type 00 class 0x058000
[    1.158018] pci 0000:01:0a.1: BAR 0: [mem 0x87e089000000-0x87e0897fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158027] pci 0000:01:0a.1: BAR 4: [mem 0x87e089f00000-0x87e089ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158150] pci 0000:01:10.0: [177d:a026] type 00 class 0x028000
[    1.158185] pci 0000:01:10.0: BAR 0: [mem 0x87e0e0000000-0x87e0e03fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158194] pci 0000:01:10.0: BAR 4: [mem 0x87e0e0400000-0x87e0e07fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158316] pci 0000:01:10.1: [177d:a026] type 00 class 0x028000
[    1.158351] pci 0000:01:10.1: BAR 0: [mem 0x87e0e1000000-0x87e0e13fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158360] pci 0000:01:10.1: BAR 4: [mem 0x87e0e1400000-0x87e0e17fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158471] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.158637] pci 0000:02:00.0: [177d:a01d] type 00 class 0x010400
[    1.158676] pci 0000:02:00.0: BAR 0: [mem 0x870000000000-0x8700007fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158685] pci 0000:02:00.0: BAR 4: [mem 0x870000f00000-0x870000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158800] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.158875] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.158889] pci 0000:00:14.0: PCI bridge to [bus 02]
[    1.158897] pci_bus 0000:00: resource 4 [mem 0x801000000000-0x807fffffffff window]
[    1.158903] pci_bus 0000:00: resource 5 [mem 0x838000000000-0x841fffffffff window]
[    1.158910] pci_bus 0000:00: resource 6 [mem 0x846000000000-0x847fffffffff window]
[    1.158916] pci_bus 0000:00: resource 7 [mem 0x868000000000-0x87e023ffffff window]
[    1.158922] pci_bus 0000:00: resource 8 [mem 0x87e026000000-0x87e0bfffffff window]
[    1.158929] pci_bus 0000:00: resource 9 [mem 0x87e0c6000000-0x87ffffffffff window]
[    1.159085] ACPI: PCI Root Bridge [PCI1] (domain 0001 [bus 00-1f])
[    1.159098] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.159240] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.159369] acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
[    1.159842] acpi PNP0A08:01: ECAM area [mem 0x849000000000-0x849001ffffff] reserved by CAVA02C:01
[    1.159858] acpi PNP0A08:01: ECAM at [mem 0x849000000000-0x849001ffffff] for [bus 00-1f]
[    1.159988] PCI host bridge to bus 0001:00
[    1.159995] pci_bus 0001:00: root bus resource [mem 0x810000000000-0x817fffffffff window]
[    1.160003] pci_bus 0001:00: root bus resource [bus 00-1f]
[    1.160031] pci 0001:00:08.0: [177d:a01c] type 00 class 0x010601
[    1.160063] pci 0001:00:08.0: BAR 0: [mem 0x814000000000-0x8140001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160072] pci 0001:00:08.0: BAR 4: [mem 0x814000200000-0x8140002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160200] pci 0001:00:09.0: [177d:a01c] type 00 class 0x010601
[    1.160239] pci 0001:00:09.0: BAR 0: [mem 0x815000000000-0x8150001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160248] pci 0001:00:09.0: BAR 4: [mem 0x815000200000-0x8150002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160378] pci 0001:00:0a.0: [177d:a01c] type 00 class 0x010601
[    1.160413] pci 0001:00:0a.0: BAR 0: [mem 0x816000000000-0x8160001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160422] pci 0001:00:0a.0: BAR 4: [mem 0x816000200000-0x8160002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160548] pci 0001:00:0b.0: [177d:a01c] type 00 class 0x010601
[    1.160583] pci 0001:00:0b.0: BAR 0: [mem 0x817000000000-0x8170001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160592] pci 0001:00:0b.0: BAR 4: [mem 0x817000200000-0x8170002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160724] pci_bus 0001:00: resource 4 [mem 0x810000000000-0x817fffffffff window]
[    1.160832] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-1f])
[    1.160845] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.160985] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.161113] acpi PNP0A08:02: _OSC: OS now controls [PCIeCapability]
[    1.161617] acpi PNP0A08:02: ECAM area [mem 0x84a000000000-0x84a001ffffff] reserved by CAVA02C:02
[    1.161632] acpi PNP0A08:02: ECAM at [mem 0x84a000000000-0x84a001ffffff] for [bus 00-1f]
[    1.161765] PCI host bridge to bus 0002:00
[    1.161773] pci_bus 0002:00: root bus resource [mem 0x842000000000-0x843fffffffff window]
[    1.161780] pci_bus 0002:00: root bus resource [bus 00-1f]
[    1.161804] pci 0002:00:02.0: [177d:a002] type 01 class 0x060400
[    1.161952] pci 0002:00:03.0: [177d:a01f] type 00 class 0x028000
[    1.161986] pci 0002:00:03.0: BAR 0: [mem 0x842000000000-0x84200000ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.161995] pci 0002:00:03.0: BAR 2: [mem 0x842040000000-0x84207fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162004] pci 0002:00:03.0: BAR 4: [mem 0x842000f00000-0x842000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162195] pci 0002:01:00.0: [177d:a01e] type 00 class 0x020000
[    1.162231] pci 0002:01:00.0: BAR 0: [mem 0x843000000000-0x84303fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162240] pci 0002:01:00.0: BAR 4: [mem 0x843060000000-0x8430600fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162249] pci 0002:01:00.0: VF BAR 0: [mem 0x8430a0000000-0x8430a01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.162257] pci 0002:01:00.0: VF BAR 4: [mem 0x8430e0000000-0x8430e01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    2.178767] pci 0002:01:00.0: VF(n) BAR0 space: [mem 0x8430a0000000-0x8430afffffff 64bit] (contains BAR0 for 128 VFs)
[    2.178782] pci 0002:01:00.0: VF(n) BAR4 space: [mem 0x8430e0000000-0x8430efffffff 64bit] (contains BAR4 for 128 VFs)
[    2.178990] pci 0002:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.179016] pci 0002:00:02.0: PCI bridge to [bus 01]
[    2.179025] pci_bus 0002:00: resource 4 [mem 0x842000000000-0x843fffffffff window]
[    2.179134] ACPI: PCI Root Bridge [PCI3] (domain 0003 [bus 00-1f])
[    2.179147] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.179291] acpi PNP0A08:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.179421] acpi PNP0A08:03: _OSC: OS now controls [PCIeCapability]
[    2.179941] acpi PNP0A08:03: ECAM area [mem 0x84b000000000-0x84b001ffffff] reserved by CAVA02C:03
[    2.179957] acpi PNP0A08:03: ECAM at [mem 0x84b000000000-0x84b001ffffff] for [bus 00-1f]
[    2.180087] PCI host bridge to bus 0003:00
[    2.180094] pci_bus 0003:00: root bus resource [mem 0x818000000000-0x81ffffffffff window]
[    2.180102] pci_bus 0003:00: root bus resource [bus 00-1f]
[    2.180136] pci_bus 0003:00: resource 4 [mem 0x818000000000-0x81ffffffffff window]
[    2.180243] ACPI: PCI Root Bridge [PEM0] (domain 0004 [bus 1f-57])
[    2.180255] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.180395] acpi PNP0A08:04: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.180525] acpi PNP0A08:04: _OSC: OS now controls [PCIeCapability]
[    2.180534] acpi PNP0A08:04: MCFG quirk: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57] with thunder_pem_ecam_ops
[    2.181084] acpi PNP0A08:04: ECAM area [mem 0x88001f000000-0x880057ffffff] reserved by CAVA02B:00
[    2.181680] acpi PNP0A08:04: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57]
[    2.181715] Remapped I/O 0x0000883000000000 to [io  0x0000-0xffff window]
[    2.181851] PCI host bridge to bus 0004:1f
[    2.181858] pci_bus 0004:1f: root bus resource [io  0x0000-0xffff window]
[    2.181866] pci_bus 0004:1f: root bus resource [mem 0x881010000000-0x881fffffffff window] (bus address [0x10000000-0xfffffffff])
[    2.181874] pci_bus 0004:1f: root bus resource [mem 0x882000000000-0x882fffffffff pref window] (bus address [0x1000000000-0x1fffffffff])
[    2.181881] pci_bus 0004:1f: root bus resource [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.181888] pci_bus 0004:1f: root bus resource [bus 1f-57]
[    2.181914] pci 0004:1f:00.0: [177d:a100] type 01 class 0x060400
[    2.181928] pci 0004:1f:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
[    2.181938] pci 0004:1f:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.181953] pci 0004:1f:00.0: enabling Extended Tags
[    2.181977] pci 0004:1f:00.0: BAR 0: [mem 0x87e0c0f00000-0x87e0c0ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.182150] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.182244] pci 0004:20:00.0: [1a03:1150] type 01 class 0x060400
[    2.182294] pci 0004:20:00.0: Upstream bridge's Max Payload Size set to 128 (was 256, max 256)
[    2.182303] pci 0004:20:00.0: Max Payload Size set to 128 (was 128, max 128)
[    2.182312] pci 0004:20:00.0: enabling Extended Tags
[    2.182398] pci 0004:20:00.0: supports D1 D2
[    2.182404] pci 0004:20:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.182575] pci 0004:20:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.182640] pci_bus 0004:21: extended config space not accessible
[    2.182701] pci 0004:21:00.0: [1a03:2000] type 00 class 0x030000
[    2.182721] pci 0004:21:00.0: reg 0x10: [mem 0x881010000000-0x881010ffffff]
[    2.182734] pci 0004:21:00.0: reg 0x14: [mem 0x881011000000-0x88101101ffff]
[    2.182746] pci 0004:21:00.0: reg 0x18: [io  0x0000-0x007f]
[    2.182806] pci 0004:21:00.0: BAR 0: assigned to efifb
[    2.182855] pci 0004:21:00.0: supports D1 D2
[    2.182861] pci 0004:21:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.183056] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.183074] pci 0004:1f:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.183082] pci 0004:1f:00.0: BAR 6: assigned [mem 0x881011800000-0x88101180ffff pref]
[    2.183090] pci 0004:1f:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.183098] pci 0004:20:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.183105] pci 0004:20:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.183113] pci 0004:21:00.0: BAR 0: assigned [mem 0x881010000000-0x881010ffffff]
[    2.183122] pci 0004:21:00.0: BAR 1: assigned [mem 0x881011000000-0x88101101ffff]
[    2.183131] pci 0004:21:00.0: BAR 2: assigned [io  0x1000-0x107f]
[    2.183140] pci 0004:20:00.0: PCI bridge to [bus 21]
[    2.183147] pci 0004:20:00.0:   bridge window [io  0x1000-0x1fff]
[    2.183153] pci 0004:20:00.0:   bridge window [mem 0x881010000000-0x8810117fffff]
[    2.183160] pci 0004:1f:00.0: PCI bridge to [bus 20-21]
[    2.183166] pci 0004:1f:00.0:   bridge window [io  0x1000-0x1fff]
[    2.183173] pci 0004:1f:00.0:   bridge window [mem 0x881010000000-0x8810117fffff]
[    2.183180] pci_bus 0004:1f: resource 4 [io  0x0000-0xffff window]
[    2.183186] pci_bus 0004:1f: resource 5 [mem 0x881010000000-0x881fffffffff window]
[    2.183193] pci_bus 0004:1f: resource 6 [mem 0x882000000000-0x882fffffffff pref window]
[    2.183199] pci_bus 0004:1f: resource 7 [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.183206] pci_bus 0004:20: resource 0 [io  0x1000-0x1fff]
[    2.183212] pci_bus 0004:20: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.183218] pci_bus 0004:21: resource 0 [io  0x1000-0x1fff]
[    2.183224] pci_bus 0004:21: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.183336] ACPI: PCI Interrupt Link [LN0A] (IRQs *48)
[    2.183387] ACPI: PCI Interrupt Link [LN0B] (IRQs *49)
[    2.183434] ACPI: PCI Interrupt Link [LN0C] (IRQs *50)
[    2.183480] ACPI: PCI Interrupt Link [LN0D] (IRQs *51)
[    2.183772] ARMH0011:00: ttyAMA0 at MMIO 0x87e024000000 (irq = 21, base_baud = 0) is a SBSA
[    6.382848] printk: console [ttyAMA0] enabled
[    6.390414] ARMH0011:01: ttyAMA1 at MMIO 0x87e025000000 (irq = 22, base_baud = 0) is a SBSA
[    6.404767] iommu: Default domain type: Translated 
[    6.409783] pci 0004:21:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    6.418137] pci 0004:21:00.0: vgaarb: bridge control possible
[    6.423876] pci 0004:21:00.0: vgaarb: setting as boot device (VGA legacy resources not available)
[    6.432738] vgaarb: loaded
[    6.436158] SCSI subsystem initialized
[    6.440133] ACPI: bus type USB registered
[    6.444193] usbcore: registered new interface driver usbfs
[    6.449703] usbcore: registered new interface driver hub
[    6.455057] usbcore: registered new device driver usb
[    6.460202] pps_core: LinuxPPS API ver. 1 registered
[    6.465165] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    6.474295] PTP clock support registered
[    6.478411] EDAC MC: Ver: 3.0.0
[    6.481993] Registered efivars operations
[    6.487793] NetLabel: Initializing
[    6.491196] NetLabel:  domain hash size = 128
[    6.495544] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    6.501233] NetLabel:  unlabeled traffic allowed by default
[    6.507333] clocksource: Switched to clocksource arch_sys_counter
[    6.553602] VFS: Disk quotas dquot_6.6.0
[    6.557582] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    6.564907] AppArmor: AppArmor Filesystem Enabled
[    6.569703] pnp: PnP ACPI init
[    6.573376] system 00:00: [mem 0x848000000000-0x848001ffffff] could not be reserved
[    6.581230] system 00:01: [mem 0x849000000000-0x849001ffffff] could not be reserved
[    6.589075] system 00:02: [mem 0x84a000000000-0x84a001ffffff] could not be reserved
[    6.596921] system 00:03: [mem 0x84b000000000-0x84b001ffffff] could not be reserved
[    6.604770] system 00:04: [mem 0x87e0c0000000-0x87e0c0ffffff] could not be reserved
[    6.612431] system 00:04: [mem 0x88001f000000-0x880057ffffff] could not be reserved
[    6.622306] pnp: PnP ACPI: found 5 devices
[    6.632232] NET: Registered protocol family 2
[    6.637376] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    6.646603] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    6.656465] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    6.664233] TCP: Hash tables configured (established 524288 bind 65536)
[    6.671254] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, linear)
[    6.679469] UDP hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.687022] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.695198] NET: Registered protocol family 1
[    6.699568] NET: Registered protocol family 44
[    6.704208] PCI: CLS 0 bytes, default 64
[    6.708237] Trying to unpack rootfs image as initramfs...
[    7.713965] Freeing initrd memory: 68180K
[    7.721067] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available
[    7.729104] kvm [1]: IPA Size Limit: 48 bits
[    7.734986] kvm [1]: GICv3: no GICV resource entry
[    7.739781] kvm [1]: disabling GICv2 emulation
[    7.744217] kvm [1]: GICv3 sysreg trapping enabled ([G0G1], reduced performance)
[    7.751617] kvm [1]: GIC system register CPU interface enabled
[    7.758605] kvm [1]: vgic interrupt IRQ9
[    7.763776] kvm [1]: Hyp mode initialized successfully
[    7.771680] Initialise system trusted keyrings
[    7.776151] Key type blacklist registered
[    7.780323] workingset: timestamp_bits=40 max_order=24 bucket_order=0
[    7.791059] zbud: loaded
[    7.794771] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    7.801149] fuse: init (API version 7.33)
[    7.805555] integrity: Platform Keyring initialized
[    7.833189] Key type asymmetric registered
[    7.837286] Asymmetric key parser 'x509' registered
[    7.842174] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    7.849699] io scheduler mq-deadline registered
[    7.857591] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    7.865127] efifb: probing for efifb
[    7.868738] efifb: No BGRT, not showing boot graphics
[    7.873783] efifb: framebuffer at 0x881010000000, using 3072k, total 3072k
[    7.880649] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    7.886647] efifb: scrolling: redraw
[    7.890212] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    7.895905] fbcon: Deferring console take-over
[    7.900350] fb0: EFI VGA frame buffer device
[    7.905289] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[    7.913714] ACPI: button: Power Button [PWRB]
[    7.919549] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address.
[    7.927289] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
[    7.936326] Mem abort info:
[    7.939108]   ESR = 0x96000004
[    7.942151]   EC = 0x25: DABT (current EL), IL = 32 bits
[    7.947451]   SET = 0, FnV = 0
[    7.950494]   EA = 0, S1PTW = 0
[    7.953624] Data abort info:
[    7.956492]   ISV = 0, ISS = 0x00000004
[    7.960316]   CM = 0, WnR = 0
[    7.963273] [0000000000000028] user address but active_mm is swapper
[    7.969616] Internal error: Oops: 96000004 [#1] SMP
[    7.974483] Modules linked in:
[    7.977531] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 5.12.0-rc8 #19
[    7.983874] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    7.990737] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
[    7.996732] pc : __ipi_send_mask+0x60/0x114
[    8.000910] lr : smp_cross_call+0x40/0xcc
[    8.004913] sp : ffff800012753c10
[    8.008216] x29: ffff800012753c10 x28: ffff000100de5d00 
[    8.013521] x27: 000000000000000a x26: ffff80001225da20 
[    8.018825] x25: 0000000000000000 x24: ffff000ff62719b0 
[    8.024129] x23: ffff80001225d000 x22: ffff800012368108 
[    8.029433] x21: ffff800010f69a20 x20: 0000000000000000 
[    8.034737] x19: ffff000100143c60 x18: 0000000000000020 
[    8.040041] x17: 000000008e74252f x16: 00000000bf0ab2ad 
[    8.045345] x15: ffffffffffffffff x14: 0000000000000000 
[    8.050649] x13: 003d090000000000 x12: 00003d0900000000 
[    8.055953] x11: 0000000000000000 x10: 00003d0900000000 
[    8.061257] x9 : ffff800010027f14 x8 : 0000000000000000 
[    8.066561] x7 : 00000000ffffffff x6 : ffff000ff6148698 
[    8.071865] x5 : ffff80001159d040 x4 : ffff80001159d110 
[    8.077169] x3 : ffff800010f69a00 x2 : 0000000000000000 
[    8.082473] x1 : ffff800010f69a20 x0 : 0000000000000000 
[    8.087777] Call trace:
[    8.090213]  __ipi_send_mask+0x60/0x114
[    8.094038]  smp_cross_call+0x40/0xcc
[    8.097691]  smp_send_reschedule+0x3c/0x50
[    8.101778]  resched_curr+0x5c/0xb0
[    8.105258]  check_preempt_curr+0x58/0x90
[    8.109258]  ttwu_do_wakeup+0x2c/0x190
[    8.112996]  ttwu_do_activate+0x7c/0x114
[    8.116909]  try_to_wake_up+0x388/0x670
[    8.120735]  wake_up_process+0x24/0x30
[    8.124474]  swake_up_one+0x48/0x9c
[    8.127953]  rcu_gp_kthread_wake+0x68/0x8c
[    8.132041]  rcu_accelerate_cbs_unlocked+0xb4/0xf0
[    8.136822]  rcu_core+0x520/0x694
[    8.140128]  rcu_core_si+0x1c/0x2c
[    8.143520]  __do_softirq+0x128/0x388
[    8.147172]  irq_exit+0xc4/0xec
[    8.150304]  __handle_domain_irq+0x8c/0xec
[    8.154394]  gic_handle_irq+0xd8/0x2f0
[    8.158132]  el1_irq+0xc0/0x180
[    8.161262]  __pi_strcmp+0x20/0x158
[    8.164742]  driver_register+0x68/0x140
[    8.168571]  __platform_driver_register+0x34/0x40
[    8.173265]  imx8mp_clk_driver_init+0x28/0x34
[    8.177614]  do_one_initcall+0x50/0x260
[    8.181440]  kernel_init_freeable+0x24c/0x2d4
[    8.185790]  kernel_init+0x20/0x134
[    8.189271]  ret_from_fork+0x10/0x18
[    8.192840] Code: a90363f7 aa0103f5 d0010957 f9401260 (b9402800) 
[    8.198955] ---[ end trace c24172add816c1f0 ]---
[    8.203562] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[    8.210442] SMP: stopping secondary CPUs
[    9.258360] SMP: failed to stop secondary CPUs 0,9
[    9.263141] Kernel Offset: disabled
[    9.266617] CPU features: 0x00040002,69101108
[    9.270963] Memory Limit: none
[    9.274024] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[Marc Zyngier]

Hi Dan,n

On Tue, 20 Apr 2021 22:25:51 +0100,
dann frazier <dann.frazier@canonical.com> wrote:
> 
> On Tue, Apr 20, 2021 at 02:37:10PM -0600, dann frazier wrote:
> > On Tue, May 19, 2020 at 05:17:52PM +0100, Marc Zyngier wrote:
> > > Change the way we deal with GIC SGIs by turning them into proper
> > > IRQs, and calling into the arch code to register the interrupt range
> > > instead of a callback.
> > > 
> > > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > 
> > hey Marc,
> > 
> >   I bisected a boot failure on our Gigabyte R120-T33 systems (ThunderX
> > CN88XX) down to this commit, but only when running in ACPI mode. See below:
> > 
> > 
> > EFI stub: Booting Linux Kernel...
> > EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> > EFI stub: Using DTB from configuration table
> > EFI stub: Exiting boot services and installing virtual address map...
> > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
> > [    0.000000] Linux version 5.11.0-13-generic (buildd@bos02-arm64-067) (gcc (Ubuntu 10.2.1-23ubuntu1) 10.2.1 20210312, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #14-Ubuntu SMP Fri Mar 19 16:57:35 UTC 2021 (Ubuntu 5.11.0-13.14-generic 5.11.7)
> 
> Sorry, realized I posted a log from an Ubuntu kernel. Here's an
> upstream one:

[...]

> 
> [    7.842174] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
> [    7.849699] io scheduler mq-deadline registered
> [    7.857591] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> [    7.865127] efifb: probing for efifb
> [    7.868738] efifb: No BGRT, not showing boot graphics
> [    7.873783] efifb: framebuffer at 0x881010000000, using 3072k, total 3072k
> [    7.880649] efifb: mode is 1024x768x32, linelength=4096, pages=1
> [    7.886647] efifb: scrolling: redraw
> [    7.890212] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> [    7.895905] fbcon: Deferring console take-over
> [    7.900350] fb0: EFI VGA frame buffer device
> [    7.905289] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
> [    7.913714] ACPI: button: Power Button [PWRB]
> [    7.919549] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address.
> [    7.927289] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
> [    7.936326] Mem abort info:
> [    7.939108]   ESR = 0x96000004
> [    7.942151]   EC = 0x25: DABT (current EL), IL = 32 bits
> [    7.947451]   SET = 0, FnV = 0
> [    7.950494]   EA = 0, S1PTW = 0
> [    7.953624] Data abort info:
> [    7.956492]   ISV = 0, ISS = 0x00000004
> [    7.960316]   CM = 0, WnR = 0
> [    7.963273] [0000000000000028] user address but active_mm is swapper
> [    7.969616] Internal error: Oops: 96000004 [#1] SMP
> [    7.974483] Modules linked in:
> [    7.977531] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 5.12.0-rc8 #19
> [    7.983874] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
> [    7.990737] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
> [    7.996732] pc : __ipi_send_mask+0x60/0x114
> [    8.000910] lr : smp_cross_call+0x40/0xcc
> [    8.004913] sp : ffff800012753c10
> [    8.008216] x29: ffff800012753c10 x28: ffff000100de5d00 
> [    8.013521] x27: 000000000000000a x26: ffff80001225da20 
> [    8.018825] x25: 0000000000000000 x24: ffff000ff62719b0 
> [    8.024129] x23: ffff80001225d000 x22: ffff800012368108 
> [    8.029433] x21: ffff800010f69a20 x20: 0000000000000000 
> [    8.034737] x19: ffff000100143c60 x18: 0000000000000020 
> [    8.040041] x17: 000000008e74252f x16: 00000000bf0ab2ad 
> [    8.045345] x15: ffffffffffffffff x14: 0000000000000000 
> [    8.050649] x13: 003d090000000000 x12: 00003d0900000000 
> [    8.055953] x11: 0000000000000000 x10: 00003d0900000000 
> [    8.061257] x9 : ffff800010027f14 x8 : 0000000000000000 
> [    8.066561] x7 : 00000000ffffffff x6 : ffff000ff6148698 
> [    8.071865] x5 : ffff80001159d040 x4 : ffff80001159d110 
> [    8.077169] x3 : ffff800010f69a00 x2 : 0000000000000000 
> [    8.082473] x1 : ffff800010f69a20 x0 : 0000000000000000 
> [    8.087777] Call trace:
> [    8.090213]  __ipi_send_mask+0x60/0x114
> [    8.094038]  smp_cross_call+0x40/0xcc
> [    8.097691]  smp_send_reschedule+0x3c/0x50
> [    8.101778]  resched_curr+0x5c/0xb0
> [    8.105258]  check_preempt_curr+0x58/0x90
> [    8.109258]  ttwu_do_wakeup+0x2c/0x190
> [    8.112996]  ttwu_do_activate+0x7c/0x114
> [    8.116909]  try_to_wake_up+0x388/0x670
> [    8.120735]  wake_up_process+0x24/0x30
> [    8.124474]  swake_up_one+0x48/0x9c
> [    8.127953]  rcu_gp_kthread_wake+0x68/0x8c
> [    8.132041]  rcu_accelerate_cbs_unlocked+0xb4/0xf0
> [    8.136822]  rcu_core+0x520/0x694
> [    8.140128]  rcu_core_si+0x1c/0x2c
> [    8.143520]  __do_softirq+0x128/0x388
> [    8.147172]  irq_exit+0xc4/0xec
> [    8.150304]  __handle_domain_irq+0x8c/0xec
> [    8.154394]  gic_handle_irq+0xd8/0x2f0
> [    8.158132]  el1_irq+0xc0/0x180
> [    8.161262]  __pi_strcmp+0x20/0x158
> [    8.164742]  driver_register+0x68/0x140
> [    8.168571]  __platform_driver_register+0x34/0x40
> [    8.173265]  imx8mp_clk_driver_init+0x28/0x34
> [    8.177614]  do_one_initcall+0x50/0x260
> [    8.181440]  kernel_init_freeable+0x24c/0x2d4
> [    8.185790]  kernel_init+0x20/0x134
> [    8.189271]  ret_from_fork+0x10/0x18
> [    8.192840] Code: a90363f7 aa0103f5 d0010957 f9401260 (b9402800) 
> [    8.198955] ---[ end trace c24172add816c1f0 ]---
> [    8.203562] Kernel panic - not syncing: Oops: Fatal exception in interrupt
> [    8.210442] SMP: stopping secondary CPUs
> [    9.258360] SMP: failed to stop secondary CPUs 0,9
> [    9.263141] Kernel Offset: disabled
> [    9.266617] CPU features: 0x00040002,69101108
> [    9.270963] Memory Limit: none
> [    9.274024] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Please feed this stacktrace to scripts/decode_stacktrace.sh so that I
can get an idea about what is going wrong. I bet something is playing
ungodly games with the one of the IPIs, and things go horribly wrong.

Now, here's a hunch: in the fine TX1 tradition, the firmware is broken
and the GTDT table looks unusable. Amusingly, the crash happens right
after the SBSA watchdog fails to probe.

And looking at the code that implements that driver, it looks dodgy as
hell, as it unmaps an interrupt it doesn't even know is valid. And it
does that right when the driver fails the way you experienced it. If,
by any chance, the interrupt field is 0 in the firmware table, this
results in SGI0 being unmapped. Given that this is the rescheduling
interrupt, fireworks happen.

Can you have a go with the patchlet below, and let me know if that
helps?

Thanks,

	M.

diff --git a/drivers/acpi/arm64/gtdt.c b/drivers/acpi/arm64/gtdt.c
index f2d0e5915dab..0a0a982f9c28 100644
--- a/drivers/acpi/arm64/gtdt.c
+++ b/drivers/acpi/arm64/gtdt.c
@@ -329,7 +329,7 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 					int index)
 {
 	struct platform_device *pdev;
-	int irq = map_gt_gsi(wd->timer_interrupt, wd->timer_flags);
+	int irq;
 
 	/*
 	 * According to SBSA specification the size of refresh and control
@@ -338,7 +338,7 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 	struct resource res[] = {
 		DEFINE_RES_MEM(wd->control_frame_address, SZ_4K),
 		DEFINE_RES_MEM(wd->refresh_frame_address, SZ_4K),
-		DEFINE_RES_IRQ(irq),
+		{},
 	};
 	int nr_res = ARRAY_SIZE(res);
 
@@ -348,10 +348,11 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 
 	if (!(wd->refresh_frame_address && wd->control_frame_address)) {
 		pr_err(FW_BUG "failed to get the Watchdog base address.\n");
-		acpi_unregister_gsi(wd->timer_interrupt);
 		return -EINVAL;
 	}
 
+	irq = map_gt_gsi(wd->timer_interrupt, wd->timer_flags);
+	res[2] = (struct resource)DEFINE_RES_IRQ(irq);
 	if (irq <= 0) {
 		pr_warn("failed to map the Watchdog interrupt.\n");
 		nr_res--;
@@ -364,7 +365,8 @@ static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
 	 */
 	pdev = platform_device_register_simple("sbsa-gwdt", index, res, nr_res);
 	if (IS_ERR(pdev)) {
-		acpi_unregister_gsi(wd->timer_interrupt);
+		if (irq > 0)
+			acpi_unregister_gsi(wd->timer_interrupt);
 		return PTR_ERR(pdev);
 	}
 

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[dann frazier]

[ + Fu Wei ]
On Wed, Apr 21, 2021 at 11:58:40AM +0100, Marc Zyngier wrote:
> Hi Dan,n
> 
> On Tue, 20 Apr 2021 22:25:51 +0100,
> dann frazier <dann.frazier@canonical.com> wrote:
> > 
> > On Tue, Apr 20, 2021 at 02:37:10PM -0600, dann frazier wrote:
> > > On Tue, May 19, 2020 at 05:17:52PM +0100, Marc Zyngier wrote:
> > > > Change the way we deal with GIC SGIs by turning them into proper
> > > > IRQs, and calling into the arch code to register the interrupt range
> > > > instead of a callback.
> > > > 
> > > > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > > 
> > > hey Marc,
> > > 
> > >   I bisected a boot failure on our Gigabyte R120-T33 systems (ThunderX
> > > CN88XX) down to this commit, but only when running in ACPI mode. See below:
> > > 
> > > 
> > > EFI stub: Booting Linux Kernel...
> > > EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> > > EFI stub: Using DTB from configuration table
> > > EFI stub: Exiting boot services and installing virtual address map...
> > > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
> > > [    0.000000] Linux version 5.11.0-13-generic (buildd@bos02-arm64-067) (gcc (Ubuntu 10.2.1-23ubuntu1) 10.2.1 20210312, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #14-Ubuntu SMP Fri Mar 19 16:57:35 UTC 2021 (Ubuntu 5.11.0-13.14-generic 5.11.7)
> > 
> > Sorry, realized I posted a log from an Ubuntu kernel. Here's an
> > upstream one:
> 
> [...]
> 
> > 
> > [    7.842174] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
> > [    7.849699] io scheduler mq-deadline registered
> > [    7.857591] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> > [    7.865127] efifb: probing for efifb
> > [    7.868738] efifb: No BGRT, not showing boot graphics
> > [    7.873783] efifb: framebuffer at 0x881010000000, using 3072k, total 3072k
> > [    7.880649] efifb: mode is 1024x768x32, linelength=4096, pages=1
> > [    7.886647] efifb: scrolling: redraw
> > [    7.890212] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> > [    7.895905] fbcon: Deferring console take-over
> > [    7.900350] fb0: EFI VGA frame buffer device
> > [    7.905289] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
> > [    7.913714] ACPI: button: Power Button [PWRB]
> > [    7.919549] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address.
> > [    7.927289] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
> > [    7.936326] Mem abort info:
> > [    7.939108]   ESR = 0x96000004
> > [    7.942151]   EC = 0x25: DABT (current EL), IL = 32 bits
> > [    7.947451]   SET = 0, FnV = 0
> > [    7.950494]   EA = 0, S1PTW = 0
> > [    7.953624] Data abort info:
> > [    7.956492]   ISV = 0, ISS = 0x00000004
> > [    7.960316]   CM = 0, WnR = 0
> > [    7.963273] [0000000000000028] user address but active_mm is swapper
> > [    7.969616] Internal error: Oops: 96000004 [#1] SMP
> > [    7.974483] Modules linked in:
> > [    7.977531] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 5.12.0-rc8 #19
> > [    7.983874] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
> > [    7.990737] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
> > [    7.996732] pc : __ipi_send_mask+0x60/0x114
> > [    8.000910] lr : smp_cross_call+0x40/0xcc
> > [    8.004913] sp : ffff800012753c10
> > [    8.008216] x29: ffff800012753c10 x28: ffff000100de5d00 
> > [    8.013521] x27: 000000000000000a x26: ffff80001225da20 
> > [    8.018825] x25: 0000000000000000 x24: ffff000ff62719b0 
> > [    8.024129] x23: ffff80001225d000 x22: ffff800012368108 
> > [    8.029433] x21: ffff800010f69a20 x20: 0000000000000000 
> > [    8.034737] x19: ffff000100143c60 x18: 0000000000000020 
> > [    8.040041] x17: 000000008e74252f x16: 00000000bf0ab2ad 
> > [    8.045345] x15: ffffffffffffffff x14: 0000000000000000 
> > [    8.050649] x13: 003d090000000000 x12: 00003d0900000000 
> > [    8.055953] x11: 0000000000000000 x10: 00003d0900000000 
> > [    8.061257] x9 : ffff800010027f14 x8 : 0000000000000000 
> > [    8.066561] x7 : 00000000ffffffff x6 : ffff000ff6148698 
> > [    8.071865] x5 : ffff80001159d040 x4 : ffff80001159d110 
> > [    8.077169] x3 : ffff800010f69a00 x2 : 0000000000000000 
> > [    8.082473] x1 : ffff800010f69a20 x0 : 0000000000000000 
> > [    8.087777] Call trace:
> > [    8.090213]  __ipi_send_mask+0x60/0x114
> > [    8.094038]  smp_cross_call+0x40/0xcc
> > [    8.097691]  smp_send_reschedule+0x3c/0x50
> > [    8.101778]  resched_curr+0x5c/0xb0
> > [    8.105258]  check_preempt_curr+0x58/0x90
> > [    8.109258]  ttwu_do_wakeup+0x2c/0x190
> > [    8.112996]  ttwu_do_activate+0x7c/0x114
> > [    8.116909]  try_to_wake_up+0x388/0x670
> > [    8.120735]  wake_up_process+0x24/0x30
> > [    8.124474]  swake_up_one+0x48/0x9c
> > [    8.127953]  rcu_gp_kthread_wake+0x68/0x8c
> > [    8.132041]  rcu_accelerate_cbs_unlocked+0xb4/0xf0
> > [    8.136822]  rcu_core+0x520/0x694
> > [    8.140128]  rcu_core_si+0x1c/0x2c
> > [    8.143520]  __do_softirq+0x128/0x388
> > [    8.147172]  irq_exit+0xc4/0xec
> > [    8.150304]  __handle_domain_irq+0x8c/0xec
> > [    8.154394]  gic_handle_irq+0xd8/0x2f0
> > [    8.158132]  el1_irq+0xc0/0x180
> > [    8.161262]  __pi_strcmp+0x20/0x158
> > [    8.164742]  driver_register+0x68/0x140
> > [    8.168571]  __platform_driver_register+0x34/0x40
> > [    8.173265]  imx8mp_clk_driver_init+0x28/0x34
> > [    8.177614]  do_one_initcall+0x50/0x260
> > [    8.181440]  kernel_init_freeable+0x24c/0x2d4
> > [    8.185790]  kernel_init+0x20/0x134
> > [    8.189271]  ret_from_fork+0x10/0x18
> > [    8.192840] Code: a90363f7 aa0103f5 d0010957 f9401260 (b9402800) 
> > [    8.198955] ---[ end trace c24172add816c1f0 ]---
> > [    8.203562] Kernel panic - not syncing: Oops: Fatal exception in interrupt
> > [    8.210442] SMP: stopping secondary CPUs
> > [    9.258360] SMP: failed to stop secondary CPUs 0,9
> > [    9.263141] Kernel Offset: disabled
> > [    9.266617] CPU features: 0x00040002,69101108
> > [    9.270963] Memory Limit: none
> > [    9.274024] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---
> 
> Please feed this stacktrace to scripts/decode_stacktrace.sh so that I
> can get an idea about what is going wrong. I bet something is playing
> ungodly games with the one of the IPIs, and things go horribly wrong.

hey Marc,
  Sure:

[    7.927289] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
[    7.936326] Mem abort info:
[    7.939108]   ESR = 0x96000004
[    7.942151]   EC = 0x25: DABT (current EL), IL = 32 bits
[    7.947451]   SET = 0, FnV = 0
[    7.950494]   EA = 0, S1PTW = 0
[    7.953624] Data abort info:
[    7.956492]   ISV = 0, ISS = 0x00000004
[    7.960316]   CM = 0, WnR = 0
[    7.963273] [0000000000000028] user address but active_mm is swapper
[    7.969616] Internal error: Oops: 96000004 [#1] SMP
[    7.974483] Modules linked in:
[    7.977531] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 5.12.0-rc8 #19
[    7.983874] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    7.990737] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
[    7.996732] pc : __ipi_send_mask (/home/ubuntu/linux/./include/linux/irqdomain.h:537 /home/ubuntu/linux/kernel/irq/ipi.c:283) 
[    8.000910] lr : smp_cross_call (/home/ubuntu/linux/arch/arm64/kernel/smp.c:958) 
[    8.004913] sp : ffff800012753c10
[    8.008216] x29: ffff800012753c10 x28: ffff000100de5d00
[    8.013521] x27: 000000000000000a x26: ffff80001225da20
[    8.018825] x25: 0000000000000000 x24: ffff000ff62719b0
[    8.024129] x23: ffff80001225d000 x22: ffff800012368108
[    8.029433] x21: ffff800010f69a20 x20: 0000000000000000
[    8.034737] x19: ffff000100143c60 x18: 0000000000000020
[    8.040041] x17: 000000008e74252f x16: 00000000bf0ab2ad
[    8.045345] x15: ffffffffffffffff x14: 0000000000000000
[    8.050649] x13: 003d090000000000 x12: 00003d0900000000
[    8.055953] x11: 0000000000000000 x10: 00003d0900000000
[    8.061257] x9 : ffff800010027f14 x8 : 0000000000000000
[    8.066561] x7 : 00000000ffffffff x6 : ffff000ff6148698
[    8.071865] x5 : ffff80001159d040 x4 : ffff80001159d110
[    8.077169] x3 : ffff800010f69a00 x2 : 0000000000000000
[    8.082473] x1 : ffff800010f69a20 x0 : 0000000000000000
[    8.087777] Call trace:
[    8.090213] __ipi_send_mask (/home/ubuntu/linux/./include/linux/irqdomain.h:537 /home/ubuntu/linux/kernel/irq/ipi.c:283) 
[    8.094038] smp_cross_call (/home/ubuntu/linux/arch/arm64/kernel/smp.c:958) 
[    8.097691] smp_send_reschedule (/home/ubuntu/linux/arch/arm64/kernel/smp.c:1011) 
[    8.101778] resched_curr (/home/ubuntu/linux/kernel/sched/core.c:621) 
[    8.105258] check_preempt_curr (/home/ubuntu/linux/kernel/sched/core.c:1715) 
[    8.109258] ttwu_do_wakeup (/home/ubuntu/linux/kernel/sched/core.c:2940) 
[    8.112996] ttwu_do_activate (/home/ubuntu/linux/kernel/sched/core.c:2991) 
[    8.116909] try_to_wake_up (/home/ubuntu/linux/kernel/sched/sched.h:1258 /home/ubuntu/linux/kernel/sched/sched.h:1353 /home/ubuntu/linux/kernel/sched/core.c:3187 /home/ubuntu/linux/kernel/sched/core.c:3464) 
[    8.120735] wake_up_process (/home/ubuntu/linux/kernel/sched/core.c:3534) 
[    8.124474] swake_up_one (/home/ubuntu/linux/./include/linux/list.h:135 /home/ubuntu/linux/./include/linux/list.h:204 /home/ubuntu/linux/kernel/sched/swait.c:31 /home/ubuntu/linux/kernel/sched/swait.c:22 /home/ubuntu/linux/kernel/sched/swait.c:53) 
[    8.127953] rcu_gp_kthread_wake (/home/ubuntu/linux/kernel/rcu/tree.c:1528) 
[    8.132041] rcu_accelerate_cbs_unlocked (/home/ubuntu/linux/kernel/rcu/tree.c:1606) 
[    8.136822] rcu_core (/home/ubuntu/linux/kernel/rcu/tree_plugin.h:2801 /home/ubuntu/linux/kernel/rcu/tree.c:2786) 
[    8.140128] rcu_core_si (/home/ubuntu/linux/kernel/rcu/tree.c:2808) 
[    8.143520] __do_softirq (/home/ubuntu/linux/./arch/arm64/include/asm/jump_label.h:21 /home/ubuntu/linux/./include/linux/jump_label.h:200 /home/ubuntu/linux/./include/trace/events/irq.h:142 /home/ubuntu/linux/kernel/softirq.c:346) 
[    8.147172] irq_exit (/home/ubuntu/linux/./arch/arm64/include/asm/percpu.h:43 /home/ubuntu/linux/kernel/softirq.c:402 /home/ubuntu/linux/kernel/softirq.c:424 /home/ubuntu/linux/kernel/softirq.c:446) 
[    8.150304] __handle_domain_irq (/home/ubuntu/linux/./include/asm-generic/irq_regs.h:29 /home/ubuntu/linux/kernel/irq/irqdesc.c:693) 
[    8.154394] gic_handle_irq (/home/ubuntu/linux/./arch/arm64/include/asm/arch_gicv3.h:45 /home/ubuntu/linux/drivers/irqchip/irq-gic-v3.c:222 /home/ubuntu/linux/drivers/irqchip/irq-gic-v3.c:649) 
[    8.158132] el1_irq (/home/ubuntu/linux/arch/arm64/kernel/entry.S:671) 
[    8.161262] __pi_strcmp (/home/ubuntu/linux/arch/arm64/lib/strcmp.S:66) 
[    8.164742] driver_register (/home/ubuntu/linux/drivers/base/driver.c:216 /home/ubuntu/linux/drivers/base/driver.c:164) 
[    8.168571] __platform_driver_register (/home/ubuntu/linux/drivers/base/platform.c:896) 
[    8.173265] imx8mp_clk_driver_init (/home/ubuntu/linux/drivers/clk/imx/clk-imx8mp.c:769) 
[    8.177614] do_one_initcall (/home/ubuntu/linux/init/main.c:1226) 
[    8.181440] kernel_init_freeable (/home/ubuntu/linux/init/main.c:1298 /home/ubuntu/linux/init/main.c:1315 /home/ubuntu/linux/init/main.c:1335 /home/ubuntu/linux/init/main.c:1537) 
[    8.185790] kernel_init (/home/ubuntu/linux/init/main.c:1426) 
[    8.189271] ret_from_fork (/home/ubuntu/linux/arch/arm64/kernel/entry.S:961) 
[ 8.192840] Code: a90363f7 aa0103f5 d0010957 f9401260 (b9402800)
All code
========
   0:	a90363f7 	stp	x23, x24, [sp, #48]
   4:	aa0103f5 	mov	x21, x1
   8:	d0010957 	adrp	x23, 0x212a000
   c:	f9401260 	ldr	x0, [x19, #32]
  10:*	b9402800 	ldr	w0, [x0, #40]		<-- trapping instruction

Code starting with the faulting instruction
===========================================
   0:	b9402800 	ldr	w0, [x0, #40]
[    8.198955] ---[ end trace c24172add816c1f0 ]---
[    8.203562] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[    8.210442] SMP: stopping secondary CPUs
[    9.258360] SMP: failed to stop secondary CPUs 0,9
[    9.263141] Kernel Offset: disabled
[    9.266617] CPU features: 0x00040002,69101108
[    9.270963] Memory Limit: none
[    9.274024] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---


> Now, here's a hunch: in the fine TX1 tradition, the firmware is broken
> and the GTDT table looks unusable. Amusingly, the crash happens right
> after the SBSA watchdog fails to probe.

Yeah, I noticed that, but didn't highlight it as I didn't see it in
the backtrace...

> And looking at the code that implements that driver, it looks dodgy as
> hell, as it unmaps an interrupt it doesn't even know is valid. And it
> does that right when the driver fails the way you experienced it. If,
> by any chance, the interrupt field is 0 in the firmware table, this
> results in SGI0 being unmapped. Given that this is the rescheduling
> interrupt, fireworks happen.

... and that explains why. I wouldn't have gotten there, but wish I'd
thought to test w/ the watchdog compiled out :(

> Can you have a go with the patchlet below, and let me know if that
> helps?

It does!

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
[    0.000000] Linux version 5.12.0-rc8+ (ubuntu@seidel) (gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #20 SMP Wed Apr 21 14:22:16 UTC 2021
[    0.000000] Machine model: Cavium ThunderX CN88XX board
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi: ESRT=0xffce0ff18 SMBIOS 3.0=0xfffb0000 ACPI 2.0=0xffec60000 MEMRESERVE=0xffc889d98 
[    0.000000] esrt: Reserving ESRT space from 0x0000000ffce0ff18 to 0x0000000ffce0ff50.
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x0000000FFEC60000 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000FFEC60028 00008C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000FFEC600B8 000114 (v06 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000FFEC601D0 00220B (v02 CAVIUM THUNDERX 00000001 INTL 20130517)
[    0.000000] ACPI: SPMI 0x0000000FFEC623E0 000041 (v05 ALASKA A M I    00000000 AMI. 00000000)
[    0.000000] ACPI: FIDT 0x0000000FFEC62428 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: APIC 0x0000000FFEC624C8 000F68 (v03 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: DBG2 0x0000000FFEC63430 000067 (v01 CAVIUM CN88XDBG 00000000 INTL 20150619)
[    0.000000] ACPI: GTDT 0x0000000FFEC63498 0000E0 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: IORT 0x0000000FFEC63578 0013D4 (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: MCFG 0x0000000FFEC64950 00006C (v01 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SSDT 0x0000000FFEC649C0 00089C (v02 CAVIUM NETWORK  00000001 INTL 20150619)
[    0.000000] ACPI: OEM1 0x0000000FFEC65260 0001E8 (v02 CAVIUM THUNDERX 00000001 INTL 20150619)
[    0.000000] ACPI: SLIT 0x0000000FFEC65448 000030 (v01 CAVIUM TEMPLATE 00000001 INTL 20150619)
[    0.000000] ACPI: SPCR 0x0000000FFEC65478 000050 (v02 A M I  APTIO V  01072009 AMI. 0005000B)
[    0.000000] ACPI: BGRT 0x0000000FFEC654C8 000038 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0x87e024000000,115200
[    0.000000] efi_bgrt: Ignoring BGRT: Incorrect BMP magic number 0x3707 (expected 0x4d42)
[    0.000000] NUMA: Failed to initialise from firmware
[    0.000000] NUMA: Faking a node at [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000] NUMA: NODE_DATA [mem 0xff677c480-0xff6780fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000500000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x0000000fff0fffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000500000-0x0000000000dfffff]
[    0.000000]   node   0: [mem 0x0000000000e00000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000010000000-0x00000000102fffff]
[    0.000000]   node   0: [mem 0x0000000010300000-0x00000000fff9ffff]
[    0.000000]   node   0: [mem 0x00000000fffa0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x0000000fee6affff]
[    0.000000]   node   0: [mem 0x0000000fee6b0000-0x0000000fee6cffff]
[    0.000000]   node   0: [mem 0x0000000fee6d0000-0x0000000ffac2ffff]
[    0.000000]   node   0: [mem 0x0000000ffac30000-0x0000000ffb0affff]
[    0.000000]   node   0: [mem 0x0000000ffb0b0000-0x0000000ffc8bffff]
[    0.000000]   node   0: [mem 0x0000000ffc8c0000-0x0000000ffc91ffff]
[    0.000000]   node   0: [mem 0x0000000ffc920000-0x0000000ffca3ffff]
[    0.000000]   node   0: [mem 0x0000000ffca40000-0x0000000ffca4ffff]
[    0.000000]   node   0: [mem 0x0000000ffca50000-0x0000000ffcdbffff]
[    0.000000]   node   0: [mem 0x0000000ffcdc0000-0x0000000ffd12ffff]
[    0.000000]   node   0: [mem 0x0000000ffd130000-0x0000000ffec2ffff]
[    0.000000]   node   0: [mem 0x0000000ffec30000-0x0000000ffec5ffff]
[    0.000000]   node   0: [mem 0x0000000ffec60000-0x0000000ffec6ffff]
[    0.000000]   node   0: [mem 0x0000000ffec70000-0x0000000ffed1ffff]
[    0.000000]   node   0: [mem 0x0000000ffed20000-0x0000000fff0fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000500000-0x0000000fff0fffff]
[    0.000000] On node 0 totalpages: 16772096
[    0.000000]   DMA zone: 16364 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 1047296 pages, LIFO batch:63
[    0.000000]   Normal zone: 245700 pages used for memmap
[    0.000000]   Normal zone: 15724800 pages, LIFO batch:63
[    0.000000]   Normal zone: 256 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x00000000fdc00000
[    0.000000] psci: probing for conduit method from ACPI.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] ACPI: SRAT not present
[    0.000000] percpu: Embedded 33 pages/cpu s98072 r8192 d28904 u135168
[    0.000000] pcpu-alloc: s98072 r8192 d28904 u135168 alloc=33*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23 
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31 
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39 
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42 [0] 43 [0] 44 [0] 45 [0] 46 [0] 47 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Software prefetching using PRFM
[    0.000000] CPU features: detected: Cavium erratum 27456
[    0.000000] CPU features: detected: Cavium erratum 30115
[    0.000000] CPU features: kernel page table isolation forced OFF by ARM64_WORKAROUND_CAVIUM_27456
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16510032
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-dannf root=UUID=76e81d96-78e4-4432-a490-bb852f6111e8 ro sysrq_always_enabled acpi=force
[    0.000000] sysrq: sysrq always enabled.
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 192512 bytes
[    0.000000] printk: log_buf_len min size: 262144 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 255872(97%)
[    0.000000] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f9c00000-0x00000000fdc00000] (64MB)
[    0.000000] Memory: 65572072K/67088384K available (15616K kernel code, 3626K rwdata, 11808K rodata, 7616K init, 1157K bss, 1483544K reserved, 32768K cma-reserved)
[    0.000000] random: get_random_u64 called from __kmem_cache_create+0x38/0x5b4 with crng_init=0
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=48, Nodes=1
[    0.000000] ftrace: allocating 49728 entries in 195 pages
[    0.000000] ftrace: allocated 195 pages with 4 groups
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=48.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=48
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GIC: enabling workaround for GICv3: Cavium erratum 38539
[    0.000000] GICv3: 128 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000801080000000
[    0.000000] ACPI: SRAT not present
[    0.000000] ITS [mem 0x801000020000-0x80100003ffff]
[    0.000000] ITS@0x0000801000020000: Devices Table too large, reduce ids 21->19
[    0.000000] ITS@0x0000801000020000: allocated 524288 Devices @100800000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000100270000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000100280000
[    0.000000] arch_timer: Failed to initialize memory-mapped timer.
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000000] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000150] Console: colour dummy device 80x25
[    0.000232] ACPI: Core revision 20210105
[    0.000452] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.000462] pid_max: default: 49152 minimum: 384
[    0.000575] LSM: Security Framework initializing
[    0.000590] Yama: becoming mindful.
[    0.000650] AppArmor: AppArmor initialized
[    0.000804] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000869] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.001661] ACPI PPTT: No PPTT table found, CPU and cache topology may be inaccurate
[    0.002248] rcu: Hierarchical SRCU implementation.
[    0.002999] Platform MSI: ITS@0x801000020000 domain created
[    0.003014] PCI/MSI: ITS@0x801000020000 domain created
[    0.003024] fsl-mc MSI: ITS@0x801000020000 domain created
[    0.003041] Remapping and enabling EFI services.
[    0.005152] smp: Bringing up secondary CPUs ...
[    0.005634] Detected VIPT I-cache on CPU1
[    0.005649] GICv3: CPU1: found redistributor 1 region 0:0x0000801080020000
[    0.005674] GICv3: CPU1: using allocated LPI pending table @0x0000000100290000
[    0.005694] CPU1: Booted secondary processor 0x0000000001 [0x431f0a11]
[    0.006251] Detected VIPT I-cache on CPU2
[    0.006262] GICv3: CPU2: found redistributor 2 region 0:0x0000801080040000
[    0.006286] GICv3: CPU2: using allocated LPI pending table @0x00000001002a0000
[    0.006305] CPU2: Booted secondary processor 0x0000000002 [0x431f0a11]
[    0.006848] Detected VIPT I-cache on CPU3
[    0.006858] GICv3: CPU3: found redistributor 3 region 0:0x0000801080060000
[    0.006884] GICv3: CPU3: using allocated LPI pending table @0x00000001002b0000
[    0.006903] CPU3: Booted secondary processor 0x0000000003 [0x431f0a11]
[    0.007444] Detected VIPT I-cache on CPU4
[    0.007455] GICv3: CPU4: found redistributor 4 region 0:0x0000801080080000
[    0.007480] GICv3: CPU4: using allocated LPI pending table @0x00000001002c0000
[    0.007500] CPU4: Booted secondary processor 0x0000000004 [0x431f0a11]
[    0.008046] Detected VIPT I-cache on CPU5
[    0.008057] GICv3: CPU5: found redistributor 5 region 0:0x00008010800a0000
[    0.008082] GICv3: CPU5: using allocated LPI pending table @0x00000001002d0000
[    0.008102] CPU5: Booted secondary processor 0x0000000005 [0x431f0a11]
[    0.008639] Detected VIPT I-cache on CPU6
[    0.008650] GICv3: CPU6: found redistributor 6 region 0:0x00008010800c0000
[    0.008677] GICv3: CPU6: using allocated LPI pending table @0x00000001002e0000
[    0.008697] CPU6: Booted secondary processor 0x0000000006 [0x431f0a11]
[    0.009242] Detected VIPT I-cache on CPU7
[    0.009253] GICv3: CPU7: found redistributor 7 region 0:0x00008010800e0000
[    0.009279] GICv3: CPU7: using allocated LPI pending table @0x00000001002f0000
[    0.009299] CPU7: Booted secondary processor 0x0000000007 [0x431f0a11]
[    0.009839] Detected VIPT I-cache on CPU8
[    0.009851] GICv3: CPU8: found redistributor 8 region 0:0x0000801080100000
[    0.009876] GICv3: CPU8: using allocated LPI pending table @0x0000000100300000
[    0.009897] CPU8: Booted secondary processor 0x0000000008 [0x431f0a11]
[    0.010450] Detected VIPT I-cache on CPU9
[    0.010462] GICv3: CPU9: found redistributor 9 region 0:0x0000801080120000
[    0.010487] GICv3: CPU9: using allocated LPI pending table @0x0000000100310000
[    0.010508] CPU9: Booted secondary processor 0x0000000009 [0x431f0a11]
[    0.011054] Detected VIPT I-cache on CPU10
[    0.011066] GICv3: CPU10: found redistributor a region 0:0x0000801080140000
[    0.011091] GICv3: CPU10: using allocated LPI pending table @0x0000000100320000
[    0.011113] CPU10: Booted secondary processor 0x000000000a [0x431f0a11]
[    0.011661] Detected VIPT I-cache on CPU11
[    0.011673] GICv3: CPU11: found redistributor b region 0:0x0000801080160000
[    0.011699] GICv3: CPU11: using allocated LPI pending table @0x0000000100330000
[    0.011721] CPU11: Booted secondary processor 0x000000000b [0x431f0a11]
[    0.012279] Detected VIPT I-cache on CPU12
[    0.012291] GICv3: CPU12: found redistributor c region 0:0x0000801080180000
[    0.012316] GICv3: CPU12: using allocated LPI pending table @0x0000000100340000
[    0.012338] CPU12: Booted secondary processor 0x000000000c [0x431f0a11]
[    0.012901] Detected VIPT I-cache on CPU13
[    0.012914] GICv3: CPU13: found redistributor d region 0:0x00008010801a0000
[    0.012941] GICv3: CPU13: using allocated LPI pending table @0x0000000100350000
[    0.012963] CPU13: Booted secondary processor 0x000000000d [0x431f0a11]
[    0.013511] Detected VIPT I-cache on CPU14
[    0.013524] GICv3: CPU14: found redistributor e region 0:0x00008010801c0000
[    0.013551] GICv3: CPU14: using allocated LPI pending table @0x0000000100360000
[    0.013573] CPU14: Booted secondary processor 0x000000000e [0x431f0a11]
[    0.014123] Detected VIPT I-cache on CPU15
[    0.014136] GICv3: CPU15: found redistributor f region 0:0x00008010801e0000
[    0.014165] GICv3: CPU15: using allocated LPI pending table @0x0000000100370000
[    0.014188] CPU15: Booted secondary processor 0x000000000f [0x431f0a11]
[    0.014732] Detected VIPT I-cache on CPU16
[    0.014745] GICv3: CPU16: found redistributor 100 region 0:0x0000801080200000
[    0.014772] GICv3: CPU16: using allocated LPI pending table @0x0000000100380000
[    0.014795] CPU16: Booted secondary processor 0x0000000100 [0x431f0a11]
[    0.015335] Detected VIPT I-cache on CPU17
[    0.015348] GICv3: CPU17: found redistributor 101 region 0:0x0000801080220000
[    0.015375] GICv3: CPU17: using allocated LPI pending table @0x0000000100390000
[    0.015398] CPU17: Booted secondary processor 0x0000000101 [0x431f0a11]
[    0.015967] Detected VIPT I-cache on CPU18
[    0.015980] GICv3: CPU18: found redistributor 102 region 0:0x0000801080240000
[    0.016007] GICv3: CPU18: using allocated LPI pending table @0x00000001003a0000
[    0.016030] CPU18: Booted secondary processor 0x0000000102 [0x431f0a11]
[    0.016592] Detected VIPT I-cache on CPU19
[    0.016606] GICv3: CPU19: found redistributor 103 region 0:0x0000801080260000
[    0.016634] GICv3: CPU19: using allocated LPI pending table @0x00000001003b0000
[    0.016658] CPU19: Booted secondary processor 0x0000000103 [0x431f0a11]
[    0.017210] Detected VIPT I-cache on CPU20
[    0.017224] GICv3: CPU20: found redistributor 104 region 0:0x0000801080280000
[    0.017252] GICv3: CPU20: using allocated LPI pending table @0x00000001003c0000
[    0.017276] CPU20: Booted secondary processor 0x0000000104 [0x431f0a11]
[    0.017823] Detected VIPT I-cache on CPU21
[    0.017837] GICv3: CPU21: found redistributor 105 region 0:0x00008010802a0000
[    0.017864] GICv3: CPU21: using allocated LPI pending table @0x00000001003d0000
[    0.017889] CPU21: Booted secondary processor 0x0000000105 [0x431f0a11]
[    0.018451] Detected VIPT I-cache on CPU22
[    0.018465] GICv3: CPU22: found redistributor 106 region 0:0x00008010802c0000
[    0.018493] GICv3: CPU22: using allocated LPI pending table @0x00000001003e0000
[    0.018517] CPU22: Booted secondary processor 0x0000000106 [0x431f0a11]
[    0.019081] Detected VIPT I-cache on CPU23
[    0.019095] GICv3: CPU23: found redistributor 107 region 0:0x00008010802e0000
[    0.019124] GICv3: CPU23: using allocated LPI pending table @0x00000001003f0000
[    0.019148] CPU23: Booted secondary processor 0x0000000107 [0x431f0a11]
[    0.019699] Detected VIPT I-cache on CPU24
[    0.019714] GICv3: CPU24: found redistributor 108 region 0:0x0000801080300000
[    0.019741] GICv3: CPU24: using allocated LPI pending table @0x0000000100c00000
[    0.019766] CPU24: Booted secondary processor 0x0000000108 [0x431f0a11]
[    0.020343] Detected VIPT I-cache on CPU25
[    0.020358] GICv3: CPU25: found redistributor 109 region 0:0x0000801080320000
[    0.020386] GICv3: CPU25: using allocated LPI pending table @0x0000000100c10000
[    0.020411] CPU25: Booted secondary processor 0x0000000109 [0x431f0a11]
[    0.020962] Detected VIPT I-cache on CPU26
[    0.020977] GICv3: CPU26: found redistributor 10a region 0:0x0000801080340000
[    0.021005] GICv3: CPU26: using allocated LPI pending table @0x0000000100c20000
[    0.021030] CPU26: Booted secondary processor 0x000000010a [0x431f0a11]
[    0.021598] Detected VIPT I-cache on CPU27
[    0.021613] GICv3: CPU27: found redistributor 10b region 0:0x0000801080360000
[    0.021641] GICv3: CPU27: using allocated LPI pending table @0x0000000100c30000
[    0.021667] CPU27: Booted secondary processor 0x000000010b [0x431f0a11]
[    0.022219] Detected VIPT I-cache on CPU28
[    0.022234] GICv3: CPU28: found redistributor 10c region 0:0x0000801080380000
[    0.022263] GICv3: CPU28: using allocated LPI pending table @0x0000000100c40000
[    0.022288] CPU28: Booted secondary processor 0x000000010c [0x431f0a11]
[    0.022850] Detected VIPT I-cache on CPU29
[    0.022865] GICv3: CPU29: found redistributor 10d region 0:0x00008010803a0000
[    0.022895] GICv3: CPU29: using allocated LPI pending table @0x0000000100c50000
[    0.022921] CPU29: Booted secondary processor 0x000000010d [0x431f0a11]
[    0.023482] Detected VIPT I-cache on CPU30
[    0.023498] GICv3: CPU30: found redistributor 10e region 0:0x00008010803c0000
[    0.023527] GICv3: CPU30: using allocated LPI pending table @0x0000000100c60000
[    0.023553] CPU30: Booted secondary processor 0x000000010e [0x431f0a11]
[    0.024116] Detected VIPT I-cache on CPU31
[    0.024132] GICv3: CPU31: found redistributor 10f region 0:0x00008010803e0000
[    0.024161] GICv3: CPU31: using allocated LPI pending table @0x0000000100c70000
[    0.024188] CPU31: Booted secondary processor 0x000000010f [0x431f0a11]
[    0.024762] Detected VIPT I-cache on CPU32
[    0.024778] GICv3: CPU32: found redistributor 200 region 0:0x0000801080400000
[    0.024806] GICv3: CPU32: using allocated LPI pending table @0x0000000100c80000
[    0.024833] CPU32: Booted secondary processor 0x0000000200 [0x431f0a11]
[    0.025402] Detected VIPT I-cache on CPU33
[    0.025418] GICv3: CPU33: found redistributor 201 region 0:0x0000801080420000
[    0.025447] GICv3: CPU33: using allocated LPI pending table @0x0000000100c90000
[    0.025475] CPU33: Booted secondary processor 0x0000000201 [0x431f0a11]
[    0.026040] Detected VIPT I-cache on CPU34
[    0.026056] GICv3: CPU34: found redistributor 202 region 0:0x0000801080440000
[    0.026086] GICv3: CPU34: using allocated LPI pending table @0x0000000100ca0000
[    0.026113] CPU34: Booted secondary processor 0x0000000202 [0x431f0a11]
[    0.026681] Detected VIPT I-cache on CPU35
[    0.026698] GICv3: CPU35: found redistributor 203 region 0:0x0000801080460000
[    0.026727] GICv3: CPU35: using allocated LPI pending table @0x0000000100cb0000
[    0.026755] CPU35: Booted secondary processor 0x0000000203 [0x431f0a11]
[    0.027317] Detected VIPT I-cache on CPU36
[    0.027334] GICv3: CPU36: found redistributor 204 region 0:0x0000801080480000
[    0.027364] GICv3: CPU36: using allocated LPI pending table @0x0000000100cc0000
[    0.027392] CPU36: Booted secondary processor 0x0000000204 [0x431f0a11]
[    0.027955] Detected VIPT I-cache on CPU37
[    0.027972] GICv3: CPU37: found redistributor 205 region 0:0x00008010804a0000
[    0.028002] GICv3: CPU37: using allocated LPI pending table @0x0000000100cd0000
[    0.028030] CPU37: Booted secondary processor 0x0000000205 [0x431f0a11]
[    0.028609] Detected VIPT I-cache on CPU38
[    0.028626] GICv3: CPU38: found redistributor 206 region 0:0x00008010804c0000
[    0.028656] GICv3: CPU38: using allocated LPI pending table @0x0000000100ce0000
[    0.028684] CPU38: Booted secondary processor 0x0000000206 [0x431f0a11]
[    0.029243] Detected VIPT I-cache on CPU39
[    0.029261] GICv3: CPU39: found redistributor 207 region 0:0x00008010804e0000
[    0.029291] GICv3: CPU39: using allocated LPI pending table @0x0000000100cf0000
[    0.029320] CPU39: Booted secondary processor 0x0000000207 [0x431f0a11]
[    0.029895] Detected VIPT I-cache on CPU40
[    0.029912] GICv3: CPU40: found redistributor 208 region 0:0x0000801080500000
[    0.029942] GICv3: CPU40: using allocated LPI pending table @0x0000000100d00000
[    0.029971] CPU40: Booted secondary processor 0x0000000208 [0x431f0a11]
[    0.030554] Detected VIPT I-cache on CPU41
[    0.030572] GICv3: CPU41: found redistributor 209 region 0:0x0000801080520000
[    0.030603] GICv3: CPU41: using allocated LPI pending table @0x0000000100d10000
[    0.030632] CPU41: Booted secondary processor 0x0000000209 [0x431f0a11]
[    0.031197] Detected VIPT I-cache on CPU42
[    0.031214] GICv3: CPU42: found redistributor 20a region 0:0x0000801080540000
[    0.031245] GICv3: CPU42: using allocated LPI pending table @0x0000000100d20000
[    0.031275] CPU42: Booted secondary processor 0x000000020a [0x431f0a11]
[    0.031853] Detected VIPT I-cache on CPU43
[    0.031871] GICv3: CPU43: found redistributor 20b region 0:0x0000801080560000
[    0.031904] GICv3: CPU43: using allocated LPI pending table @0x0000000100d30000
[    0.031933] CPU43: Booted secondary processor 0x000000020b [0x431f0a11]
[    0.032528] Detected VIPT I-cache on CPU44
[    0.032546] GICv3: CPU44: found redistributor 20c region 0:0x0000801080580000
[    0.032578] GICv3: CPU44: using allocated LPI pending table @0x0000000100d40000
[    0.032607] CPU44: Booted secondary processor 0x000000020c [0x431f0a11]
[    0.033191] Detected VIPT I-cache on CPU45
[    0.033210] GICv3: CPU45: found redistributor 20d region 0:0x00008010805a0000
[    0.033241] GICv3: CPU45: using allocated LPI pending table @0x0000000100d50000
[    0.033271] CPU45: Booted secondary processor 0x000000020d [0x431f0a11]
[    0.033843] Detected VIPT I-cache on CPU46
[    0.033861] GICv3: CPU46: found redistributor 20e region 0:0x00008010805c0000
[    0.033893] GICv3: CPU46: using allocated LPI pending table @0x0000000100d60000
[    0.033924] CPU46: Booted secondary processor 0x000000020e [0x431f0a11]
[    0.034503] Detected VIPT I-cache on CPU47
[    0.034522] GICv3: CPU47: found redistributor 20f region 0:0x00008010805e0000
[    0.034554] GICv3: CPU47: using allocated LPI pending table @0x0000000100d70000
[    0.034584] CPU47: Booted secondary processor 0x000000020f [0x431f0a11]
[    0.034694] smp: Brought up 1 node, 48 CPUs
[    0.034960] SMP: Total of 48 processors activated.
[    0.034966] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.034971] CPU features: detected: CRC32 instructions
[    0.035202] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.044471] CPU: All CPU(s) started at EL2
[    0.044630] alternatives: patching kernel code
[    0.058585] devtmpfs: initialized
[    0.078235] Registered cp15_barrier emulation handler
[    0.078260] Registered setend emulation handler
[    0.078268] KASLR disabled due to lack of seed
[    0.078435] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.078504] futex hash table entries: 16384 (order: 8, 1048576 bytes, linear)
[    0.079924] pinctrl core: initialized pinctrl subsystem
[    0.080485] SMBIOS 3.0.0 present.
[    0.080500] DMI: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
[    0.080988] NET: Registered protocol family 16
[    0.082760] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.083231] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.083710] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.083743] audit: initializing netlink subsys (disabled)
[    0.083898] audit: type=2000 audit(0.080:1): state=initialized audit_enabled=0 res=1
[    0.084439] thermal_sys: Registered thermal governor 'fair_share'
[    0.084445] thermal_sys: Registered thermal governor 'bang_bang'
[    0.084450] thermal_sys: Registered thermal governor 'step_wise'
[    0.084454] thermal_sys: Registered thermal governor 'user_space'
[    0.084459] thermal_sys: Registered thermal governor 'power_allocator'
[    0.084640] cpuidle: using governor ladder
[    0.084902] cpuidle: using governor menu
[    0.085048] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.085632] ASID allocator initialised with 65536 entries
[    0.085922] ACPI: bus type PCI registered
[    0.085930] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.086167] Serial: AMBA PL011 UART driver
[    0.096900] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.096908] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.096914] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.096919] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.101966] ACPI: Added _OSI(Module Device)
[    0.101973] ACPI: Added _OSI(Processor Device)
[    0.101978] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.101983] ACPI: Added _OSI(Processor Aggregator Device)
[    0.101988] ACPI: Added _OSI(Linux-Dell-Video)
[    0.101993] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.101998] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.105918] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.107812] ACPI: Interpreter enabled
[    0.107820] ACPI: Using GIC for interrupt routing
[    0.107848] ACPI: MCFG table detected, 4 entries
[    0.131510] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1f])
[    0.131526] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.131675] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    0.131804] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability]
[    0.132264] acpi PNP0A08:00: ECAM area [mem 0x848000000000-0x848001ffffff] reserved by CAVA02C:00
[    0.132282] acpi PNP0A08:00: ECAM at [mem 0x848000000000-0x848001ffffff] for [bus 00-1f]
[    0.132438] PCI host bridge to bus 0000:00
[    0.132446] pci_bus 0000:00: root bus resource [mem 0x801000000000-0x807fffffffff window]
[    0.132453] pci_bus 0000:00: root bus resource [mem 0x838000000000-0x841fffffffff window]
[    0.132460] pci_bus 0000:00: root bus resource [mem 0x846000000000-0x847fffffffff window]
[    0.132466] pci_bus 0000:00: root bus resource [mem 0x868000000000-0x87e023ffffff window]
[    0.132472] pci_bus 0000:00: root bus resource [mem 0x87e026000000-0x87e0bfffffff window]
[    0.132479] pci_bus 0000:00: root bus resource [mem 0x87e0c6000000-0x87ffffffffff window]
[    0.132486] pci_bus 0000:00: root bus resource [bus 00-1f]
[    0.132516] pci 0000:00:01.0: [177d:a002] type 01 class 0x060400
[    0.132688] pci 0000:00:09.0: [177d:a018] type 00 class 0x120000
[    0.132723] pci 0000:00:09.0: BAR 0: [mem 0x87e040000000-0x87e0400fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    0.132733] pci 0000:00:09.0: VF BAR 0: [mem 0x840000800000-0x8400008fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.153628] pci 0000:00:09.0: VF(n) BAR0 space: [mem 0x840000800000-0x8400008fffff 64bit] (contains BAR0 for 1 VFs)
[    1.153780] pci 0000:00:10.0: [177d:a01b] type 00 class 0x0c0330
[    1.153817] pci 0000:00:10.0: BAR 0: [mem 0x868000000000-0x8680001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153826] pci 0000:00:10.0: BAR 4: [mem 0x868000200000-0x8680002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153953] pci 0000:00:11.0: [177d:a01b] type 00 class 0x0c0330
[    1.153989] pci 0000:00:11.0: BAR 0: [mem 0x869000000000-0x8690001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.153998] pci 0000:00:11.0: BAR 4: [mem 0x869000200000-0x8690002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154123] pci 0000:00:14.0: [177d:a002] type 01 class 0x060400
[    1.154325] pci 0000:01:00.0: [177d:a001] type 00 class 0x088000
[    1.154361] pci 0000:01:00.0: BAR 0: [mem 0x87e0fc000000-0x87e0fc0fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154370] pci 0000:01:00.0: BAR 4: [mem 0x87e0fcf00000-0x87e0fcffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154497] pci 0000:01:00.1: [177d:a00e] type 00 class 0x088000
[    1.154532] pci 0000:01:00.1: BAR 0: [mem 0x87e006000000-0x87e0067fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154541] pci 0000:01:00.1: BAR 4: [mem 0x87e006f00000-0x87e006ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154661] pci 0000:01:01.3: [177d:a02b] type 00 class 0x0c8000
[    1.154696] pci 0000:01:01.3: BAR 0: [mem 0x87e005000000-0x87e0057fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154814] pci 0000:01:01.4: [177d:a010] type 00 class 0x058000
[    1.154853] pci 0000:01:01.4: BAR 0: [mem 0x87e009000000-0x87e0097fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154862] pci 0000:01:01.4: BAR 4: [mem 0x87e009f00000-0x87e009ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.154983] pci 0000:01:06.0: [177d:a02e] type 00 class 0x058000
[    1.155020] pci 0000:01:06.0: BAR 0: [mem 0x87e050000000-0x87e0507fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155029] pci 0000:01:06.0: BAR 4: [mem 0x87e050f00000-0x87e050ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155155] pci 0000:01:06.1: [177d:a02e] type 00 class 0x058000
[    1.155191] pci 0000:01:06.1: BAR 0: [mem 0x87e051000000-0x87e0517fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155200] pci 0000:01:06.1: BAR 4: [mem 0x87e051f00000-0x87e051ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155322] pci 0000:01:06.2: [177d:a02e] type 00 class 0x058000
[    1.155357] pci 0000:01:06.2: BAR 0: [mem 0x87e052000000-0x87e0527fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155366] pci 0000:01:06.2: BAR 4: [mem 0x87e052f00000-0x87e052ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155485] pci 0000:01:06.3: [177d:a02e] type 00 class 0x058000
[    1.155521] pci 0000:01:06.3: BAR 0: [mem 0x87e053000000-0x87e0537fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155530] pci 0000:01:06.3: BAR 4: [mem 0x87e053f00000-0x87e053ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155649] pci 0000:01:06.4: [177d:a02e] type 00 class 0x058000
[    1.155685] pci 0000:01:06.4: BAR 0: [mem 0x87e054000000-0x87e0547fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155694] pci 0000:01:06.4: BAR 4: [mem 0x87e054f00000-0x87e054ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155816] pci 0000:01:06.5: [177d:a02e] type 00 class 0x058000
[    1.155852] pci 0000:01:06.5: BAR 0: [mem 0x87e055000000-0x87e0557fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155861] pci 0000:01:06.5: BAR 4: [mem 0x87e055f00000-0x87e055ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.155981] pci 0000:01:06.6: [177d:a02e] type 00 class 0x058000
[    1.156016] pci 0000:01:06.6: BAR 0: [mem 0x87e056000000-0x87e0567fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156025] pci 0000:01:06.6: BAR 4: [mem 0x87e056f00000-0x87e056ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156145] pci 0000:01:06.7: [177d:a02e] type 00 class 0x058000
[    1.156184] pci 0000:01:06.7: BAR 0: [mem 0x87e057000000-0x87e0577fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156193] pci 0000:01:06.7: BAR 4: [mem 0x87e057f00000-0x87e057ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156316] pci 0000:01:07.0: [177d:a02f] type 00 class 0x058000
[    1.156352] pci 0000:01:07.0: BAR 0: [mem 0x87e058000000-0x87e0587fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156361] pci 0000:01:07.0: BAR 4: [mem 0x87e058f00000-0x87e058ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156488] pci 0000:01:07.1: [177d:a02f] type 00 class 0x058000
[    1.156524] pci 0000:01:07.1: BAR 0: [mem 0x87e059000000-0x87e0597fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156533] pci 0000:01:07.1: BAR 4: [mem 0x87e059f00000-0x87e059ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156654] pci 0000:01:07.2: [177d:a02f] type 00 class 0x058000
[    1.156689] pci 0000:01:07.2: BAR 0: [mem 0x87e05a000000-0x87e05a7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156698] pci 0000:01:07.2: BAR 4: [mem 0x87e05af00000-0x87e05affffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156821] pci 0000:01:07.3: [177d:a02f] type 00 class 0x058000
[    1.156856] pci 0000:01:07.3: BAR 0: [mem 0x87e05b000000-0x87e05b7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156865] pci 0000:01:07.3: BAR 4: [mem 0x87e05bf00000-0x87e05bffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.156985] pci 0000:01:07.4: [177d:a030] type 00 class 0x058000
[    1.157020] pci 0000:01:07.4: BAR 0: [mem 0x87e05c000000-0x87e05c7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157029] pci 0000:01:07.4: BAR 4: [mem 0x87e05cf00000-0x87e05cffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157149] pci 0000:01:07.5: [177d:a030] type 00 class 0x058000
[    1.157185] pci 0000:01:07.5: BAR 0: [mem 0x87e05d000000-0x87e05d7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157194] pci 0000:01:07.5: BAR 4: [mem 0x87e05df00000-0x87e05dffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157316] pci 0000:01:07.6: [177d:a030] type 00 class 0x058000
[    1.157352] pci 0000:01:07.6: BAR 0: [mem 0x87e05e000000-0x87e05e7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157361] pci 0000:01:07.6: BAR 4: [mem 0x87e05ef00000-0x87e05effffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157484] pci 0000:01:07.7: [177d:a030] type 00 class 0x058000
[    1.157532] pci 0000:01:07.7: BAR 0: [mem 0x87e05f000000-0x87e05f7fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157542] pci 0000:01:07.7: BAR 4: [mem 0x87e05ff00000-0x87e05fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157677] pci 0000:01:09.2: [177d:a012] type 00 class 0x0c8000
[    1.157714] pci 0000:01:09.2: BAR 0: [mem 0x87e0d2000000-0x87e0d27fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157723] pci 0000:01:09.2: BAR 4: [mem 0x87e0d2f00000-0x87e0d2ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157851] pci 0000:01:09.4: [177d:a012] type 00 class 0x0c8000
[    1.157887] pci 0000:01:09.4: BAR 0: [mem 0x87e0d4000000-0x87e0d47fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.157896] pci 0000:01:09.4: BAR 4: [mem 0x87e0d4f00000-0x87e0d4ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158018] pci 0000:01:0a.0: [177d:a022] type 00 class 0x058000
[    1.158055] pci 0000:01:0a.0: BAR 0: [mem 0x87e088000000-0x87e0887fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158064] pci 0000:01:0a.0: BAR 4: [mem 0x87e088f00000-0x87e088ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158186] pci 0000:01:0a.1: [177d:a022] type 00 class 0x058000
[    1.158222] pci 0000:01:0a.1: BAR 0: [mem 0x87e089000000-0x87e0897fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158231] pci 0000:01:0a.1: BAR 4: [mem 0x87e089f00000-0x87e089ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158354] pci 0000:01:10.0: [177d:a026] type 00 class 0x028000
[    1.158389] pci 0000:01:10.0: BAR 0: [mem 0x87e0e0000000-0x87e0e03fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158398] pci 0000:01:10.0: BAR 4: [mem 0x87e0e0400000-0x87e0e07fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158520] pci 0000:01:10.1: [177d:a026] type 00 class 0x028000
[    1.158555] pci 0000:01:10.1: BAR 0: [mem 0x87e0e1000000-0x87e0e13fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158564] pci 0000:01:10.1: BAR 4: [mem 0x87e0e1400000-0x87e0e17fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158675] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.158842] pci 0000:02:00.0: [177d:a01d] type 00 class 0x010400
[    1.158880] pci 0000:02:00.0: BAR 0: [mem 0x870000000000-0x8700007fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.158890] pci 0000:02:00.0: BAR 4: [mem 0x870000f00000-0x870000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.159004] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    1.159080] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.159093] pci 0000:00:14.0: PCI bridge to [bus 02]
[    1.159101] pci_bus 0000:00: resource 4 [mem 0x801000000000-0x807fffffffff window]
[    1.159107] pci_bus 0000:00: resource 5 [mem 0x838000000000-0x841fffffffff window]
[    1.159114] pci_bus 0000:00: resource 6 [mem 0x846000000000-0x847fffffffff window]
[    1.159120] pci_bus 0000:00: resource 7 [mem 0x868000000000-0x87e023ffffff window]
[    1.159127] pci_bus 0000:00: resource 8 [mem 0x87e026000000-0x87e0bfffffff window]
[    1.159133] pci_bus 0000:00: resource 9 [mem 0x87e0c6000000-0x87ffffffffff window]
[    1.159289] ACPI: PCI Root Bridge [PCI1] (domain 0001 [bus 00-1f])
[    1.159302] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.159444] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.159573] acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
[    1.160046] acpi PNP0A08:01: ECAM area [mem 0x849000000000-0x849001ffffff] reserved by CAVA02C:01
[    1.160063] acpi PNP0A08:01: ECAM at [mem 0x849000000000-0x849001ffffff] for [bus 00-1f]
[    1.160192] PCI host bridge to bus 0001:00
[    1.160199] pci_bus 0001:00: root bus resource [mem 0x810000000000-0x817fffffffff window]
[    1.160207] pci_bus 0001:00: root bus resource [bus 00-1f]
[    1.160235] pci 0001:00:08.0: [177d:a01c] type 00 class 0x010601
[    1.160268] pci 0001:00:08.0: BAR 0: [mem 0x814000000000-0x8140001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160276] pci 0001:00:08.0: BAR 4: [mem 0x814000200000-0x8140002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160404] pci 0001:00:09.0: [177d:a01c] type 00 class 0x010601
[    1.160443] pci 0001:00:09.0: BAR 0: [mem 0x815000000000-0x8150001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160453] pci 0001:00:09.0: BAR 4: [mem 0x815000200000-0x8150002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160583] pci 0001:00:0a.0: [177d:a01c] type 00 class 0x010601
[    1.160618] pci 0001:00:0a.0: BAR 0: [mem 0x816000000000-0x8160001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160627] pci 0001:00:0a.0: BAR 4: [mem 0x816000200000-0x8160002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160752] pci 0001:00:0b.0: [177d:a01c] type 00 class 0x010601
[    1.160787] pci 0001:00:0b.0: BAR 0: [mem 0x817000000000-0x8170001fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160796] pci 0001:00:0b.0: BAR 4: [mem 0x817000200000-0x8170002fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.160928] pci_bus 0001:00: resource 4 [mem 0x810000000000-0x817fffffffff window]
[    1.161037] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-1f])
[    1.161049] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.161189] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    1.161317] acpi PNP0A08:02: _OSC: OS now controls [PCIeCapability]
[    1.161821] acpi PNP0A08:02: ECAM area [mem 0x84a000000000-0x84a001ffffff] reserved by CAVA02C:02
[    1.161837] acpi PNP0A08:02: ECAM at [mem 0x84a000000000-0x84a001ffffff] for [bus 00-1f]
[    1.161970] PCI host bridge to bus 0002:00
[    1.161977] pci_bus 0002:00: root bus resource [mem 0x842000000000-0x843fffffffff window]
[    1.161985] pci_bus 0002:00: root bus resource [bus 00-1f]
[    1.162008] pci 0002:00:02.0: [177d:a002] type 01 class 0x060400
[    1.162156] pci 0002:00:03.0: [177d:a01f] type 00 class 0x028000
[    1.162191] pci 0002:00:03.0: BAR 0: [mem 0x842000000000-0x84200000ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162200] pci 0002:00:03.0: BAR 2: [mem 0x842040000000-0x84207fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162208] pci 0002:00:03.0: BAR 4: [mem 0x842000f00000-0x842000ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162399] pci 0002:01:00.0: [177d:a01e] type 00 class 0x020000
[    1.162436] pci 0002:01:00.0: BAR 0: [mem 0x843000000000-0x84303fffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162445] pci 0002:01:00.0: BAR 4: [mem 0x843060000000-0x8430600fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    1.162453] pci 0002:01:00.0: VF BAR 0: [mem 0x8430a0000000-0x8430a01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    1.162462] pci 0002:01:00.0: VF BAR 4: [mem 0x8430e0000000-0x8430e01fffff 64bit] (from Enhanced Allocation, properties 0x4)
[    2.178959] pci 0002:01:00.0: VF(n) BAR0 space: [mem 0x8430a0000000-0x8430afffffff 64bit] (contains BAR0 for 128 VFs)
[    2.178973] pci 0002:01:00.0: VF(n) BAR4 space: [mem 0x8430e0000000-0x8430efffffff 64bit] (contains BAR4 for 128 VFs)
[    2.179182] pci 0002:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.179208] pci 0002:00:02.0: PCI bridge to [bus 01]
[    2.179217] pci_bus 0002:00: resource 4 [mem 0x842000000000-0x843fffffffff window]
[    2.179326] ACPI: PCI Root Bridge [PCI3] (domain 0003 [bus 00-1f])
[    2.179339] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.179483] acpi PNP0A08:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.179614] acpi PNP0A08:03: _OSC: OS now controls [PCIeCapability]
[    2.180134] acpi PNP0A08:03: ECAM area [mem 0x84b000000000-0x84b001ffffff] reserved by CAVA02C:03
[    2.180150] acpi PNP0A08:03: ECAM at [mem 0x84b000000000-0x84b001ffffff] for [bus 00-1f]
[    2.180280] PCI host bridge to bus 0003:00
[    2.180287] pci_bus 0003:00: root bus resource [mem 0x818000000000-0x81ffffffffff window]
[    2.180295] pci_bus 0003:00: root bus resource [bus 00-1f]
[    2.180329] pci_bus 0003:00: resource 4 [mem 0x818000000000-0x81ffffffffff window]
[    2.180436] ACPI: PCI Root Bridge [PEM0] (domain 0004 [bus 1f-57])
[    2.180448] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    2.180588] acpi PNP0A08:04: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR]
[    2.180717] acpi PNP0A08:04: _OSC: OS now controls [PCIeCapability]
[    2.180726] acpi PNP0A08:04: MCFG quirk: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57] with thunder_pem_ecam_ops
[    2.181277] acpi PNP0A08:04: ECAM area [mem 0x88001f000000-0x880057ffffff] reserved by CAVA02B:00
[    2.181874] acpi PNP0A08:04: ECAM at [mem 0x88001f000000-0x880057ffffff] for [bus 1f-57]
[    2.181909] Remapped I/O 0x0000883000000000 to [io  0x0000-0xffff window]
[    2.182046] PCI host bridge to bus 0004:1f
[    2.182053] pci_bus 0004:1f: root bus resource [io  0x0000-0xffff window]
[    2.182061] pci_bus 0004:1f: root bus resource [mem 0x881010000000-0x881fffffffff window] (bus address [0x10000000-0xfffffffff])
[    2.182069] pci_bus 0004:1f: root bus resource [mem 0x882000000000-0x882fffffffff pref window] (bus address [0x1000000000-0x1fffffffff])
[    2.182076] pci_bus 0004:1f: root bus resource [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.182083] pci_bus 0004:1f: root bus resource [bus 1f-57]
[    2.182108] pci 0004:1f:00.0: [177d:a100] type 01 class 0x060400
[    2.182123] pci 0004:1f:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
[    2.182133] pci 0004:1f:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    2.182148] pci 0004:1f:00.0: enabling Extended Tags
[    2.182172] pci 0004:1f:00.0: BAR 0: [mem 0x87e0c0f00000-0x87e0c0ffffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.182344] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.182439] pci 0004:20:00.0: [1a03:1150] type 01 class 0x060400
[    2.182488] pci 0004:20:00.0: Upstream bridge's Max Payload Size set to 128 (was 256, max 256)
[    2.182498] pci 0004:20:00.0: Max Payload Size set to 128 (was 128, max 128)
[    2.182507] pci 0004:20:00.0: enabling Extended Tags
[    2.182592] pci 0004:20:00.0: supports D1 D2
[    2.182597] pci 0004:20:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.182768] pci 0004:20:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.182834] pci_bus 0004:21: extended config space not accessible
[    2.182894] pci 0004:21:00.0: [1a03:2000] type 00 class 0x030000
[    2.182915] pci 0004:21:00.0: reg 0x10: [mem 0x881010000000-0x881010ffffff]
[    2.182928] pci 0004:21:00.0: reg 0x14: [mem 0x881011000000-0x88101101ffff]
[    2.182940] pci 0004:21:00.0: reg 0x18: [io  0x0000-0x007f]
[    2.182999] pci 0004:21:00.0: BAR 0: assigned to efifb
[    2.183049] pci 0004:21:00.0: supports D1 D2
[    2.183054] pci 0004:21:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.183249] pci 0004:1f:00.0: Primary bus is hard wired to 0
[    2.183267] pci 0004:1f:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.183275] pci 0004:1f:00.0: BAR 6: assigned [mem 0x881011800000-0x88101180ffff pref]
[    2.183283] pci 0004:1f:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.183292] pci 0004:20:00.0: BAR 14: assigned [mem 0x881010000000-0x8810117fffff]
[    2.183298] pci 0004:20:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    2.183306] pci 0004:21:00.0: BAR 0: assigned [mem 0x881010000000-0x881010ffffff]
[    2.183315] pci 0004:21:00.0: BAR 1: assigned [mem 0x881011000000-0x88101101ffff]
[    2.183324] pci 0004:21:00.0: BAR 2: assigned [io  0x1000-0x107f]
[    2.183333] pci 0004:20:00.0: PCI bridge to [bus 21]
[    2.183340] pci 0004:20:00.0:   bridge window [io  0x1000-0x1fff]
[    2.183346] pci 0004:20:00.0:   bridge window [mem 0x881010000000-0x8810117fffff]
[    2.183353] pci 0004:1f:00.0: PCI bridge to [bus 20-21]
[    2.183360] pci 0004:1f:00.0:   bridge window [io  0x1000-0x1fff]
[    2.183366] pci 0004:1f:00.0:   bridge window [mem 0x881010000000-0x8810117fffff]
[    2.183374] pci_bus 0004:1f: resource 4 [io  0x0000-0xffff window]
[    2.183380] pci_bus 0004:1f: resource 5 [mem 0x881010000000-0x881fffffffff window]
[    2.183386] pci_bus 0004:1f: resource 6 [mem 0x882000000000-0x882fffffffff pref window]
[    2.183392] pci_bus 0004:1f: resource 7 [mem 0x87e0c0000000-0x87e0c0ffffff window]
[    2.183399] pci_bus 0004:20: resource 0 [io  0x1000-0x1fff]
[    2.183405] pci_bus 0004:20: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.183411] pci_bus 0004:21: resource 0 [io  0x1000-0x1fff]
[    2.183417] pci_bus 0004:21: resource 1 [mem 0x881010000000-0x8810117fffff]
[    2.183529] ACPI: PCI Interrupt Link [LN0A] (IRQs *48)
[    2.183581] ACPI: PCI Interrupt Link [LN0B] (IRQs *49)
[    2.183627] ACPI: PCI Interrupt Link [LN0C] (IRQs *50)
[    2.183674] ACPI: PCI Interrupt Link [LN0D] (IRQs *51)
[    2.183964] ARMH0011:00: ttyAMA0 at MMIO 0x87e024000000 (irq = 21, base_baud = 0) is a SBSA
[    6.381459] printk: console [ttyAMA0] enabled
[    6.389031] ARMH0011:01: ttyAMA1 at MMIO 0x87e025000000 (irq = 22, base_baud = 0) is a SBSA
[    6.403390] iommu: Default domain type: Translated 
[    6.408406] pci 0004:21:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    6.416759] pci 0004:21:00.0: vgaarb: bridge control possible
[    6.422498] pci 0004:21:00.0: vgaarb: setting as boot device (VGA legacy resources not available)
[    6.431360] vgaarb: loaded
[    6.434782] SCSI subsystem initialized
[    6.438664] libata version 3.00 loaded.
[    6.438725] ACPI: bus type USB registered
[    6.442785] usbcore: registered new interface driver usbfs
[    6.448294] usbcore: registered new interface driver hub
[    6.453646] usbcore: registered new device driver usb
[    6.458787] pps_core: LinuxPPS API ver. 1 registered
[    6.463750] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    6.472880] PTP clock support registered
[    6.477015] EDAC MC: Ver: 3.0.0
[    6.480631] Registered efivars operations
[    6.486471] NetLabel: Initializing
[    6.489874] NetLabel:  domain hash size = 128
[    6.494221] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    6.499910] NetLabel:  unlabeled traffic allowed by default
[    6.506019] clocksource: Switched to clocksource arch_sys_counter
[    6.552176] VFS: Disk quotas dquot_6.6.0
[    6.556156] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    6.563484] AppArmor: AppArmor Filesystem Enabled
[    6.568273] pnp: PnP ACPI init
[    6.571957] system 00:00: [mem 0x848000000000-0x848001ffffff] could not be reserved
[    6.579633] system 00:00: Plug and Play ACPI device, IDs CAVa02c PNP0c02 (active)
[    6.579813] system 00:01: [mem 0x849000000000-0x849001ffffff] could not be reserved
[    6.587483] system 00:01: Plug and Play ACPI device, IDs CAVa02c PNP0c02 (active)
[    6.587663] system 00:02: [mem 0x84a000000000-0x84a001ffffff] could not be reserved
[    6.595333] system 00:02: Plug and Play ACPI device, IDs CAVa02c PNP0c02 (active)
[    6.595513] system 00:03: [mem 0x84b000000000-0x84b001ffffff] could not be reserved
[    6.603185] system 00:03: Plug and Play ACPI device, IDs CAVa02c PNP0c02 (active)
[    6.603373] system 00:04: [mem 0x87e0c0000000-0x87e0c0ffffff] could not be reserved
[    6.611034] system 00:04: [mem 0x88001f000000-0x880057ffffff] could not be reserved
[    6.618694] system 00:04: Plug and Play ACPI device, IDs CAVa02b PNP0c02 (active)
[    6.620919] pnp: PnP ACPI: found 5 devices
[    6.630876] NET: Registered protocol family 2
[    6.636036] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    6.645263] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    6.655124] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    6.662890] TCP: Hash tables configured (established 524288 bind 65536)
[    6.669909] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, linear)
[    6.678126] UDP hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.685679] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.693858] NET: Registered protocol family 1
[    6.698228] NET: Registered protocol family 44
[    6.702864] PCI: CLS 0 bytes, default 64
[    6.706894] Trying to unpack rootfs image as initramfs...
[    7.711420] Freeing initrd memory: 68180K
[    7.718497] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available
[    7.726530] kvm [1]: IPA Size Limit: 48 bits
[    7.732420] kvm [1]: GICv3: no GICV resource entry
[    7.737213] kvm [1]: disabling GICv2 emulation
[    7.741649] kvm [1]: GICv3 sysreg trapping enabled ([G0G1], reduced performance)
[    7.749050] kvm [1]: GIC system register CPU interface enabled
[    7.756017] kvm [1]: vgic interrupt IRQ9
[    7.761141] kvm [1]: Hyp mode initialized successfully
[    7.769051] Initialise system trusted keyrings
[    7.773521] Key type blacklist registered
[    7.777691] workingset: timestamp_bits=40 max_order=24 bucket_order=0
[    7.788417] zbud: loaded
[    7.792124] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    7.798498] fuse: init (API version 7.33)
[    7.802906] integrity: Platform Keyring initialized
[    7.830790] Key type asymmetric registered
[    7.834887] Asymmetric key parser 'x509' registered
[    7.839775] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    7.847300] io scheduler mq-deadline registered
[    7.855218] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    7.862764] efifb: probing for efifb
[    7.866372] efifb: No BGRT, not showing boot graphics
[    7.871416] efifb: framebuffer at 0x881010000000, using 3072k, total 3072k
[    7.878282] efifb: mode is 1024x768x32, linelength=4096, pages=1
[    7.884280] efifb: scrolling: redraw
[    7.887845] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    7.893537] fbcon: Deferring console take-over
[    7.897981] fb0: EFI VGA frame buffer device
[    7.902922] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[    7.911343] ACPI: button: Power Button [PWRB]
[    7.917165] ACPI GTDT: [Firmware Bug]: failed to get the Watchdog base address.
[    7.928506] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    7.941108] msm_serial: driver initialized
[    7.946004] arm-smmu arm-smmu.0.auto: probing hardware configuration...
[    7.952620] arm-smmu arm-smmu.0.auto: SMMUv2 with:
[    7.957404] arm-smmu arm-smmu.0.auto: 	stage 1 translation
[    7.962880] arm-smmu arm-smmu.0.auto: 	stage 2 translation
[    7.968357] arm-smmu arm-smmu.0.auto: 	nested translation
[    7.973747] arm-smmu arm-smmu.0.auto: 	non-coherent table walk
[    7.979571] arm-smmu arm-smmu.0.auto: 	(IDR0.CTTW overridden by FW configuration)
[    7.987044] arm-smmu arm-smmu.0.auto: 	stream matching with 128 register groups
[    7.994346] arm-smmu arm-smmu.0.auto: 	128 context banks (0 stage-2 only)
[    8.001129] arm-smmu arm-smmu.0.auto: 	enabling workaround for Cavium erratum 27704
[    8.008777] arm-smmu arm-smmu.0.auto: 	Supported page sizes: 0x62215000
[    8.015384] arm-smmu arm-smmu.0.auto: 	Stage-1: 48-bit VA -> 48-bit IPA
[    8.021991] arm-smmu arm-smmu.0.auto: 	Stage-2: 48-bit IPA -> 48-bit PA
[    8.028829] arm-smmu arm-smmu.1.auto: probing hardware configuration...
[    8.035443] arm-smmu arm-smmu.1.auto: SMMUv2 with:
[    8.040228] arm-smmu arm-smmu.1.auto: 	stage 1 translation
[    8.045704] arm-smmu arm-smmu.1.auto: 	stage 2 translation
[    8.051180] arm-smmu arm-smmu.1.auto: 	nested translation
[    8.056571] arm-smmu arm-smmu.1.auto: 	non-coherent table walk
[    8.062396] arm-smmu arm-smmu.1.auto: 	(IDR0.CTTW overridden by FW configuration)
[    8.069871] arm-smmu arm-smmu.1.auto: 	stream matching with 128 register groups
[    8.077181] arm-smmu arm-smmu.1.auto: 	128 context banks (0 stage-2 only)
[    8.083968] arm-smmu arm-smmu.1.auto: 	enabling workaround for Cavium erratum 27704
[    8.091616] arm-smmu arm-smmu.1.auto: 	Supported page sizes: 0x62215000
[    8.098222] arm-smmu arm-smmu.1.auto: 	Stage-1: 48-bit VA -> 48-bit IPA
[    8.104830] arm-smmu arm-smmu.1.auto: 	Stage-2: 48-bit IPA -> 48-bit PA
[    8.111616] arm-smmu arm-smmu.2.auto: probing hardware configuration...
[    8.118232] arm-smmu arm-smmu.2.auto: SMMUv2 with:
[    8.123015] arm-smmu arm-smmu.2.auto: 	stage 1 translation
[    8.128492] arm-smmu arm-smmu.2.auto: 	stage 2 translation
[    8.133969] arm-smmu arm-smmu.2.auto: 	nested translation
[    8.139361] arm-smmu arm-smmu.2.auto: 	non-coherent table walk
[    8.145186] arm-smmu arm-smmu.2.auto: 	(IDR0.CTTW overridden by FW configuration)
[    8.152660] arm-smmu arm-smmu.2.auto: 	stream matching with 128 register groups
[    8.159962] arm-smmu arm-smmu.2.auto: 	128 context banks (0 stage-2 only)
[    8.166748] arm-smmu arm-smmu.2.auto: 	enabling workaround for Cavium erratum 27704
[    8.174398] arm-smmu arm-smmu.2.auto: 	Supported page sizes: 0x62215000
[    8.181004] arm-smmu arm-smmu.2.auto: 	Stage-1: 48-bit VA -> 48-bit IPA
[    8.187610] arm-smmu arm-smmu.2.auto: 	Stage-2: 48-bit IPA -> 48-bit PA
[    8.194389] arm-smmu arm-smmu.3.auto: probing hardware configuration...
[    8.201006] arm-smmu arm-smmu.3.auto: SMMUv2 with:
[    8.205789] arm-smmu arm-smmu.3.auto: 	stage 1 translation
[    8.211268] arm-smmu arm-smmu.3.auto: 	stage 2 translation
[    8.216745] arm-smmu arm-smmu.3.auto: 	nested translation
[    8.222136] arm-smmu arm-smmu.3.auto: 	non-coherent table walk
[    8.227960] arm-smmu arm-smmu.3.auto: 	(IDR0.CTTW overridden by FW configuration)
[    8.235435] arm-smmu arm-smmu.3.auto: 	stream matching with 128 register groups
[    8.242737] arm-smmu arm-smmu.3.auto: 	128 context banks (0 stage-2 only)
[    8.249518] arm-smmu arm-smmu.3.auto: 	enabling workaround for Cavium erratum 27704
[    8.257166] arm-smmu arm-smmu.3.auto: 	Supported page sizes: 0x62215000
[    8.263772] arm-smmu arm-smmu.3.auto: 	Stage-1: 48-bit VA -> 48-bit IPA
[    8.270378] arm-smmu arm-smmu.3.auto: 	Stage-2: 48-bit IPA -> 48-bit PA
[    8.278228] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    8.300471] loop: module loaded
[    8.304862] libphy: Fixed MDIO Bus: probed
[    8.309004] tun: Universal TUN/TAP device driver, 1.6
[    8.314446] PPP generic driver version 2.4.2
[    8.318858] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    8.325395] ehci-pci: EHCI PCI platform driver
[    8.329877] ehci-orion: EHCI orion driver
[    8.333922] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    8.340103] ohci-pci: OHCI PCI platform driver
[    8.344581] uhci_hcd: USB Universal Host Controller Interface driver
[    8.351159] mousedev: PS/2 mouse device common for all mice
[    8.557146] rtc-efi rtc-efi.0: registered as rtc0
[    8.661912] rtc-efi rtc-efi.0: setting system clock to 2021-04-21T14:37:38 UTC (1619015858)
[    8.670456] i2c /dev entries driver
[    8.674560] device-mapper: uevent: version 1.0.3
[    8.679352] device-mapper: ioctl: 4.44.0-ioctl (2021-02-01) initialised: dm-devel@redhat.com
[    8.689096] ledtrig-cpu: registered to indicate activity on CPUs
[    8.696877] pstore: Registered efi as persistent store backend
[    8.703575] drop_monitor: Initializing network drop monitor service
[    8.710343] NET: Registered protocol family 10
[    8.737123] Segment Routing with IPv6
[    8.740837] NET: Registered protocol family 17
[    8.745394] Key type dns_resolver registered
[    8.749897] registered taskstats version 1
[    8.754046] Loading compiled-in X.509 certificates
[    8.760576] Loaded X.509 cert 'Build time autogenerated kernel key: b2705ab2481bcbdfae85dcbf8e7e0248227cf5e0'
[    8.772791] zswap: loaded using pool lzo/zbud
[    8.777549] Key type ._fscrypt registered
[    8.781562] Key type .fscrypt registered
[    8.785479] Key type fscrypt-provisioning registered
[    8.790614] pstore: Using crash dump compression: deflate
[    8.806123] Key type encrypted registered
[    8.810138] AppArmor: AppArmor sha1 policy hashing enabled
[    8.816085] ima: No TPM chip found, activating TPM-bypass!
[    8.821581] ima: Allocated hash algorithm: sha1
[    8.826128] ima: No architecture policies found
[    8.830690] evm: Initialising EVM extended attributes:
[    8.835822] evm: security.selinux
[    8.839129] evm: security.SMACK64
[    8.842434] evm: security.SMACK64EXEC
[    8.846087] evm: security.SMACK64TRANSMUTE
[    8.850173] evm: security.SMACK64MMAP
[    8.853823] evm: security.apparmor
[    8.857215] evm: security.ima
[    8.860173] evm: security.capability
[    8.863739] evm: HMAC attrs: 0x1
[    8.867907] pcieport 0000:00:01.0: Adding to iommu group 0
[    8.873929] pcieport 0000:00:14.0: Adding to iommu group 1
[    8.879884] pcieport 0002:00:02.0: Adding to iommu group 2
[    8.885864] pcieport 0004:1f:00.0: Adding to iommu group 3
[    8.891806] PCI Interrupt Link [LN0A] enabled at IRQ 48
[    8.897695] pcieport 0004:20:00.0: Adding to iommu group 4
[    8.909838] Freeing unused kernel memory: 7616K

  -dann

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 08/11] irqchip/gic: Configure SGIs as standard interrupts

[Marc Zyngier]

On Wed, 21 Apr 2021 15:52:52 +0100,
dann frazier <dann.frazier@canonical.com> wrote:
> 
> [ + Fu Wei ]

[...]

> >
> > Please feed this stacktrace to scripts/decode_stacktrace.sh so that I
> > can get an idea about what is going wrong. I bet something is playing
> > ungodly games with the one of the IPIs, and things go horribly wrong.
> 
> hey Marc,
>   Sure:
> 
> [    7.927289] Unable to handle kernel read from unreadable memory at virtual address 0000000000000028
> [    7.936326] Mem abort info:
> [    7.939108]   ESR = 0x96000004
> [    7.942151]   EC = 0x25: DABT (current EL), IL = 32 bits
> [    7.947451]   SET = 0, FnV = 0
> [    7.950494]   EA = 0, S1PTW = 0
> [    7.953624] Data abort info:
> [    7.956492]   ISV = 0, ISS = 0x00000004
> [    7.960316]   CM = 0, WnR = 0
> [    7.963273] [0000000000000028] user address but active_mm is swapper
> [    7.969616] Internal error: Oops: 96000004 [#1] SMP
> [    7.974483] Modules linked in:
> [    7.977531] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 5.12.0-rc8 #19
> [    7.983874] Hardware name: GIGABYTE R120-T33/MT30-GS1, BIOS F02 08/06/2019
> [    7.990737] pstate: 40400085 (nZcv daIf +PAN -UAO -TCO BTYPE=--)
> [    7.996732] pc : __ipi_send_mask (/home/ubuntu/linux/./include/linux/irqdomain.h:537 /home/ubuntu/linux/kernel/irq/ipi.c:283) 
> [    8.000910] lr : smp_cross_call (/home/ubuntu/linux/arch/arm64/kernel/smp.c:958) 
> [    8.004913] sp : ffff800012753c10
> [    8.008216] x29: ffff800012753c10 x28: ffff000100de5d00
> [    8.013521] x27: 000000000000000a x26: ffff80001225da20
> [    8.018825] x25: 0000000000000000 x24: ffff000ff62719b0
> [    8.024129] x23: ffff80001225d000 x22: ffff800012368108
> [    8.029433] x21: ffff800010f69a20 x20: 0000000000000000
> [    8.034737] x19: ffff000100143c60 x18: 0000000000000020
> [    8.040041] x17: 000000008e74252f x16: 00000000bf0ab2ad
> [    8.045345] x15: ffffffffffffffff x14: 0000000000000000
> [    8.050649] x13: 003d090000000000 x12: 00003d0900000000
> [    8.055953] x11: 0000000000000000 x10: 00003d0900000000
> [    8.061257] x9 : ffff800010027f14 x8 : 0000000000000000
> [    8.066561] x7 : 00000000ffffffff x6 : ffff000ff6148698
> [    8.071865] x5 : ffff80001159d040 x4 : ffff80001159d110
> [    8.077169] x3 : ffff800010f69a00 x2 : 0000000000000000
> [    8.082473] x1 : ffff800010f69a20 x0 : 0000000000000000
> [    8.087777] Call trace:
> [    8.090213] __ipi_send_mask (/home/ubuntu/linux/./include/linux/irqdomain.h:537 /home/ubuntu/linux/kernel/irq/ipi.c:283) 

Thanks for that. This resolves to:

	if (irq_domain_is_ipi_per_cpu(data->domain)) {

data->domain is NULL, and we probably are using freed memory...

> > Now, here's a hunch: in the fine TX1 tradition, the firmware is broken
> > and the GTDT table looks unusable. Amusingly, the crash happens right
> > after the SBSA watchdog fails to probe.
> 
> Yeah, I noticed that, but didn't highlight it as I didn't see it in
> the backtrace...
> 
> > And looking at the code that implements that driver, it looks dodgy as
> > hell, as it unmaps an interrupt it doesn't even know is valid. And it
> > does that right when the driver fails the way you experienced it. If,
> > by any chance, the interrupt field is 0 in the firmware table, this
> > results in SGI0 being unmapped. Given that this is the rescheduling
> > interrupt, fireworks happen.
> 
> ... and that explains why. I wouldn't have gotten there, but wish I'd
> thought to test w/ the watchdog compiled out :(

No worries. This IRQ series has uncovered a number of terrible driver
behaviours since I merged it, and these bugs are worth every penny.

> > Can you have a go with the patchlet below, and let me know if that
> > helps?
> 
> It does!

Awesome. I'll Cc you on the actual patch, feel free to respond with a
Tested-by: if you want.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 09/11] irqchip/gic-common: Don't enable SGIs by default

[Marc Zyngier]

The architecture code now enables the IPIs as required, so no
need to enable SGIs by default in the GIC code.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-common.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/irqchip/irq-gic-common.c b/drivers/irqchip/irq-gic-common.c
index 82520006195d..f47b41dfd023 100644
--- a/drivers/irqchip/irq-gic-common.c
+++ b/drivers/irqchip/irq-gic-common.c
@@ -152,9 +152,6 @@ void gic_cpu_config(void __iomem *base, int nr, void (*sync_access)(void))
 		writel_relaxed(GICD_INT_DEF_PRI_X4,
 					base + GIC_DIST_PRI + i * 4 / 4);
 
-	/* Ensure all SGI interrupts are now enabled */
-	writel_relaxed(GICD_INT_EN_SET_SGI, base + GIC_DIST_ENABLE_SET);
-
 	if (sync_access)
 		sync_access();
 }
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 10/11] irqchip/bcm2836: Configure mailbox interrupts as standard interrupts

[Marc Zyngier]

In order to switch the bcm2836 driver to privide standard interrupts
for IPIs, it first needs to stop lying about the way things work.

The mailbox interrupt is actually a multiplexer, with enough
bits to store 32 pending interrupts per CPU. So let's turn it
into a chained irqchip.

Once this is done, we can instanciate the corresponding IPIs,
and pass them to the architecture code.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-bcm2836.c | 151 ++++++++++++++++++++++++++++------
 1 file changed, 125 insertions(+), 26 deletions(-)

diff --git a/drivers/irqchip/irq-bcm2836.c b/drivers/irqchip/irq-bcm2836.c
index 2038693f074c..85df6ddad9be 100644
--- a/drivers/irqchip/irq-bcm2836.c
+++ b/drivers/irqchip/irq-bcm2836.c
@@ -10,6 +10,7 @@
 #include <linux/of_irq.h>
 #include <linux/irqchip.h>
 #include <linux/irqdomain.h>
+#include <linux/irqchip/chained_irq.h>
 #include <linux/irqchip/irq-bcm2836.h>
 
 #include <asm/exception.h>
@@ -89,12 +90,24 @@ static struct irq_chip bcm2836_arm_irqchip_gpu = {
 	.irq_unmask	= bcm2836_arm_irqchip_unmask_gpu_irq,
 };
 
+static void bcm2836_arm_irqchip_dummy_op(struct irq_data *d)
+{
+}
+
+static struct irq_chip bcm2836_arm_irqchip_dummy = {
+	.name		= "bcm2836-dummy",
+	.irq_eoi	= bcm2836_arm_irqchip_dummy_op,
+};
+
 static int bcm2836_map(struct irq_domain *d, unsigned int irq,
 		       irq_hw_number_t hw)
 {
 	struct irq_chip *chip;
 
 	switch (hw) {
+	case LOCAL_IRQ_MAILBOX0:
+		chip = &bcm2836_arm_irqchip_dummy;
+		break;
 	case LOCAL_IRQ_CNTPSIRQ:
 	case LOCAL_IRQ_CNTPNSIRQ:
 	case LOCAL_IRQ_CNTHPIRQ:
@@ -127,17 +140,7 @@ __exception_irq_entry bcm2836_arm_irqchip_handle_irq(struct pt_regs *regs)
 	u32 stat;
 
 	stat = readl_relaxed(intc.base + LOCAL_IRQ_PENDING0 + 4 * cpu);
-	if (stat & BIT(LOCAL_IRQ_MAILBOX0)) {
-#ifdef CONFIG_SMP
-		void __iomem *mailbox0 = (intc.base +
-					  LOCAL_MAILBOX0_CLR0 + 16 * cpu);
-		u32 mbox_val = readl(mailbox0);
-		u32 ipi = ffs(mbox_val) - 1;
-
-		writel(1 << ipi, mailbox0);
-		handle_IPI(ipi, regs);
-#endif
-	} else if (stat) {
+	if (stat) {
 		u32 hwirq = ffs(stat) - 1;
 
 		handle_domain_irq(intc.domain, hwirq, regs);
@@ -145,8 +148,35 @@ __exception_irq_entry bcm2836_arm_irqchip_handle_irq(struct pt_regs *regs)
 }
 
 #ifdef CONFIG_SMP
-static void bcm2836_arm_irqchip_send_ipi(const struct cpumask *mask,
-					 unsigned int ipi)
+static struct irq_domain *ipi_domain;
+
+static void bcm2836_arm_irqchip_handle_ipi(struct irq_desc *desc)
+{
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	int cpu = smp_processor_id();
+	u32 mbox_val;
+
+	chained_irq_enter(chip, desc);
+
+	mbox_val = readl_relaxed(intc.base + LOCAL_MAILBOX0_CLR0 + 16 * cpu);
+	if (mbox_val) {
+		int hwirq = ffs(mbox_val) - 1;
+		generic_handle_irq(irq_find_mapping(ipi_domain, hwirq));
+	}
+
+	chained_irq_exit(chip, desc);
+}
+
+static void bcm2836_arm_irqchip_ipi_eoi(struct irq_data *d)
+{
+	int cpu = smp_processor_id();
+
+	writel_relaxed(BIT(d->hwirq),
+		       intc.base + LOCAL_MAILBOX0_CLR0 + 16 * cpu);
+}
+
+static void bcm2836_arm_irqchip_ipi_send_mask(struct irq_data *d,
+					      const struct cpumask *mask)
 {
 	int cpu;
 	void __iomem *mailbox0_base = intc.base + LOCAL_MAILBOX0_SET0;
@@ -157,11 +187,45 @@ static void bcm2836_arm_irqchip_send_ipi(const struct cpumask *mask,
 	 */
 	smp_wmb();
 
-	for_each_cpu(cpu, mask)	{
-		writel(1 << ipi, mailbox0_base + 16 * cpu);
+	for_each_cpu(cpu, mask)
+		writel_relaxed(BIT(d->hwirq), mailbox0_base + 16 * cpu);
+}
+
+static struct irq_chip bcm2836_arm_irqchip_ipi = {
+	.name		= "IPI",
+	.irq_eoi	= bcm2836_arm_irqchip_ipi_eoi,
+	.ipi_send_mask	= bcm2836_arm_irqchip_ipi_send_mask,
+};
+
+static int bcm2836_arm_irqchip_ipi_alloc(struct irq_domain *d,
+					 unsigned int virq,
+					 unsigned int nr_irqs, void *args)
+{
+	int i;
+
+	for (i = 0; i < nr_irqs; i++) {
+		irq_set_percpu_devid(virq + i);
+		irq_domain_set_info(d, virq + i, i, &bcm2836_arm_irqchip_ipi,
+				    d->host_data,
+				    handle_percpu_devid_fasteoi_ipi,
+				    NULL, NULL);
 	}
+
+	return 0;
 }
 
+static void bcm2836_arm_irqchip_ipi_free(struct irq_domain *d,
+					 unsigned int virq,
+					 unsigned int nr_irqs)
+{
+	/* Not freeing IPIs */
+}
+
+static const struct irq_domain_ops ipi_domain_ops = {
+	.alloc	= bcm2836_arm_irqchip_ipi_alloc,
+	.free	= bcm2836_arm_irqchip_ipi_free,
+};
+
 static int bcm2836_cpu_starting(unsigned int cpu)
 {
 	bcm2836_arm_irqchip_unmask_per_cpu_irq(LOCAL_MAILBOX_INT_CONTROL0, 0,
@@ -175,25 +239,58 @@ static int bcm2836_cpu_dying(unsigned int cpu)
 					     cpu);
 	return 0;
 }
-#endif
 
-static const struct irq_domain_ops bcm2836_arm_irqchip_intc_ops = {
-	.xlate = irq_domain_xlate_onetwocell,
-	.map = bcm2836_map,
-};
+#define BITS_PER_MBOX	32
 
-static void
-bcm2836_arm_irqchip_smp_init(void)
+static void bcm2836_arm_irqchip_smp_init(void)
 {
-#ifdef CONFIG_SMP
+	struct irq_fwspec ipi_fwspec = {
+		.fwnode		= intc.domain->fwnode,
+		.param_count	= 1,
+		.param		= {
+			[0]	= LOCAL_IRQ_MAILBOX0,
+		},
+	};
+	int base_ipi, mux_irq;
+
+	mux_irq = irq_create_fwspec_mapping(&ipi_fwspec);
+	if (WARN_ON(mux_irq <= 0))
+		return;
+
+	ipi_domain = irq_domain_create_linear(intc.domain->fwnode,
+					      BITS_PER_MBOX, &ipi_domain_ops,
+					      NULL);
+	if (WARN_ON(!ipi_domain))
+		return;
+
+	ipi_domain->flags |= IRQ_DOMAIN_FLAG_IPI_SINGLE;
+	irq_domain_update_bus_token(ipi_domain, DOMAIN_BUS_IPI);
+
+	base_ipi = __irq_domain_alloc_irqs(ipi_domain, -1, BITS_PER_MBOX,
+					   NUMA_NO_NODE, NULL,
+					   false, NULL);
+
+	if (WARN_ON(!base_ipi))
+		return;
+
+	set_smp_ipi_range(base_ipi, BITS_PER_MBOX);
+
+	irq_set_chained_handler_and_data(mux_irq,
+					 bcm2836_arm_irqchip_handle_ipi, NULL);
+
 	/* Unmask IPIs to the boot CPU. */
 	cpuhp_setup_state(CPUHP_AP_IRQ_BCM2836_STARTING,
 			  "irqchip/bcm2836:starting", bcm2836_cpu_starting,
 			  bcm2836_cpu_dying);
-
-	set_smp_cross_call(bcm2836_arm_irqchip_send_ipi);
-#endif
 }
+#else
+#define bcm2836_arm_irqchip_smp_init()	do { } while(0)
+#endif
+
+static const struct irq_domain_ops bcm2836_arm_irqchip_intc_ops = {
+	.xlate = irq_domain_xlate_onetwocell,
+	.map = bcm2836_map,
+};
 
 /*
  * The LOCAL_IRQ_CNT* timer firings are based off of the external
@@ -232,6 +329,8 @@ static int __init bcm2836_arm_irqchip_l1_intc_of_init(struct device_node *node,
 	if (!intc.domain)
 		panic("%pOF: unable to create IRQ domain\n", node);
 
+	irq_domain_update_bus_token(intc.domain, DOMAIN_BUS_WIRED);
+
 	bcm2836_arm_irqchip_smp_init();
 
 	set_handle_irq(bcm2836_arm_irqchip_handle_irq);
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 11/11] arm64: Kill __smp_cross_call and co

[Marc Zyngier]

The old IPI registration interface is now unused on arm64, so let's
get rid of it.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/irq_work.h |  4 +---
 arch/arm64/include/asm/smp.h      |  7 -------
 arch/arm64/kernel/smp.c           | 34 +++++--------------------------
 3 files changed, 6 insertions(+), 39 deletions(-)

diff --git a/arch/arm64/include/asm/irq_work.h b/arch/arm64/include/asm/irq_work.h
index 8a1ef1907760..a1020285ea75 100644
--- a/arch/arm64/include/asm/irq_work.h
+++ b/arch/arm64/include/asm/irq_work.h
@@ -2,11 +2,9 @@
 #ifndef __ASM_IRQ_WORK_H
 #define __ASM_IRQ_WORK_H
 
-#include <asm/smp.h>
-
 static inline bool arch_irq_work_has_interrupt(void)
 {
-	return !!__smp_cross_call;
+	return true;
 }
 
 #endif /* __ASM_IRQ_WORK_H */
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 42f366ba88bf..bec6ef0eb642 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -75,13 +75,6 @@ extern void handle_IPI(int ipinr, struct pt_regs *regs);
  */
 extern void smp_init_cpus(void);
 
-/*
- * Provide a function to raise an IPI cross call on CPUs in callmap.
- */
-extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
-
-extern void (*__smp_cross_call)(const struct cpumask *, unsigned int);
-
 /*
  * Register IPI interrupts with the arch SMP code
  */
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 93ba0025e7b9..d29823ac515d 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -793,13 +793,6 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 	}
 }
 
-void (*__smp_cross_call)(const struct cpumask *, unsigned int);
-
-void __init set_smp_cross_call(void (*fn)(const struct cpumask *, unsigned int))
-{
-	__smp_cross_call = fn;
-}
-
 static const char *ipi_types[NR_IPI] __tracepoint_string = {
 #define S(x,s)	[x] = s
 	S(IPI_RESCHEDULE, "Rescheduling interrupts"),
@@ -811,11 +804,7 @@ static const char *ipi_types[NR_IPI] __tracepoint_string = {
 	S(IPI_WAKEUP, "CPU wake-up interrupts"),
 };
 
-static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
-{
-	trace_ipi_raise(target, ipi_types[ipinr]);
-	__smp_cross_call(target, ipinr);
-}
+static void smp_cross_call(const struct cpumask *target, unsigned int ipinr);
 
 void show_ipi_list(struct seq_file *p, int prec)
 {
@@ -862,8 +851,7 @@ void arch_send_wakeup_ipi_mask(const struct cpumask *mask)
 #ifdef CONFIG_IRQ_WORK
 void arch_irq_work_raise(void)
 {
-	if (__smp_cross_call)
-		smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
+	smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
 }
 #endif
 
@@ -970,27 +958,16 @@ static void do_handle_IPI(int ipinr)
 		trace_ipi_exit_rcuidle(ipi_types[ipinr]);
 }
 
-/* Legacy version, should go away once all irqchips have been converted */
-void handle_IPI(int ipinr, struct pt_regs *regs)
-{
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	irq_enter();
-	do_handle_IPI(ipinr);
-	irq_exit();
-
-	set_irq_regs(old_regs);
-}
-
 static irqreturn_t ipi_handler(int irq, void *data)
 {
 	do_handle_IPI(irq - ipi_irq_base);
 	return IRQ_HANDLED;
 }
 
-static void ipi_send(const struct cpumask *target, unsigned int ipi)
+static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
 {
-	__ipi_send_mask(ipi_desc[ipi], target);
+	trace_ipi_raise(target, ipi_types[ipinr]);
+	__ipi_send_mask(ipi_desc[ipinr], target);
 }
 
 static void ipi_setup(int cpu)
@@ -1032,7 +1009,6 @@ void __init set_smp_ipi_range(int ipi_base, int n)
 	}
 
 	ipi_irq_base = ipi_base;
-	__smp_cross_call = ipi_send;
 
 	/* Setup the boot CPU immediately */
 	ipi_setup(smp_processor_id());
-- 
2.26.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Florian Fainelli]

On 5/19/2020 9:17 AM, Marc Zyngier wrote:
> For as long as SMP ARM has existed, IPIs have been handled as
> something special. The arch code and the interrupt controller exchange
> a couple of hooks (one to generate an IPI, another to handle it).
> 
> Although this is perfectly manageable, it prevents the use of features
> that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
> also means that each interrupt controller driver has to follow an
> architecture-specific interface instead of just implementing the base
> irqchip functionnalities. The arch code also duplicates a number of
> things that the core irq code already does (such as calling
> set_irq_regs(), irq_enter()...).
> 
> This series tries to remedy this on arm/arm64 by offering a new
> registration interface where the irqchip gives the arch code a range
> of interrupts to use for IPIs. The arch code requests these as normal
> interrupts.
> 
> The bulk of the work is at the interrupt controller level, where all 3
> irqchips used on arm64 get converted.
> 
> Finally, the arm64 code drops the legacy registration interface. The
> same thing could be done on 32bit as well once the two remaining
> irqchips using that interface get converted.
> 
> There is probably more that could be done: statistics are still
> architecture-private code, for example, and no attempt is made to
> solve that (apart from hidding the IRQs from /proc/interrupt).
> 
> This has been tested on a bunch of 32 and 64bit guests.

Does this patch series change your position on this patch series

https://lore.kernel.org/linux-arm-kernel/20191023000547.7831-3-f.fainelli@gmail.com/T/

or is this still a no-no?

> 
> Marc Zyngier (11):
>   genirq: Add fasteoi IPI flow
>   genirq: Allow interrupts to be excluded from /proc/interrupts
>   arm64: Allow IPIs to be handled as normal interrupts
>   ARM: Allow IPIs to be handled as normal interrupts
>   irqchip/gic-v3: Describe the SGI range
>   irqchip/gic-v3: Configure SGIs as standard interrupts
>   irqchip/gic: Refactor SMP configuration
>   irqchip/gic: Configure SGIs as standard interrupts
>   irqchip/gic-common: Don't enable SGIs by default
>   irqchip/bcm2836: Configure mailbox interrupts as standard interrupts
>   arm64: Kill __smp_cross_call and co
> 
>  arch/arm/Kconfig                  |   1 +
>  arch/arm/include/asm/smp.h        |   5 +
>  arch/arm/kernel/smp.c             |  97 +++++++++++---
>  arch/arm64/Kconfig                |   1 +
>  arch/arm64/include/asm/irq_work.h |   4 +-
>  arch/arm64/include/asm/smp.h      |   6 +-
>  arch/arm64/kernel/smp.c           |  98 +++++++++++----
>  drivers/irqchip/irq-bcm2836.c     | 151 ++++++++++++++++++----
>  drivers/irqchip/irq-gic-common.c  |   3 -
>  drivers/irqchip/irq-gic-v3.c      | 109 ++++++++++------
>  drivers/irqchip/irq-gic.c         | 203 ++++++++++++++++++------------
>  include/linux/irq.h               |   4 +-
>  kernel/irq/chip.c                 |  26 ++++
>  kernel/irq/debugfs.c              |   1 +
>  kernel/irq/proc.c                 |   2 +-
>  kernel/irq/settings.h             |   7 ++
>  16 files changed, 515 insertions(+), 203 deletions(-)
> 

-- 
Florian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Florian Fainelli]

On 5/19/2020 10:50 AM, Florian Fainelli wrote:
> 
> 
> On 5/19/2020 9:17 AM, Marc Zyngier wrote:
>> For as long as SMP ARM has existed, IPIs have been handled as
>> something special. The arch code and the interrupt controller exchange
>> a couple of hooks (one to generate an IPI, another to handle it).
>>
>> Although this is perfectly manageable, it prevents the use of features
>> that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
>> also means that each interrupt controller driver has to follow an
>> architecture-specific interface instead of just implementing the base
>> irqchip functionnalities. The arch code also duplicates a number of
>> things that the core irq code already does (such as calling
>> set_irq_regs(), irq_enter()...).
>>
>> This series tries to remedy this on arm/arm64 by offering a new
>> registration interface where the irqchip gives the arch code a range
>> of interrupts to use for IPIs. The arch code requests these as normal
>> interrupts.
>>
>> The bulk of the work is at the interrupt controller level, where all 3
>> irqchips used on arm64 get converted.
>>
>> Finally, the arm64 code drops the legacy registration interface. The
>> same thing could be done on 32bit as well once the two remaining
>> irqchips using that interface get converted.
>>
>> There is probably more that could be done: statistics are still
>> architecture-private code, for example, and no attempt is made to
>> solve that (apart from hidding the IRQs from /proc/interrupt).
>>
>> This has been tested on a bunch of 32 and 64bit guests.
> 
> Does this patch series change your position on this patch series
> 
> https://lore.kernel.org/linux-arm-kernel/20191023000547.7831-3-f.fainelli@gmail.com/T/
> 
> or is this still a no-no?

Our firmware specifies SGI interrupts with the first interrupt cell
specifier set to 2, so changing GIC_IRQ_TYPE_SGI to 2 allows me to use a
nearly unmodified firmware with your changes, sweet! I know this is not
supposed to be used that way, but the temptation was too big.

FWIW, on ARM64:

Tested-by: Florian Fainelli <f.fainelli@gmail.com>
-- 
Florian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Marc Zyngier]

Hi Florian,

On Tue, 19 May 2020 10:50:46 -0700
Florian Fainelli <f.fainelli@gmail.com> wrote:

> On 5/19/2020 9:17 AM, Marc Zyngier wrote:
> > For as long as SMP ARM has existed, IPIs have been handled as
> > something special. The arch code and the interrupt controller exchange
> > a couple of hooks (one to generate an IPI, another to handle it).
> > 
> > Although this is perfectly manageable, it prevents the use of features
> > that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
> > also means that each interrupt controller driver has to follow an
> > architecture-specific interface instead of just implementing the base
> > irqchip functionnalities. The arch code also duplicates a number of
> > things that the core irq code already does (such as calling
> > set_irq_regs(), irq_enter()...).
> > 
> > This series tries to remedy this on arm/arm64 by offering a new
> > registration interface where the irqchip gives the arch code a range
> > of interrupts to use for IPIs. The arch code requests these as normal
> > interrupts.
> > 
> > The bulk of the work is at the interrupt controller level, where all 3
> > irqchips used on arm64 get converted.
> > 
> > Finally, the arm64 code drops the legacy registration interface. The
> > same thing could be done on 32bit as well once the two remaining
> > irqchips using that interface get converted.
> > 
> > There is probably more that could be done: statistics are still
> > architecture-private code, for example, and no attempt is made to
> > solve that (apart from hidding the IRQs from /proc/interrupt).
> > 
> > This has been tested on a bunch of 32 and 64bit guests.  
> 
> Does this patch series change your position on this patch series
> 
> https://lore.kernel.org/linux-arm-kernel/20191023000547.7831-3-f.fainelli@gmail.com/T/
> 
> or is this still a no-no?

I don't think this series changes anything. There is no easy way to
reserve SGIs in a way that would work for all combination of OS and FW,
and the prospect of sending SGIs between S and NS has already been
dubious (yes, the GIC architecture allows it, but it has been written
by people who have never designed any large piece of SW).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Florian Fainelli]

On 6/12/2020 2:49 AM, Marc Zyngier wrote:
> Hi Florian,
> 
> On Tue, 19 May 2020 10:50:46 -0700
> Florian Fainelli <f.fainelli@gmail.com> wrote:
> 
>> On 5/19/2020 9:17 AM, Marc Zyngier wrote:
>>> For as long as SMP ARM has existed, IPIs have been handled as
>>> something special. The arch code and the interrupt controller exchange
>>> a couple of hooks (one to generate an IPI, another to handle it).
>>>
>>> Although this is perfectly manageable, it prevents the use of features
>>> that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
>>> also means that each interrupt controller driver has to follow an
>>> architecture-specific interface instead of just implementing the base
>>> irqchip functionnalities. The arch code also duplicates a number of
>>> things that the core irq code already does (such as calling
>>> set_irq_regs(), irq_enter()...).
>>>
>>> This series tries to remedy this on arm/arm64 by offering a new
>>> registration interface where the irqchip gives the arch code a range
>>> of interrupts to use for IPIs. The arch code requests these as normal
>>> interrupts.
>>>
>>> The bulk of the work is at the interrupt controller level, where all 3
>>> irqchips used on arm64 get converted.
>>>
>>> Finally, the arm64 code drops the legacy registration interface. The
>>> same thing could be done on 32bit as well once the two remaining
>>> irqchips using that interface get converted.
>>>
>>> There is probably more that could be done: statistics are still
>>> architecture-private code, for example, and no attempt is made to
>>> solve that (apart from hidding the IRQs from /proc/interrupt).
>>>
>>> This has been tested on a bunch of 32 and 64bit guests.  
>>
>> Does this patch series change your position on this patch series
>>
>> https://lore.kernel.org/linux-arm-kernel/20191023000547.7831-3-f.fainelli@gmail.com/T/
>>
>> or is this still a no-no?
> 
> I don't think this series changes anything. There is no easy way to
> reserve SGIs in a way that would work for all combination of OS and FW,
> and the prospect of sending SGIs between S and NS has already been
> dubious (yes, the GIC architecture allows it, but it has been written
> by people who have never designed any large piece of SW).

That is fair enough, we have transitioned since then to using SPIs and
that appears to work nicely for what we want to do without requiring
your patch series.

In premise it is still possible for someone to specify 0x561 as the
first interrupt cell specifier  in the Device Tree in order to specify a
SGI interrupt and this will happily be parsed as a valid interrupt.

It would most likely fail some time later while trying to set the
interrupt type though. I do not think you can do better than this, as
there is no way for you to know the caller of gic_irq_domain_translate()
and reject them.
-- 
Florian

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts

[Valentin Schneider]

Hi Marc,

On 19/05/20 17:17, Marc Zyngier wrote:
>
> This has been tested on a bunch of 32 and 64bit guests.
>

I gave this a brief spin on Juno and on HiKey960, nothing to report there.

FWIW I'd like to stare a bit at the rest; I already have a rookie question
on the accounting, but the rest will be for after I get some sleep, and
maybe even a mug of coffee :-)

> Marc Zyngier (11):
>   genirq: Add fasteoi IPI flow
>   genirq: Allow interrupts to be excluded from /proc/interrupts
>   arm64: Allow IPIs to be handled as normal interrupts
>   ARM: Allow IPIs to be handled as normal interrupts
>   irqchip/gic-v3: Describe the SGI range
>   irqchip/gic-v3: Configure SGIs as standard interrupts
>   irqchip/gic: Refactor SMP configuration
>   irqchip/gic: Configure SGIs as standard interrupts
>   irqchip/gic-common: Don't enable SGIs by default
>   irqchip/bcm2836: Configure mailbox interrupts as standard interrupts
>   arm64: Kill __smp_cross_call and co
>
>  arch/arm/Kconfig                  |   1 +
>  arch/arm/include/asm/smp.h        |   5 +
>  arch/arm/kernel/smp.c             |  97 +++++++++++---
>  arch/arm64/Kconfig                |   1 +
>  arch/arm64/include/asm/irq_work.h |   4 +-
>  arch/arm64/include/asm/smp.h      |   6 +-
>  arch/arm64/kernel/smp.c           |  98 +++++++++++----
>  drivers/irqchip/irq-bcm2836.c     | 151 ++++++++++++++++++----
>  drivers/irqchip/irq-gic-common.c  |   3 -
>  drivers/irqchip/irq-gic-v3.c      | 109 ++++++++++------
>  drivers/irqchip/irq-gic.c         | 203 ++++++++++++++++++------------
>  include/linux/irq.h               |   4 +-
>  kernel/irq/chip.c                 |  26 ++++
>  kernel/irq/debugfs.c              |   1 +
>  kernel/irq/proc.c                 |   2 +-
>  kernel/irq/settings.h             |   7 ++
>  16 files changed, 515 insertions(+), 203 deletions(-)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel