[Questions]: how to extend nr_irqs for arch arm64

[Questions]: how to extend nr_irqs for arch arm64

[Zhou Qiao(周侨)]

Hi Thomas, Jason

I've a question about nr_irqs(with CONFIG_SPARSE_IRQ enabled) on arch arm64, and want to ask for your suggestions. 

The nr_irqs is set to NR_IRQS by default and NR_IRQS is small(64). Arch/arm platforms will re-set nr_irqs to a larger value. I didn't find similar way for arm64. The way I can think of is to redefine NR_IRQS. But according comments in include/asm-generic/irq.h, NR_IRQS should not be a large value. so what's the preferred way to extend nr_irqs, for arm64 specifically? Thanks in advance.

Best Regards
Qiao

Re: [Questions]: how to extend nr_irqs for arch arm64

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1180 bytes --]

On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:

> The nr_irqs is set to NR_IRQS by default and NR_IRQS is small(64). Arch/arm
> platforms will re-set nr_irqs to a larger value. I didn't find similar way
> for arm64. The way I can think of is to redefine NR_IRQS. But according
> comments in include/asm-generic/irq.h, NR_IRQS should not be a large
> value. so what's the preferred way to extend nr_irqs, for arm64
> specifically? Thanks in advance.

With SPARSE_IRQ enabled the core will modify nr_irqs with the following
mechanims:

 - Early boot: arch_probe_nr_irqs(). If the return value of that function is >
   	       NR_IRQS then it will be set to that.

	       This is used for interrupts which have to be allocated and
	       assigned in early boot, but that's really only for legacy
	       purposes. So arm64 should not use that at all.

 - Runtime:    The core dynamically increases nr_irqs up to IRQ_BITMAP_BITS,
    	       which is NR_IRQS + 8192.

The runtime increase is happening, when interrupts are allocated. And you
won't notice unless you try to extend nr_irqs above NR_IRQS + 8192. So there
is nothing to do for arm64, it should just work.

Thanks,

	tglx

RE: [Questions]: how to extend nr_irqs for arch arm64

[Zhou Qiao(周侨)]

> -----Original Message-----
> From: Thomas Gleixner [mailto:tglx@linutronix.de]
> Sent: Monday, January 11, 2016 4:04 PM
> To: Zhou Qiao(周侨)
> Cc: jason@lakedaemon.net; linux-kernel@vger.kernel.org
> Subject: Re: [Questions]: how to extend nr_irqs for arch arm64
> 
> On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> 
> > The nr_irqs is set to NR_IRQS by default and NR_IRQS is small(64).
> > Arch/arm platforms will re-set nr_irqs to a larger value. I didn't
> > find similar way for arm64. The way I can think of is to redefine
> > NR_IRQS. But according comments in include/asm-generic/irq.h, NR_IRQS
> > should not be a large value. so what's the preferred way to extend
> > nr_irqs, for arm64 specifically? Thanks in advance.
> 
> With SPARSE_IRQ enabled the core will modify nr_irqs with the following
> mechanims:
> 
>  - Early boot: arch_probe_nr_irqs(). If the return value of that
> function is >
>    	       NR_IRQS then it will be set to that.
> 
> 	       This is used for interrupts which have to be allocated and
> 	       assigned in early boot, but that's really only for legacy
> 	       purposes. So arm64 should not use that at all.
Agree. Thanks for the details.
> 
>  - Runtime:    The core dynamically increases nr_irqs up to
> IRQ_BITMAP_BITS,
>     	       which is NR_IRQS + 8192.
> 
> The runtime increase is happening, when interrupts are allocated. And
> you won't notice unless you try to extend nr_irqs above NR_IRQS + 8192.
> So there is nothing to do for arm64, it should just work.
I met an issue here. The nr_irqs is extended when the start_irq + cnt >
nr_irqs. So when nr_irqs is not extended(still 64), a device with a large
hr_irq(70) will get a wrong virq. since it will be mod by nr_irqs before
actually allocating the descriptor in irq_domain_alloc_descs. I don't know
what I missed.

static int irq_domain_alloc_descs(int virq, unsigned int cnt,
                  irq_hw_number_t hwirq, int node)
{
    unsigned int hint;

    if (virq >= 0) {
        virq = irq_alloc_descs(virq, virq, cnt, node);
    } else {
        hint = hwirq % nr_irqs;
        if (hint == 0)
            hint++;
        virq = irq_alloc_descs_from(hint, cnt, node);
        if (virq <= 0 && hint > 1)
            virq = irq_alloc_descs_from(1, cnt, node);
    }

    return virq;
}
> 
> Thanks,
> 
> 	tglx

RE: [Questions]: how to extend nr_irqs for arch arm64

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 446 bytes --]

On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> I met an issue here. The nr_irqs is extended when the start_irq + cnt >
> nr_irqs. So when nr_irqs is not extended(still 64), a device with a large
> hr_irq(70) will get a wrong virq. since it will be mod by nr_irqs before

No, it won't get a wrong virq. It will get a virq and it does not matter at
all whether the hardware irq number is 10, 70, 500 or 5000. What's your
problem?

Thanks,

	tglx

RE: [Questions]: how to extend nr_irqs for arch arm64

[Zhou Qiao(周侨)]

> -----Original Message-----
> From: Thomas Gleixner [mailto:tglx@linutronix.de]
> Sent: Monday, January 11, 2016 4:51 PM
> To: Zhou Qiao(周侨)
> Cc: jason@lakedaemon.net; linux-kernel@vger.kernel.org
> Subject: RE: [Questions]: how to extend nr_irqs for arch arm64
> 
> On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> > I met an issue here. The nr_irqs is extended when the start_irq + cnt
> > > nr_irqs. So when nr_irqs is not extended(still 64), a device with a
> > large
> > hr_irq(70) will get a wrong virq. since it will be mod by nr_irqs
> > before
> 
> No, it won't get a wrong virq. It will get a virq and it does not
> matter at all whether the hardware irq number is 10, 70, 500 or 5000.
> What's your problem?
Let's assume the hw_irq is 70, and after the mod operation, it will call irq_alloc_descs_from with parameter from = 6, instead of 70. So the newly allocated descriptor is mapped to hw_irq = 6, not 70. Later when driver registers irq with irq = 70, it will fail since there is no irq descriptor for irq = 70.
> 
> Thanks,
> 
> 	tglx
> 
> 

RE: [Questions]: how to extend nr_irqs for arch arm64

[Thomas Gleixner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1110 bytes --]

On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> > From: Thomas Gleixner [mailto:tglx@linutronix.de]
> > On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> > > I met an issue here. The nr_irqs is extended when the start_irq + cnt
> > > > nr_irqs. So when nr_irqs is not extended(still 64), a device with a
> > > large
> > > hr_irq(70) will get a wrong virq. since it will be mod by nr_irqs
> > > before
> > 
> > No, it won't get a wrong virq. It will get a virq and it does not
> > matter at all whether the hardware irq number is 10, 70, 500 or 5000.
> > What's your problem?
> 
> Let's assume the hw_irq is 70, and after the mod operation, it will call
> irq_alloc_descs_from with parameter from = 6, instead of 70. So the newly
> allocated descriptor is mapped to hw_irq = 6, not 70. Later when driver
> registers irq with irq = 70, it will fail since there is no irq descriptor
> for irq = 70.

Rightfully so. The driver cannot assume irq=70. That's what the device tree
mapping is for. It will tell the driver that it has to request virq 6. There
is no 1:1 mapping between hardware irqs and virqs.

Thanks,

	tglx

RE: [Questions]: how to extend nr_irqs for arch arm64

[Zhou Qiao(周侨)]

> -----Original Message-----
> From: Thomas Gleixner [mailto:tglx@linutronix.de]
> Sent: Monday, January 11, 2016 5:18 PM
> To: Zhou Qiao(周侨)
> Cc: jason@lakedaemon.net; linux-kernel@vger.kernel.org
> Subject: RE: [Questions]: how to extend nr_irqs for arch arm64
> 
> On Mon, 11 Jan 2016, Zhou Qiao(周侨) wrote:
> > > From: Thomas Gleixner [mailto:tglx@linutronix.de] On Mon, 11 Jan
> > > 2016, Zhou Qiao(周侨) wrote:
> > > > I met an issue here. The nr_irqs is extended when the start_irq +
> > > > cnt
> > > > > nr_irqs. So when nr_irqs is not extended(still 64), a device
> > > > > with a
> > > > large
> > > > hr_irq(70) will get a wrong virq. since it will be mod by nr_irqs
> > > > before
> > >
> > > No, it won't get a wrong virq. It will get a virq and it does not
> > > matter at all whether the hardware irq number is 10, 70, 500 or
> 5000.
> > > What's your problem?
> >
> > Let's assume the hw_irq is 70, and after the mod operation, it will
> > call irq_alloc_descs_from with parameter from = 6, instead of 70. So
> > the newly allocated descriptor is mapped to hw_irq = 6, not 70. Later
> > when driver registers irq with irq = 70, it will fail since there is
> > no irq descriptor for irq = 70.
> 
> Rightfully so. The driver cannot assume irq=70. That's what the device
> tree mapping is for. It will tell the driver that it has to request
> virq 6. There is no 1:1 mapping between hardware irqs and virqs.
You're right. Actually the virqs is just a next_zero_area picked runtime and it doesn't matter whether the hw_irq is mod or not(or mod by whichever). Thanks a lot for the help.
> 
> Thanks,
> 
> 	tglx