nr_irqs_gsi

nr_irqs_gsi

[Joe Jin]

Hi,

IHAC whose server have 12 NICs, but when bring up found the 12 NIC not be
enabled, I found below from xm-dmesg
(XEN) physdev.c:122: dom0: map invalid irq 55

The corresponding codes:

115     /* Verify or get irq. */
116     switch ( map->type )
117     {
118     case MAP_PIRQ_TYPE_GSI:
119         if ( map->index < 0 || map->index >= nr_irqs_gsi )
120         {
121             dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
122                     d->domain_id, map->index);
123             ret = -EINVAL;
124             goto free_domain;
125         }
126 

Log all info to xm-dmesg I got:
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
(XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) ERST table is invalid
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) IRQ limits: 55 GSI, 3033 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)

nr_irqs_gsi = 55, 
So the condition of #119 should be (map->index > nr_irqs_gsi)?
for GSI irq 55 should be available as well?

Thanks,
Joe

Re: nr_irqs_gsi

[Andrew Cooper]

On 11/07/12 04:31, Joe Jin wrote:
> Hi,
>
> IHAC whose server have 12 NICs, but when bring up found the 12 NIC not be
> enabled, I found below from xm-dmesg
> (XEN) physdev.c:122: dom0: map invalid irq 55
>
> The corresponding codes:
>
> 115     /* Verify or get irq. */
> 116     switch ( map->type )
> 117     {
> 118     case MAP_PIRQ_TYPE_GSI:
> 119         if ( map->index < 0 || map->index >= nr_irqs_gsi )
> 120         {
> 121             dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
> 122                     d->domain_id, map->index);
> 123             ret = -EINVAL;
> 124             goto free_domain;
> 125         }
> 126 
>
> Log all info to xm-dmesg I got:
> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> (XEN) PCI: MCFG area at e0000000 reserved in E820
> (XEN) ERST table is invalid
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) IRQ limits: 55 GSI, 3033 MSI/MSI-X
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>
> nr_irqs_gsi = 55, 
> So the condition of #119 should be (map->index > nr_irqs_gsi)?
> for GSI irq 55 should be available as well?
>
> Thanks,
> Joe

No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
off-by-one error by changing the condition.

The more interesting question is why you are attempting to map more GSIs
than you actually have.

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: nr_irqs_gsi

[Konrad Rzeszutek Wilk]

On Wed, Jul 11, 2012 at 10:42:10AM +0100, Andrew Cooper wrote:
> 
> On 11/07/12 04:31, Joe Jin wrote:
> > Hi,
> >
> > IHAC whose server have 12 NICs, but when bring up found the 12 NIC not be
> > enabled, I found below from xm-dmesg
> > (XEN) physdev.c:122: dom0: map invalid irq 55
> >
> > The corresponding codes:
> >
> > 115     /* Verify or get irq. */
> > 116     switch ( map->type )
> > 117     {
> > 118     case MAP_PIRQ_TYPE_GSI:
> > 119         if ( map->index < 0 || map->index >= nr_irqs_gsi )
> > 120         {
> > 121             dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
> > 122                     d->domain_id, map->index);
> > 123             ret = -EINVAL;
> > 124             goto free_domain;
> > 125         }
> > 126 
> >
> > Log all info to xm-dmesg I got:
> > (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> > (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> > (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> > (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > (XEN) ACPI: IRQ0 used by override.
> > (XEN) ACPI: IRQ2 used by override.
> > (XEN) ACPI: IRQ9 used by override.
> > (XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
> > (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
> > (XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> > (XEN) PCI: MCFG area at e0000000 reserved in E820
> > (XEN) ERST table is invalid
> > (XEN) Using ACPI (MADT) for SMP configuration information
> > (XEN) IRQ limits: 55 GSI, 3033 MSI/MSI-X
> > (XEN) Using scheduler: SMP Credit Scheduler (credit)
> >
> > nr_irqs_gsi = 55, 
> > So the condition of #119 should be (map->index > nr_irqs_gsi)?
> > for GSI irq 55 should be available as well?
> >
> > Thanks,
> > Joe
> 
> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
> off-by-one error by changing the condition.
> 
> The more interesting question is why you are attempting to map more GSIs
> than you actually have.

Could it be the ACPI DSDT _PRT being confused. Joe, what does dom0
print out for:

xen: registering gsi ...

for the devices? (maybe all to see if there is a trend?)
arch/x86/pci/xen.c to print out the 
> 
> ~Andrew
> 
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> -- 
> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
> T: +44 (0)1223 225 900, http://www.citrix.com
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: nr_irqs_gsi

[Joe Jin]

On 07/11/12 17:42, Andrew Cooper wrote:
> 
> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
> off-by-one error by changing the condition.

Per xen log, I think it should be 0-55? for it start from 0 yet, please confirm!

>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55

> 
> The more interesting question is why you are attempting to map more GSIs
> than you actually have.
> 

The request came from Dom0 kernel(upstream kernel 3.0.x) for physical device. 

Thanks,
Joe

Re: nr_irqs_gsi

[Joe Jin]

[-- Attachment #1: Type: text/plain, Size: 4718 bytes --]

On 07/11/12 22:00, Konrad Rzeszutek Wilk wrote:
> On Wed, Jul 11, 2012 at 10:42:10AM +0100, Andrew Cooper wrote:
>>
>> On 11/07/12 04:31, Joe Jin wrote:
>>> Hi,
>>>
>>> IHAC whose server have 12 NICs, but when bring up found the 12 NIC not be
>>> enabled, I found below from xm-dmesg
>>> (XEN) physdev.c:122: dom0: map invalid irq 55
>>>
>>> The corresponding codes:
>>>
>>> 115     /* Verify or get irq. */
>>> 116     switch ( map->type )
>>> 117     {
>>> 118     case MAP_PIRQ_TYPE_GSI:
>>> 119         if ( map->index < 0 || map->index >= nr_irqs_gsi )
>>> 120         {
>>> 121             dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
>>> 122                     d->domain_id, map->index);
>>> 123             ret = -EINVAL;
>>> 124             goto free_domain;
>>> 125         }
>>> 126 
>>>
>>> Log all info to xm-dmesg I got:
>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
>>> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>>> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>>> (XEN) ACPI: IRQ0 used by override.
>>> (XEN) ACPI: IRQ2 used by override.
>>> (XEN) ACPI: IRQ9 used by override.
>>> (XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
>>> (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
>>> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
>>> (XEN) PCI: MCFG area at e0000000 reserved in E820
>>> (XEN) ERST table is invalid
>>> (XEN) Using ACPI (MADT) for SMP configuration information
>>> (XEN) IRQ limits: 55 GSI, 3033 MSI/MSI-X
>>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>>>
>>> nr_irqs_gsi = 55, 
>>> So the condition of #119 should be (map->index > nr_irqs_gsi)?
>>> for GSI irq 55 should be available as well?
>>>
>>> Thanks,
>>> Joe
>>
>> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
>> off-by-one error by changing the condition.
>>
>> The more interesting question is why you are attempting to map more GSIs
>> than you actually have.
> 
> Could it be the ACPI DSDT _PRT being confused. Joe, what does dom0
> print out for:
> 
> xen: registering gsi ...
> 
> for the devices? (maybe all to see if there is a trend?)
> arch/x86/pci/xen.c to print out the 

Hi Konrad,

Below came from dom0:

ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255
ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a301 base: 0xfed00000
SMP: Allowing 32 CPUs, 16 hotplug CPUs
nr_irqs_gsi: 304

<--snip-->
NR_IRQS:262400 nr_irqs:5424 16
xen: sci override: global_irq=9 trigger=0 polarity=0
xen: registering gsi 9 triggering 0 polarity 0
xen: --> pirq=9 -> irq=9 (gsi=9)
xen: acpi sci 9
xen: --> pirq=1 -> irq=1 (gsi=1)
xen: --> pirq=2 -> irq=2 (gsi=2)
xen: --> pirq=3 -> irq=3 (gsi=3)
xen: --> pirq=4 -> irq=4 (gsi=4)
xen: --> pirq=5 -> irq=5 (gsi=5)
xen: --> pirq=6 -> irq=6 (gsi=6)
xen: --> pirq=7 -> irq=7 (gsi=7)
xen: --> pirq=8 -> irq=8 (gsi=8)
xen_map_pirq_gsi: returning irq 9 for gsi 9
xen: --> pirq=9 -> irq=9 (gsi=9)
xen: --> pirq=10 -> irq=10 (gsi=10)
xen: --> pirq=11 -> irq=11 (gsi=11)
xen: --> pirq=12 -> irq=12 (gsi=12)
xen: --> pirq=13 -> irq=13 (gsi=13)
xen: --> pirq=14 -> irq=14 (gsi=14)
xen: --> pirq=15 -> irq=15 (gsi=15)

...

also, please find full dmesg output from attachment.

Thanks,
Joe

>>
>> ~Andrew
>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>
>> -- 
>> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
>> T: +44 (0)1223 225 900, http://www.citrix.com
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 
> 


-- 
Oracle <http://www.oracle.com>
Joe Jin | Software Development Senior Manager | +8610.6106.5624
ORACLE | Linux and Virtualization
No. 24 Zhongguancun Software Park, Haidian District | 100193 Beijing 



[-- Attachment #2: dmesg.gz --]
[-- Type: application/x-gzip, Size: 33797 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: nr_irqs_gsi

[Konrad Rzeszutek Wilk]

On Thu, Jul 12, 2012 at 10:17:13AM +0800, Joe Jin wrote:
> On 07/11/12 22:00, Konrad Rzeszutek Wilk wrote:
> > On Wed, Jul 11, 2012 at 10:42:10AM +0100, Andrew Cooper wrote:
> >>
> >> On 11/07/12 04:31, Joe Jin wrote:
> >>> Hi,
> >>>
> >>> IHAC whose server have 12 NICs, but when bring up found the 12 NIC not be
> >>> enabled, I found below from xm-dmesg
> >>> (XEN) physdev.c:122: dom0: map invalid irq 55
> >>>
> >>> The corresponding codes:
> >>>
> >>> 115     /* Verify or get irq. */
> >>> 116     switch ( map->type )
> >>> 117     {
> >>> 118     case MAP_PIRQ_TYPE_GSI:
> >>> 119         if ( map->index < 0 || map->index >= nr_irqs_gsi )
> >>> 120         {
> >>> 121             dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
> >>> 122                     d->domain_id, map->index);
> >>> 123             ret = -EINVAL;
> >>> 124             goto free_domain;
> >>> 125         }
> >>> 126 
> >>>
> >>> Log all info to xm-dmesg I got:
> >>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> >>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> >>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> >>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> >>> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> >>> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> >>> (XEN) ACPI: IRQ0 used by override.
> >>> (XEN) ACPI: IRQ2 used by override.
> >>> (XEN) ACPI: IRQ9 used by override.
> >>> (XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
> >>> (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
> >>> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> >>> (XEN) PCI: MCFG area at e0000000 reserved in E820
> >>> (XEN) ERST table is invalid
> >>> (XEN) Using ACPI (MADT) for SMP configuration information
> >>> (XEN) IRQ limits: 55 GSI, 3033 MSI/MSI-X
> >>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> >>>
> >>> nr_irqs_gsi = 55, 
> >>> So the condition of #119 should be (map->index > nr_irqs_gsi)?
> >>> for GSI irq 55 should be available as well?
> >>>
> >>> Thanks,
> >>> Joe
> >>
> >> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
> >> off-by-one error by changing the condition.
> >>
> >> The more interesting question is why you are attempting to map more GSIs
> >> than you actually have.
> > 
> > Could it be the ACPI DSDT _PRT being confused. Joe, what does dom0
> > print out for:
> > 
> > xen: registering gsi ...
> > 
> > for the devices? (maybe all to see if there is a trend?)
> > arch/x86/pci/xen.c to print out the 
> 
> Hi Konrad,
> 
> Below came from dom0:

Please rerun with 'debug loglevel=8' on the Linux command line. Otherwise
the information is not present.

Re: nr_irqs_gsi

[Jan Beulich]

>>> On 12.07.12 at 04:17, Joe Jin <joe.jin@oracle.com> wrote:
>>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
>
> Below came from dom0:
> 
> ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255

Now this and ...

> ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287

... this is clearly bogus, contradicting what Xen itself prints (see
above).

Jan

Re: nr_irqs_gsi

[Jan Beulich]

>>> On 12.07.12 at 01:56, Joe Jin <joe.jin@oracle.com> wrote:
> On 07/11/12 17:42, Andrew Cooper wrote:
>> 
>> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
>> off-by-one error by changing the condition.
> 
> Per xen log, I think it should be 0-55? for it start from 0 yet, please 
> confirm!

Indeed it should. And I think it's this line

    nr_irqs_gsi = max(nr_irqs_gsi, highest_gsi());

in io_apic.c:init_ioapic_mappings(), which fails to add 1 to the
result of highest_gsi() (regression from c/s 20076:2b8b6ee95c93).

>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55

Would you retry with the patch below?

Jan

--- a/xen/arch/x86/io_apic.c
+++ b/xen/arch/x86/io_apic.c
@@ -2531,7 +2531,9 @@ void __init init_ioapic_mappings(void)
         }
     }
 
-    nr_irqs_gsi = max(nr_irqs_gsi, highest_gsi());
+    i = highest_gsi();
+    if ( i >= nr_irqs_gsi )
+        nr_irqs_gsi = i + 1;
 
     if ( max_gsi_irqs == 0 )
         max_gsi_irqs = nr_irqs ? nr_irqs / 8 : PAGE_SIZE;

Re: nr_irqs_gsi

[Konrad Rzeszutek Wilk]

On Mon, Jul 23, 2012 at 02:30:37PM +0100, Jan Beulich wrote:
> >>> On 12.07.12 at 04:17, Joe Jin <joe.jin@oracle.com> wrote:
> >>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> >>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> >>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> >>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> >
> > Below came from dom0:
> > 
> > ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> > ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> > IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255
> 
> Now this and ...
> 
> > ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> > IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287
> 
> ... this is clearly bogus, contradicting what Xen itself prints (see
> above).

Yeah, that got fixed up in v3.5 (merge git f08b9c2f8af0d61faa1170aeae4fbca1eff6a504)
specifically:

      x86/apic: Fix UP boot crash
      xen/apic: implement io apic read with hypercall
      xen/x86: Implement x86_apic_ops
      x86/apic: Replace io_apic_ops with x86_io_apic_ops.
      x86/ioapic: Add io_apic_ops driver layer to allow interception

and the kernel Joe is using (UEK2) should have those back-ported.. Ah, he is
using the older version I think.

Re: nr_irqs_gsi

[Joe Jin]

On 07/23/12 21:42, Jan Beulich wrote:
>>>> On 12.07.12 at 01:56, Joe Jin <joe.jin@oracle.com> wrote:
>> On 07/11/12 17:42, Andrew Cooper wrote:
>>>
>>> No.  There are 55 GSIs, indexed 0 thru 54.  You would be introducing an
>>> off-by-one error by changing the condition.
>>
>> Per xen log, I think it should be 0-55? for it start from 0 yet, please 
>> confirm!
> 
> Indeed it should. And I think it's this line
> 
>     nr_irqs_gsi = max(nr_irqs_gsi, highest_gsi());
> 
> in io_apic.c:init_ioapic_mappings(), which fails to add 1 to the
> result of highest_gsi() (regression from c/s 20076:2b8b6ee95c93).
> 
>>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> 
> Would you retry with the patch below?
> 
> Jan
> 
> --- a/xen/arch/x86/io_apic.c
> +++ b/xen/arch/x86/io_apic.c
> @@ -2531,7 +2531,9 @@ void __init init_ioapic_mappings(void)
>          }
>      }
>  
> -    nr_irqs_gsi = max(nr_irqs_gsi, highest_gsi());
> +    i = highest_gsi();
> +    if ( i >= nr_irqs_gsi )
> +        nr_irqs_gsi = i + 1;
>  
>      if ( max_gsi_irqs == 0 )
>          max_gsi_irqs = nr_irqs ? nr_irqs / 8 : PAGE_SIZE;
> 
> 
> 
> 
> 

Hi Jan,

I created similar patch for my customer, now waiting their response.

Thanks,
Joe

Re: nr_irqs_gsi

[Konrad Rzeszutek Wilk]

On Tue, Jul 24, 2012 at 10:03:17AM +0800, Joe Jin wrote:
> On 07/23/12 23:34, Konrad Rzeszutek Wilk wrote:
> > On Mon, Jul 23, 2012 at 02:30:37PM +0100, Jan Beulich wrote:
> >>>>> On 12.07.12 at 04:17, Joe Jin <joe.jin@oracle.com> wrote:
> >>>>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> >>>>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> >>>>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> >>>>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
> >>>
> >>> Below came from dom0:
> >>>
> >>> ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> >>> ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> >>> IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255
> >>
> >> Now this and ...
> >>
> >>> ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
> >>> IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287
> >>
> >> ... this is clearly bogus, contradicting what Xen itself prints (see
> >> above).
> > 
> > Yeah, that got fixed up in v3.5 (merge git f08b9c2f8af0d61faa1170aeae4fbca1eff6a504)
> > specifically:
> > 
> >       x86/apic: Fix UP boot crash
> >       xen/apic: implement io apic read with hypercall
> >       xen/x86: Implement x86_apic_ops
> >       x86/apic: Replace io_apic_ops with x86_io_apic_ops.
> >       x86/ioapic: Add io_apic_ops driver layer to allow interception
> > 
> 
> Konrad,
> 
> Would you please point out which commit is make sense for this issue? so far I still
> suspend it's a hypervisor issue rather than dom0 kernel.

I concur. The patches above are just for reporting the GSI properly in the initial domain.

Re: nr_irqs_gsi

[Joe Jin]

On 07/23/12 23:34, Konrad Rzeszutek Wilk wrote:
> On Mon, Jul 23, 2012 at 02:30:37PM +0100, Jan Beulich wrote:
>>>>> On 12.07.12 at 04:17, Joe Jin <joe.jin@oracle.com> wrote:
>>>>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>>>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>>>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>>>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
>>>
>>> Below came from dom0:
>>>
>>> ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
>>> ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>> IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255
>>
>> Now this and ...
>>
>>> ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>> IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287
>>
>> ... this is clearly bogus, contradicting what Xen itself prints (see
>> above).
> 
> Yeah, that got fixed up in v3.5 (merge git f08b9c2f8af0d61faa1170aeae4fbca1eff6a504)
> specifically:
> 
>       x86/apic: Fix UP boot crash
>       xen/apic: implement io apic read with hypercall
>       xen/x86: Implement x86_apic_ops
>       x86/apic: Replace io_apic_ops with x86_io_apic_ops.
>       x86/ioapic: Add io_apic_ops driver layer to allow interception
> 

Konrad,

Would you please point out which commit is make sense for this issue? so far I still
suspend it's a hypervisor issue rather than dom0 kernel.

Thanks in advance,
Joe

Re: nr_irqs_gsi

[Joe Jin]

On 07/24/12 10:02, Konrad Rzeszutek Wilk wrote:
> On Tue, Jul 24, 2012 at 10:03:17AM +0800, Joe Jin wrote:
>> On 07/23/12 23:34, Konrad Rzeszutek Wilk wrote:
>>> On Mon, Jul 23, 2012 at 02:30:37PM +0100, Jan Beulich wrote:
>>>>>>> On 12.07.12 at 04:17, Joe Jin <joe.jin@oracle.com> wrote:
>>>>>>>> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>>>>>>> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>>>>>>>> (XEN) ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>>>>>>> (XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
>>>>>
>>>>> Below came from dom0:
>>>>>
>>>>> ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
>>>>> ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
>>>>> IOAPIC[0]: apic_id 0, version 255, address 0xfec00000, GSI 0-255
>>>>
>>>> Now this and ...
>>>>
>>>>> ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
>>>>> IOAPIC[1]: apic_id 1, version 255, address 0xfec80000, GSI 32-287
>>>>
>>>> ... this is clearly bogus, contradicting what Xen itself prints (see
>>>> above).
>>>
>>> Yeah, that got fixed up in v3.5 (merge git f08b9c2f8af0d61faa1170aeae4fbca1eff6a504)
>>> specifically:
>>>
>>>       x86/apic: Fix UP boot crash
>>>       xen/apic: implement io apic read with hypercall
>>>       xen/x86: Implement x86_apic_ops
>>>       x86/apic: Replace io_apic_ops with x86_io_apic_ops.
>>>       x86/ioapic: Add io_apic_ops driver layer to allow interception
>>>
>>
>> Konrad,
>>
>> Would you please point out which commit is make sense for this issue? so far I still
>> suspend it's a hypervisor issue rather than dom0 kernel.
> 
> I concur. The patches above are just for reporting the GSI properly in the initial domain.
Thanks for the input, I'll waiting the customer's response, for I did not reproduced it, once
confirm it works, we can include Jan's patch.

Thanks,
Joe