Re: cpuidle and un-eoid interrupts at the local apic

From: Andrew Cooper <andrew.cooper3@citrix.com>
To: "Thimo E." <abc@digithi.de>
Cc: "Zhang, Yang Z" <yang.z.zhang@intel.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	Keir Fraser <keir@xen.org>, Jan Beulich <JBeulich@suse.com>
Subject: Re: cpuidle and un-eoid interrupts at the local apic
Date: Sun, 8 Sep 2013 10:53:31 +0100	[thread overview]
Message-ID: <522C491B.6000004@citrix.com> (raw)
In-Reply-To: <522BB8BE.8060800@digithi.de>

On 08/09/2013 00:37, Thimo E. wrote:
> Hello Andrew,
>
> ok, thanks. This is what I assumed.
>
> The output of "xl debug-keys iMQ" is empty.

Sorry - I should have been more clear.  `xl debug-keys` dumps its
information into the xen dmesg buffer, so `xl dmesg` will capture the
results.

~Andrew

>
> [root@localhost ~]#  dmesg |grep arcmsr
> [    8.159321] arcmsr 0000:01:00.0: PCI INT A -> GSI 16 (level, low)
> -> IRQ 16
> [    8.159413] arcmsr 0000:01:00.0: setting latency timer to 64
> [    8.170316] arcmsr 0000:01:00.0: get owner: 7ff0
> [    8.170414] arcmsr 0000:01:00.0: irq 1276 (276) for MSI/MSI-X
> [    8.170421] IRQ 1276/arcmsr: IRQF_DISABLED is not guaranteed on
> shared IRQs
> [    8.170654] arcmsr0: msi enabled
>
> [root@localhost /]# cat /proc/irq/1276/spurious
> count 61007
> unhandled 8
> last_unhandled 36736990 ms
>
> arcmsr is the driver of the Areca Storage Raid Controller. Used it
> already before with Xenserver 6.0.2 for years, no problems.
>
> THe messages "IRQF_DISABLED is not guaranteed...." and "8 unhandled
> interrupts" look interesting. I am not a kernel hacker but what I
> interpret from
> http://lxr.free-electrons.com/source/kernel/irq/manage.c?v=2.6.32:
>
> 1025         if ((irqflags & (IRQF_SHARED|IRQF_DISABLED)) ==
> 1026 (IRQF_SHARED|IRQF_DISABLED)) {
> 1027                 pr_warning(
> 1028                   "IRQ %d/%s: IRQF_DISABLED is not guaranteed on
> shared IRQs\n",
> 1029                         irq, devname);
> ...
> 738                  * Force MSI interrupts to run with interrupts
> 739                  * disabled. The multi vector cards can cause stack
> 740                  * overflows due to nested interrupts when enough of
> 741                  * them are directed to a core and fire at the same
> 742                  * time.
> 743                  */
> 744                 if (desc->msi_desc)
> 745                         new->flags |= IRQF_DISABLED;
>
> --> "IRQF_DISABLED is not guaranteed on shared IRQs" warning is only
> printed when irqflags IRQF_SHARED and IRQF_DISABLED are set
> --> Is that what we see in the kernel oops the stack overflow the
> comment in lines 738-742 is talking about ?!
> --> IRQF_SHARED is set, so MSI interrupt 1276 is shared ?! I thought
> that it is not possible that MSI interrupts are shared. Attached
> you'll see my /proc/interrupts
>
> So what I do now is disabling MSI for the arcmsr driver. Could this be
> the source of the problem ?! But why is 1276 shared ?!
>
> Best regards
>   Thimo
>
> Am 07.09.2013 19:02, schrieb Andrew Cooper:
>>
>> irq 29 is just an internal Xen number for accounting all interrupts.  It
>> doesn't mean anything specific regarding hardware etc.  The vector and
>> affinity would expect to change as dom0s vcpus are moved around by the
>> scheduler.
>>
>> domain-list=0 means that this interrupt is targeted at dom0 (It is a
>> list because certain interrupts have to be shared my more than 1
>> domain).  Helpfully, the keyhandler truncates the pirq field, so 276 is
>> unlikely to be correct.  As it is a dom0 MSI, I am guessing it actually
>> matches up with interrupt 1276 in /proc/interrupts, if there is one.
>>
>> Can you provide the results of `xl debug-keys iMQ`, and attach
>> /proc/interrupts to this email (just in case the setup has changed after
>> playing with your BIOS)
>>
>> ~Andrew
>>
>