Re: [xen-unstable test] 104131: regressions - FAIL

From: Andrew Cooper <andrew.cooper3@citrix.com>
To: "Xuquan (Quan Xu)" <xuquan8@huawei.com>
Cc: Kevin Tian <kevin.tian@intel.com>, Chao Gao <chao.gao@intel.com>,
	osstest service owner <osstest-admin@xenproject.org>,
	Jan Beulich <JBeulich@suse.com>,
	"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: [xen-unstable test] 104131: regressions - FAIL
Date: Thu, 12 Jan 2017 12:15:46 +0000	[thread overview]
Message-ID: <56880ae2-198c-a8e5-1d2c-a440193c6e65@citrix.com> (raw)
In-Reply-To: <E0A769A898ADB6449596C41F51EF62C6ADE819@SZXEMI506-MBX.china.huawei.com>

On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>> On 12/01/2017 06:46, osstest service owner wrote:
>>> flight 104131 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking, including tests which
>>> could not be run:
>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>> REGR. vs. 104119
>>
>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>> intr.c:321
>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
>> Not tainted ]----
>> Jan 12 01:25:37.141577 (XEN) CPU:    14
>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>> vmx_intr_assist+0x35e/0x51d
>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
>> hypervisor (d15v0)
>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>> ffff830079e1e000   rcx: 0000000000000030
>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>> 0000000000000030   rdi: ffff830079e1e000
>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
>> r8:  ffff82c00022f000
>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>> ffff830176386560   r11: 000001955ee79bd0
>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>> 0000000000003002   r14: 0000000000000030
>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>> 0000000080050033   cr4: 00000000003526e0
>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>> 0000000002487034
>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
>> ss: 0000   cs: e008
>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>> (vmx_intr_assist+0x35e/0x51d):
>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
> >from rsp=ffff83047de2fea8:
>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>> ffff83047de2ffff ffff83023fec2000
>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>> ffff830079e1e000 ffff830079e1e000
>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
>> ffff830233117000 ffff83023fec2000
>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>> 0000000000000004 00000000000000c2
>> Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
>> ffff8800e8d28000 ffffffff81add0a0
>> Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
>> ffff880142400008 0000000000000004
>> Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
>> 00000000000003f8 00000000000003f8
>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>> ffffffff813899a4 000000bf0000beef
>> Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
>> 000000000000beef 1cec835356e5beef
>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
>> 8914458d3174beef 2444c7100000000e
>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
>> 00000000003526e0
>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>> vmx_intr_assist+0x35e/0x51d
>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>> vmx_asm_vmexit_handler+0x41/0x120
>> Jan 12 01:25:37.325598 (XEN)
>> Jan 12 01:25:37.325624 (XEN)
>> Jan 12 01:25:37.325647 (XEN)
>> ****************************************
>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
>> ****************************************
>> Jan 12 01:25:37.341603 (XEN)
>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>> RESET_REG.
>>
>> This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
>> not a deterministic issue, as it appears to have survived a week of testing
>> already, but there is clearly something still problematic with the code.
>>
>
> Andrew,
> If you have, could you give more information?

No further information sorry.  This was found by the automated test system.

Full logs are available from
http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i386-xl-qemuu-debianhvm-amd64/
but I doubt any of them will help in diagnosing the issue any further.

> Such as the value of intack.vector / pt_vector..
> I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..
>
> Or there is a corner case that intack.vector is __not__ the highest priority vector..
>
> Kevin / Jan,  any thoughts?

It happened during domain shutdown.  It might be an edge case
interaction with qemu-raised interrupts via hypercall?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel