[BUG] Xen panic with VT-d PI

[BUG] Xen panic with VT-d PI

[Hao, Xudong]

[-- Attachment #1: Type: text/plain, Size: 3740 bytes --]

Xen 4.7 RC3 enabled VT-d PI(iommu=on,intpost), and created HVM guest with VF NIC assigned repeatedly, it will cause Xen panic(maybe 2 times ,5 times ,10 times).
It's not a RC3 regression, at least this Xen commit 1bd52e1f got failure as well.

HW: Intel Broadwell server, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz. 
Xen: Xen 4.7.0 RC3
Dom0: Linux 4.4.4
VF: Intel 82599 NIC

...
(XEN) irq.c:275: Dom8 PCI link 3 changed 5 -> 0
[ 1319.177226] ixgbe 0000:05:00.0 eth30: VF Reset msg received from vf 1
[ 1321.949952] ixgbe 0000:05:00.0 eth30: 1 Spoofed packets detected
(XEN) ----[ Xen-4.7-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    3
(XEN) RIP:    e008:[<ffff82d0801f2910>] vmx.c#pi_wakeup_interrupt+0xcf/0x15b
(XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
(XEN) rax: ffff830078a71760   rbx: 01001001000fff60   rcx: 0000000000000003
(XEN) rdx: 0100100100100100   rsi: 0000000000000046   rdi: ffff830078a715c0
(XEN) rbp: ffff83043c477d18   rsp: ffff83043c477ce8   r8:  0000000000000003
(XEN) r9:  ffff830425ae2710   r10: ffff830425ae28e0   r11: 000001354d22b2fb
(XEN) r12: ffff83043c47dde0   r13: ffff83043c47ddf0   r14: 0100100100100100
(XEN) r15: 0200200200200200   cr0: 000000008005003b   cr4: 00000000003526e0
(XEN) cr3: 0000000424276000   cr2: 0000003abfaaca80
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d0801f2910> (vmx.c#pi_wakeup_interrupt+0xcf/0x15b):
(XEN)  00 5a 48 8b 97 a0 01 00 <48> 3b 42 08 74 04 0f 0b eb fe 48 8b 8f a8 01 00
(XEN) Xen stack trace from rsp=ffff83043c477ce8:
(XEN)    ffff83043c477d08 00000000000000f5 ffff83043c477db8 00000000000000f5
(XEN)    ffff83043c47f788 00000000ffffffff ffff83043c477da8 ffff82d080171c7a
(XEN)    0000000000000000 0000000000000000 deadbeefdeadf00d ffff8300782fe000
(XEN)    0000000000000003 ffff830422e70000 0000000000000000 800000003c477d90
(XEN)    ffff82d0801949fa 8c7adb3d00000002 ffff83043c477e18 00000135394f6d83
(XEN)    ffff83043c47f6d0 ffff82d080807280 ffff83043c47f788 00000000ffffffff
(XEN)    00007cfbc3b88227 ffff82d08023feb2 00000000ffffffff ffff83043c47f788
(XEN)    ffff82d080807280 ffff83043c47f6d0 ffff83043c477ee0 00000135394f6d83
(XEN)    000001354d22b2fb ffff830425ae28e0 ffff830425ae2710 0000000000000001
(XEN)    0000000000000008 20c49ba5e353f7cf ffff83043c470000 00000135394f6d83
(XEN)    ffff83043c47f700 000000f500000000 ffff82d0801ba6d2 000000000000e008
(XEN)    0000000000000286 ffff83043c477e60 0000000000000000 0000000800000020
(XEN)    0000013534e2391b ffffffffffffffff ffff830078a71000 ffffffff81c17dc0
(XEN)    ffff83043c477f08 0000000000000000 0000000000000000 ffff82d080832980
(XEN)    00001eba00006a07 ffffffffffffffff ffff83043c477f18 ffff82d080807280
(XEN)    ffff83043c477f18 ffff830078a71000 ffff83043c423000 ffff83043c477f10
(XEN)    ffff82d0801660fd ffffffff81df8000 ffff8300782fe000 0000000000000003
(XEN)    ffff830422e70000 ffff83043c477d78 ffffffff81df8000 ffffffffffffffff
(XEN)    0000000000000000 ffffffff81c17dc0 ffffffff81a03ea8 0000000000000000
(XEN)    0000000000000000 00000002d788742b 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d0801f2910>] vmx.c#pi_wakeup_interrupt+0xcf/0x15b
(XEN)    [<ffff82d080171c7a>] do_IRQ+0x9d/0x65e
(XEN)    [<ffff82d08023feb2>] common_interrupt+0x62/0x70
(XEN)    [<ffff82d0801ba6d2>] mwait-idle.c#mwait_idle+0x2b3/0x30a
(XEN)    [<ffff82d0801660fd>] domain.c#idle_loop+0x6c/0x89
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 3:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=0000]
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...


Best Regards,
Xudong


[-- Attachment #2: config.vmxvtd_ass_02 --]
[-- Type: application/octet-stream, Size: 1620 bytes --]

# =====================================================================
# Example HVM guest configuration
# =====================================================================
#
# This is a fairly minimal example of what is required for an
# HVM guest. For a more complete guide see xl.cfg(5)

# This configures an HVM rather than PV guest
builder = "hvm"

# Guest name
name = "vVTD_ASS_02_1457580189"

# 128-bit UUID for the domain as a hexadecimal number.
# Use "uuidgen" to generate one if required.
# The default behavior is to generate a new UUID each time the guest is started.
#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

# Enable Microsoft Hyper-V compatibile paravirtualisation /
# enlightenment interfaces. Turning this on can improve Windows guest
# performance and is therefore recommended
#viridian = 1

# Initial memory allocation (MB)
memory = 512

# Maximum memory (MB)
# If this is greater than `memory' then the slack will start ballooned
# (this assumes guest kernel support for ballooning)
#maxmem = 512

# Number of VCPUS
vcpus=2

# Network devices
# A list of '#vifspec' entries as described in
# docs/misc/xl-network-configuration.markdown
#vif = [ 'type=ioemu, mac=3e:97:20:8d:45:21, bridge=xenbr0' ]

# Disk Devices
# A list of `diskspec' entries as described in
# docs/misc/xl-disk-configuration.txt
disk = [ '/share/xvs/var/img.vmxVTD_ASS_02,qcow2,hda,rw' ]
device_model_override = '/usr/local/lib/xen/bin/qemu-system-i386'
device_model_version = 'qemu-xen'

# Guest VGA console configuration, either SDL or VNC
sdl=0
vnc=1
stdvga=1
hap=1
acpi=1
gfx_passthru=0
pci = [ '05:10.2' ]
hpet=1

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Wu, Feng]

The patch fixing this issue has already been sent to upstream.

" [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu blocking list "

v2 will be sent out soon.

Thanks,
Feng

> -----Original Message-----
> From: Hao, Xudong
> Sent: Thursday, May 26, 2016 9:45 AM
> To: Xen-devel <xen-devel@lists.xenproject.org>
> Cc: Wu, Feng <feng.wu@intel.com>; Wei Liu <wei.liu2@citrix.com>
> Subject: [BUG] Xen panic with VT-d PI
> 
> Xen 4.7 RC3 enabled VT-d PI(iommu=on,intpost), and created HVM guest with VF
> NIC assigned repeatedly, it will cause Xen panic(maybe 2 times ,5 times ,10
> times).
> It's not a RC3 regression, at least this Xen commit 1bd52e1f got failure as well.
> 
> HW: Intel Broadwell server, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz.
> Xen: Xen 4.7.0 RC3
> Dom0: Linux 4.4.4
> VF: Intel 82599 NIC
> 
> ...
> (XEN) irq.c:275: Dom8 PCI link 3 changed 5 -> 0
> [ 1319.177226] ixgbe 0000:05:00.0 eth30: VF Reset msg received from vf 1
> [ 1321.949952] ixgbe 0000:05:00.0 eth30: 1 Spoofed packets detected
> (XEN) ----[ Xen-4.7-unstable  x86_64  debug=y  Tainted:    C ]----
> (XEN) CPU:    3
> (XEN) RIP:    e008:[<ffff82d0801f2910>] vmx.c#pi_wakeup_interrupt+0xcf/0x15b
> (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
> (XEN) rax: ffff830078a71760   rbx: 01001001000fff60   rcx: 0000000000000003
> (XEN) rdx: 0100100100100100   rsi: 0000000000000046   rdi: ffff830078a715c0
> (XEN) rbp: ffff83043c477d18   rsp: ffff83043c477ce8   r8:  0000000000000003
> (XEN) r9:  ffff830425ae2710   r10: ffff830425ae28e0   r11: 000001354d22b2fb
> (XEN) r12: ffff83043c47dde0   r13: ffff83043c47ddf0   r14: 0100100100100100
> (XEN) r15: 0200200200200200   cr0: 000000008005003b   cr4:
> 00000000003526e0
> (XEN) cr3: 0000000424276000   cr2: 0000003abfaaca80
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen code around <ffff82d0801f2910>
> (vmx.c#pi_wakeup_interrupt+0xcf/0x15b):
> (XEN)  00 5a 48 8b 97 a0 01 00 <48> 3b 42 08 74 04 0f 0b eb fe 48 8b 8f a8 01 00
> (XEN) Xen stack trace from rsp=ffff83043c477ce8:
> (XEN)    ffff83043c477d08 00000000000000f5 ffff83043c477db8
> 00000000000000f5
> (XEN)    ffff83043c47f788 00000000ffffffff ffff83043c477da8 ffff82d080171c7a
> (XEN)    0000000000000000 0000000000000000 deadbeefdeadf00d
> ffff8300782fe000
> (XEN)    0000000000000003 ffff830422e70000 0000000000000000
> 800000003c477d90
> (XEN)    ffff82d0801949fa 8c7adb3d00000002 ffff83043c477e18
> 00000135394f6d83
> (XEN)    ffff83043c47f6d0 ffff82d080807280 ffff83043c47f788 00000000ffffffff
> (XEN)    00007cfbc3b88227 ffff82d08023feb2 00000000ffffffff ffff83043c47f788
> (XEN)    ffff82d080807280 ffff83043c47f6d0 ffff83043c477ee0
> 00000135394f6d83
> (XEN)    000001354d22b2fb ffff830425ae28e0 ffff830425ae2710
> 0000000000000001
> (XEN)    0000000000000008 20c49ba5e353f7cf ffff83043c470000
> 00000135394f6d83
> (XEN)    ffff83043c47f700 000000f500000000 ffff82d0801ba6d2
> 000000000000e008
> (XEN)    0000000000000286 ffff83043c477e60 0000000000000000
> 0000000800000020
> (XEN)    0000013534e2391b ffffffffffffffff ffff830078a71000 ffffffff81c17dc0
> (XEN)    ffff83043c477f08 0000000000000000 0000000000000000
> ffff82d080832980
> (XEN)    00001eba00006a07 ffffffffffffffff ffff83043c477f18 ffff82d080807280
> (XEN)    ffff83043c477f18 ffff830078a71000 ffff83043c423000 ffff83043c477f10
> (XEN)    ffff82d0801660fd ffffffff81df8000 ffff8300782fe000 0000000000000003
> (XEN)    ffff830422e70000 ffff83043c477d78 ffffffff81df8000 ffffffffffffffff
> (XEN)    0000000000000000 ffffffff81c17dc0 ffffffff81a03ea8
> 0000000000000000
> (XEN)    0000000000000000 00000002d788742b 0000000000000000
> 0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<ffff82d0801f2910>] vmx.c#pi_wakeup_interrupt+0xcf/0x15b
> (XEN)    [<ffff82d080171c7a>] do_IRQ+0x9d/0x65e
> (XEN)    [<ffff82d08023feb2>] common_interrupt+0x62/0x70
> (XEN)    [<ffff82d0801ba6d2>] mwait-idle.c#mwait_idle+0x2b3/0x30a
> (XEN)    [<ffff82d0801660fd>] domain.c#idle_loop+0x6c/0x89
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 3:
> (XEN) GENERAL PROTECTION FAULT
> (XEN) [error_code=0000]
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> 
> 
> Best Regards,
> Xudong


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Wei Liu]

On Thu, May 26, 2016 at 01:56:28AM +0000, Wu, Feng wrote:
> The patch fixing this issue has already been sent to upstream.
> 
> " [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu blocking list "
> 
> v2 will be sent out soon.
> 

I just went through that thread. There are some open questions. Also Jan
requested that you explore other possible options.

My current plan is to not block the release for this -- we're very close
to the anticipated release date, and posted-interrupt is either
technical preview or experimental, so bugs there are expected.

This issue will be listed as a known issue for PI. And you can continue
to develop a fix for it. Your fix will be integrated with next version
of Xen. Then you can request for backport to 4.7 if you want to.

What do you think about this plan?

Wei.

> Thanks,
> Feng

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Wu, Feng]

> -----Original Message-----
> From: Wei Liu [mailto:wei.liu2@citrix.com]
> Sent: Thursday, May 26, 2016 6:27 PM
> To: Wu, Feng <feng.wu@intel.com>
> Cc: Hao, Xudong <xudong.hao@intel.com>; Xen-devel <xen-
> devel@lists.xenproject.org>; Wei Liu <wei.liu2@citrix.com>
> Subject: Re: [BUG] Xen panic with VT-d PI
> 
> On Thu, May 26, 2016 at 01:56:28AM +0000, Wu, Feng wrote:
> > The patch fixing this issue has already been sent to upstream.
> >
> > " [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu blocking list "
> >
> > v2 will be sent out soon.
> >
> 
> I just went through that thread. There are some open questions. Also Jan
> requested that you explore other possible options.
> 
> My current plan is to not block the release for this -- we're very close
> to the anticipated release date, and posted-interrupt is either
> technical preview or experimental, so bugs there are expected.
> 
> This issue will be listed as a known issue for PI. And you can continue
> to develop a fix for it. Your fix will be integrated with next version
> of Xen. Then you can request for backport to 4.7 if you want to.
> 
> What do you think about this plan?

I am going to send out the next version of this series, let's see how
it is going on. But basically, I am fine with your release plan.

Thanks,
Feng

> 
> Wei.
> 
> > Thanks,
> > Feng

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Wei Liu]

On Thu, May 26, 2016 at 01:03:53PM +0000, Wu, Feng wrote:
> 
> 
> > -----Original Message-----
> > From: Wei Liu [mailto:wei.liu2@citrix.com]
> > Sent: Thursday, May 26, 2016 6:27 PM
> > To: Wu, Feng <feng.wu@intel.com>
> > Cc: Hao, Xudong <xudong.hao@intel.com>; Xen-devel <xen-
> > devel@lists.xenproject.org>; Wei Liu <wei.liu2@citrix.com>
> > Subject: Re: [BUG] Xen panic with VT-d PI
> > 
> > On Thu, May 26, 2016 at 01:56:28AM +0000, Wu, Feng wrote:
> > > The patch fixing this issue has already been sent to upstream.
> > >
> > > " [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu blocking list "
> > >
> > > v2 will be sent out soon.
> > >
> > 
> > I just went through that thread. There are some open questions. Also Jan
> > requested that you explore other possible options.
> > 
> > My current plan is to not block the release for this -- we're very close
> > to the anticipated release date, and posted-interrupt is either
> > technical preview or experimental, so bugs there are expected.
> > 
> > This issue will be listed as a known issue for PI. And you can continue
> > to develop a fix for it. Your fix will be integrated with next version
> > of Xen. Then you can request for backport to 4.7 if you want to.
> > 
> > What do you think about this plan?
> 
> I am going to send out the next version of this series, let's see how
> it is going on. But basically, I am fine with your release plan.
> 

Ack. Thanks for confirming.

Wei.

> Thanks,
> Feng
> 
> > 
> > Wei.
> > 
> > > Thanks,
> > > Feng

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1795 bytes --]

On Thu, 2016-05-26 at 13:03 +0000, Wu, Feng wrote:
> > On Thu, May 26, 2016 at 01:56:28AM +0000, Wu, Feng wrote:
> > > 
> > > The patch fixing this issue has already been sent to upstream.
> > > 
> > > " [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu
> > > blocking list "
> > > 
> > > v2 will be sent out soon.
> > > 
> > I just went through that thread. There are some open questions.
> > Also Jan
> > requested that you explore other possible options.
> > 
> > My current plan is to not block the release for this -- we're very
> > close
> > to the anticipated release date, and posted-interrupt is either
> > technical preview or experimental, so bugs there are expected.
> > 
> > This issue will be listed as a known issue for PI. And you can
> > continue
> > to develop a fix for it. Your fix will be integrated with next
> > version
> > of Xen. Then you can request for backport to 4.7 if you want to.
> > 
> > What do you think about this plan?
> I am going to send out the next version of this series, let's see how
> it is going on. But basically, I am fine with your release plan.
> 
This does not have anything to do with Wei's plan (with which I also
agree), but are you sure that the bug being reported here is related to
and would be fixed by the patch you're mentioning?

Have you also seen it, and is that what led to the patch? If yes,
please, add a bit more information (including excerpts of, or insights
about the splat) in the patch changelog.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Xen panic with VT-d PI

[Wu, Feng]

> -----Original Message-----
> From: Dario Faggioli [mailto:dario.faggioli@citrix.com]
> Sent: Friday, May 27, 2016 12:11 AM
> To: Wu, Feng <feng.wu@intel.com>; Wei Liu <wei.liu2@citrix.com>
> Cc: Xen-devel <xen-devel@lists.xenproject.org>; Hao, Xudong
> <xudong.hao@intel.com>
> Subject: Re: [Xen-devel] [BUG] Xen panic with VT-d PI
> 
> On Thu, 2016-05-26 at 13:03 +0000, Wu, Feng wrote:
> > > On Thu, May 26, 2016 at 01:56:28AM +0000, Wu, Feng wrote:
> > > >
> > > > The patch fixing this issue has already been sent to upstream.
> > > >
> > > > " [PATCH 0/3] VMX: Properly handle pi descriptor and per-cpu
> > > > blocking list "
> > > >
> > > > v2 will be sent out soon.
> > > >
> > > I just went through that thread. There are some open questions.
> > > Also Jan
> > > requested that you explore other possible options.
> > >
> > > My current plan is to not block the release for this -- we're very
> > > close
> > > to the anticipated release date, and posted-interrupt is either
> > > technical preview or experimental, so bugs there are expected.
> > >
> > > This issue will be listed as a known issue for PI. And you can
> > > continue
> > > to develop a fix for it. Your fix will be integrated with next
> > > version
> > > of Xen. Then you can request for backport to 4.7 if you want to.
> > >
> > > What do you think about this plan?
> > I am going to send out the next version of this series, let's see how
> > it is going on. But basically, I am fine with your release plan.
> >
> This does not have anything to do with Wei's plan (with which I also
> agree), but are you sure that the bug being reported here is related to
> and would be fixed by the patch you're mentioning?

Yes, this bug is caused by device hotplug, which is fixed in this patch
set.

> 
> Have you also seen it, and is that what led to the patch? If yes,
> please, add a bit more information (including excerpts of, or insights
> about the splat) in the patch changelog.

The bug leads to this patch, and I also saw issues beyond it, such as,
the domain destroy, pCPU hotplug cases. Sure, I will add some info
about it in the next version.

Thanks,
Feng

> 
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel