From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dante Cinco Subject: Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Fri, 12 Nov 2010 10:29:57 -0800 Message-ID: References: <20101111190351.GB15530@dumpdata.com> <20101112155659.GA5529@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101112155659.GA5529@dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: Xen-devel , "Lin, Ray" List-Id: xen-devel@lists.xenproject.org On Fri, Nov 12, 2010 at 7:56 AM, Konrad Rzeszutek Wilk wrote: > On Thu, Nov 11, 2010 at 12:42:03PM -0700, Lin, Ray wrote: >> >> Konrad, >> >> =A0 =A0See my response in red. > > Please don't top post. >> >> >> -Ray >> >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@li= sts.xensource.com] On Behalf Of Konrad Rzeszutek Wilk >> Sent: Thursday, November 11, 2010 11:04 AM >> To: Dante Cinco >> Cc: Xen-devel >> Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 = pvops domU kernel with PCI passthrough >> >> On Thu, Nov 11, 2010 at 10:31:48AM -0800, Dante Cinco wrote: >> > Konrad, >> > >> > Without swiotlb=3Dforce, I don't see "PCI-DMA: Using software bounce >> > buffering for IO" in /var/log/kern.log. >> > >> > With iommu=3Dsoft and without swiotlb=3Dforce, I see the "software bou= nce >> > buffering" in /var/log/kern.log and an NMI (see below) when I load the >> > kernel module drivers. I made sure the NMI is reproducible and not a >> >> What is the kernel module doing to cause this? DMA? > > ??? What did it do? > >> > one-time event. >> >> So doing 64-bit DMA causes an NMI. Do you have the Hypervisor's IOMMU VT= -d enabled or disabled? (iommu=3Doff,verbose) If you turn it off does this = work? >> >> We have IOMMU VT-d enabled. If we turn it off (iommu=3Doff,verbose), the= DMA doesn't work properly and the driver code is unable to detect the sour= ce of interrupt. The interrupts of our device would be disabled by kernel e= ventually due to nobody services the interrupts for more than 100000 times. > > That does not sound right. You should be able to use the PCI passthrough = without the IOMMU. Since it > is an interrupt issue it sounds like that you are using x2APIC and that i= s enabled without the IOMMU. > Had you tried disabling IOMMU and x2apic? (this is all on the hypervisor = line?) > > I set the hypervisor boot options to iommu=3D0 x2apic=3D0. I booted the pvops domU with swiotlb=3Dforce initially since that's the option that worked in the past. Not long after loading the kernel module drivers, domU hung/froze but dom0 stayed up. I checked the Xen interrupt bindings and I see the PCI-passthrough devices have either (PS--) or (-S--). (XEN) IRQ: 66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:72 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:127(-S--), (XEN) IRQ: 67 affinity:00000000,00000000,00000000,00000200 vec:3b type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:126(-S--), (XEN) IRQ: 68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:125(-S--), (XEN) IRQ: 69 affinity:00000000,00000000,00000000,00000800 vec:43 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:124(PS--), (XEN) IRQ: 70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:123(-S--), (XEN) IRQ: 71 affinity:fffffff,ffffffff,ffffffff,ffffffff vec:a2 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:122(PS--), (XEN) IRQ: 72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:121(-S--), (XEN) IRQ: 73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:120(PS--), (XEN) IRQ: 74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:119(-S--), (XEN) IRQ: 75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:118(PS--), (XEN) IRQ: 76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:117(-S--), (XEN) IRQ: 77 affinity:00000000,00000000,00000000,00080000 vec:4b type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:116(PS--), (XEN) IRQ: 78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:115(-S--), (XEN) IRQ: 79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:114(PS--), (XEN) IRQ: 80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:113(-S--), (XEN) IRQ: 81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:33 type=3DPCI-MSI status=3D00000010 in-flight=3D0 domain-list=3D1:112(PS--), I will reboot the pvops domU with iommu=3Dsoft and without swiotlb=3Dforce = next. - Dante