From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lin, Ray" Subject: RE: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Fri, 12 Nov 2010 09:20:50 -0700 Message-ID: References: <20101112155659.GA5529@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101112155659.GA5529@dumpdata.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: Xen-devel , Dante Cinco List-Id: xen-devel@lists.xenproject.org =20 -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20 Sent: Friday, November 12, 2010 7:57 AM To: Lin, Ray Cc: Dante Cinco; Xen-devel Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 pvo= ps domU kernel with PCI passthrough On Thu, Nov 11, 2010 at 12:42:03PM -0700, Lin, Ray wrote: >=20 > Konrad, >=20 > See my response in red. Please don't top post. >=20 >=20 > -Ray >=20 > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com=20 > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Konrad=20 > Rzeszutek Wilk > Sent: Thursday, November 11, 2010 11:04 AM > To: Dante Cinco > Cc: Xen-devel > Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2=20 > pvops domU kernel with PCI passthrough >=20 > On Thu, Nov 11, 2010 at 10:31:48AM -0800, Dante Cinco wrote: > > Konrad, > > > > Without swiotlb=3Dforce, I don't see "PCI-DMA: Using software bounce=20 > > buffering for IO" in /var/log/kern.log. > > > > With iommu=3Dsoft and without swiotlb=3Dforce, I see the "software=20 > > bounce buffering" in /var/log/kern.log and an NMI (see below) when I=20 > > load the kernel module drivers. I made sure the NMI is reproducible=20 > > and not a >=20 > What is the kernel module doing to cause this? DMA? ??? What did it do? > > one-time event. >=20 > So doing 64-bit DMA causes an NMI. Do you have the Hypervisor's IOMMU VT-= d enabled or disabled? (iommu=3Doff,verbose) If you turn it off does this w= ork? >=20 > We have IOMMU VT-d enabled. If we turn it off (iommu=3Doff,verbose), the = DMA doesn't work properly and the driver code is unable to detect the sourc= e of interrupt. The interrupts of our device would be disabled by kernel ev= entually due to nobody services the interrupts for more than 100000 times. That does not sound right. You should be able to use the PCI passthrough wi= thout the IOMMU. Since it is an interrupt issue it sounds like that you are= using x2APIC and that is enabled without the IOMMU. Had you tried disabling IOMMU and x2apic? (this is all on the hypervisor li= ne?) Konrad, It's unlikely the interrupt issue but DMA issue. Here is the sequence how t= he tachyon device generates the DMA/interrupts, - the tachyon device does the DMA to update the memory which indicates the = source of interrupt. - After the DMA is done, the tachyon device trigger an interrupt. - The interrupt service routine of software driver is invoked due to the in= terrupt - The interrupt service routine checks the source of interrupts by examinin= g the memory which is supposed to be updated by previous DMA. - Even though the interrupt happens, the driver code can't find the source = of interrupt since the DMA doesn't work properly.