RE: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough

["Lin, Ray" <Ray.Lin@lsi.com>]

 

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Friday, November 12, 2010 7:57 AM
To: Lin, Ray
Cc: Dante Cinco; Xen-devel
Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough

On Thu, Nov 11, 2010 at 12:42:03PM -0700, Lin, Ray wrote:
> 
> Konrad,
> 
>    See my response in red.

Please don't top post.
> 
> 
> -Ray
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Konrad 
> Rzeszutek Wilk
> Sent: Thursday, November 11, 2010 11:04 AM
> To: Dante Cinco
> Cc: Xen-devel
> Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 
> pvops domU kernel with PCI passthrough
> 
> On Thu, Nov 11, 2010 at 10:31:48AM -0800, Dante Cinco wrote:
> > Konrad,
> >
> > Without swiotlb=force, I don't see "PCI-DMA: Using software bounce 
> > buffering for IO" in /var/log/kern.log.
> >
> > With iommu=soft and without swiotlb=force, I see the "software 
> > bounce buffering" in /var/log/kern.log and an NMI (see below) when I 
> > load the kernel module drivers. I made sure the NMI is reproducible 
> > and not a
> 
> What is the kernel module doing to cause this? DMA?

??? What did it do?

> > one-time event.
> 
> So doing 64-bit DMA causes an NMI. Do you have the Hypervisor's IOMMU VT-d enabled or disabled? (iommu=off,verbose) If you turn it off does this work?
> 
> We have IOMMU VT-d enabled. If we turn it off (iommu=off,verbose), the DMA doesn't work properly and the driver code is unable to detect the source of interrupt. The interrupts of our device would be disabled by kernel eventually due to nobody services the interrupts for more than 100000 times.

That does not sound right. You should be able to use the PCI passthrough without the IOMMU. Since it is an interrupt issue it sounds like that you are using x2APIC and that is enabled without the IOMMU.
Had you tried disabling IOMMU and x2apic? (this is all on the hypervisor line?)

Konrad,
It's unlikely the interrupt issue but DMA issue. Here is the sequence how the tachyon device generates the DMA/interrupts,
- the tachyon device does the DMA to update the memory which indicates the source of interrupt.
- After the DMA is done, the tachyon device trigger an interrupt.
- The interrupt service routine of software driver is invoked due to the interrupt
- The interrupt service routine checks the source of interrupts by examining the memory which is supposed to be updated by previous DMA.
- Even though the interrupt happens, the driver code can't find the source of interrupt since the DMA doesn't work properly.