From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lin, Ray" Subject: RE: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Fri, 12 Nov 2010 12:38:59 -0700 Message-ID: References: <20101112165541.GA10339@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101112165541.GA10339@dumpdata.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: Xen-devel , Dante Cinco List-Id: xen-devel@lists.xenproject.org =20 -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20 Sent: Friday, November 12, 2010 8:56 AM To: Lin, Ray Cc: Dante Cinco; Xen-devel Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 pvo= ps domU kernel with PCI passthrough > That does not sound right. You should be able to use the PCI passthrough = without the IOMMU. Since it is an interrupt issue it sounds like that you a= re using x2APIC and that is enabled without the IOMMU. > Had you tried disabling IOMMU and x2apic? (this is all on the=20 > hypervisor line?) >=20 > Konrad, > It's unlikely the interrupt issue but DMA issue. Here is the sequence=20 > how the tachyon device generates the DMA/interrupts, > - the tachyon device does the DMA to update the memory which indicates th= e source of interrupt. > - After the DMA is done, the tachyon device trigger an interrupt. > - The interrupt service routine of software driver is invoked due to=20 > the interrupt > - The interrupt service routine checks the source of interrupts by examin= ing the memory which is supposed to be updated by previous DMA. > - Even though the interrupt happens, the driver code can't find the sourc= e of interrupt since the DMA doesn't work properly. That sounds like the tachyon device is updating the wrong memory location. = How are you programming the memory location where thetachyon device is supp= ose to touch? Are you using the value from pci_map_page or are you using vi= rt_to_phys? The virt_to_phys should be different from the pci_map_page.. un= less you allocated a coherent DMA pool using pci_alloc_coherent in which ca= se the virt_to_phys() values for that pool should be the right MFNs. Our driver uses pci_map_single to get the physical addr to program the chip= . One way you can figure this is doing something like this to make sure you g= ot the right MFN: add these two: #include #include phys_addr_t phys =3D page_to_phys(mem->pages[i]); + if (xen_pv_domain()) { + phys_addr_t xen_phys =3D PFN_PHYS(pfn_to_mfn( + page_to_pfn(mem->pages[i]))); + if (phys !=3D xen_phys) { + printk(KERN_ERR "Fixing up: (0x%lx->0x%lx).= " \ + " CODE UNTESTED!\n", + (unsigned long)phys, + (unsigned long)xen_phys); + WARN_ON_ONCE(phys !=3D xen_phys); + phys =3D xen_phys; + } + } and using the 'phys' value from now. If this sounds like black magic, here is a short writeup http://wiki.xensou= rce.com/xenwiki/XenPVOPSDRM look at "Why those patches" section. Lastly, are you using unsigned long for or the phys_addr_t typedefs? The driver uses dma_addr_t for physical address.=20 The more I think about your problem the more it sounds like a truncating is= sue. You said that it works just right (albeit slow) if you use 'swiotlb=3D= force'. The slowness could be due to not using the pci_sync_* APIs to sync = the DMA buffers.. But irregardless using bounce buffers will slow the DMA o= perations down. The driver do use pci_dma_sync_single_for_cpu or pci_dma_sync_single_for_de= vice to sync the DMA buffers. Without these syncs, the driver would not wor= k at all. Using the bounce buffers limits the DMA operations to under 32-bit. So coul= d it be that you are using some casting macro that casts a PFN to unsigned = long or vice-versa and we end up truncating it to 32-bit? (I've seen this i= ssue actually with InfiniBand drivers back in RHEL5 days..). Lastly, do you= set your DMA mask on the device to 32BIT? The tachyon chip supports both 32-bit & 45-bit dma. Some features need to s= et 32-bit physical addr to chip. Others need to set 45-bit physical addr to= chip.=20 The driver doesn't set DMA mask on the device to 32 bit. I'm looking at the driver code to see anything wrong or not. We appreciate = your help Konrad.