From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dante Cinco Subject: Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Tue, 16 Nov 2010 09:07:28 -0800 Message-ID: References: <20101112165541.GA10339@dumpdata.com> <20101112223333.GD26189@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101112223333.GD26189@dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: Xen-devel List-Id: xen-devel@lists.xenproject.org On Fri, Nov 12, 2010 at 2:33 PM, Konrad Rzeszutek Wilk wrote: >>> That sounds like the tachyon device is updating the wrong memory locati= on. How are you programming the memory location where thetachyon device is = suppose to touch? Are you using the value from pci_map_page or are you usin= g virt_to_phys? The virt_to_phys should be different from the pci_map_page.= . unless you allocated a coherent DMA pool using pci_alloc_coherent in whic= h case the virt_to_phys() values for that pool should be the right MFNs. >> >> Our driver uses pci_map_single to get the physical addr to program the c= hip. > > OK. Good. >> >> >> One way you can figure this is doing something like this to make sure yo= u got the right MFN: >> >> add these two: >> #include >> #include >> >> =A0 =A0 =A0 =A0 =A0 phys_addr_t phys =3D page_to_phys(mem->pages[i]); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (xen_pv_domain()) { >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 phys_addr_t xen_phys =3D P= FN_PHYS(pfn_to_mfn( >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 page_to_pfn(mem->pages[i]))); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (phys !=3D xen_phys) { >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KER= N_ERR "Fixing up: (0x%lx->0x%lx)." \ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 " CODE UNTESTED!\n", >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 (unsigned long)phys, >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 (unsigned long)xen_phys); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 WARN_ON_ON= CE(phys !=3D xen_phys); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 phys =3D x= en_phys; >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } >> =A0 =A0 =A0 and using the 'phys' value from now. >> >> >> >> If this sounds like black magic, here is a short writeup http://wiki.xen= source.com/xenwiki/XenPVOPSDRM >> >> look at "Why those patches" section. >> >> Lastly, are you using unsigned long for or the phys_addr_t typedefs? >> >> The driver uses dma_addr_t for physical address. > > Excellent. >> >> The more I think about your problem the more it sounds like a truncating= issue. You said that it works just right (albeit slow) if you use 'swiotlb= =3Dforce'. The slowness could be due to not using the pci_sync_* APIs to sy= nc the DMA buffers.. But irregardless using bounce buffers will slow the DM= A operations down. >> >> The driver do use pci_dma_sync_single_for_cpu or pci_dma_sync_single_for= _device to sync the DMA buffers. Without these syncs, the driver would not = work at all. > > That makes sense. >> >> Using the bounce buffers limits the DMA operations to under 32-bit. So c= ould it be that you are using some casting macro that casts a PFN to unsign= ed long or vice-versa and we end up truncating it to 32-bit? (I've seen thi= s issue actually with InfiniBand drivers back in RHEL5 days..). Lastly, do = you set your DMA mask on the device to 32BIT? >> >> The tachyon chip supports both 32-bit & 45-bit dma. Some features need t= o set 32-bit physical addr to chip. Others need to set 45-bit physical addr= to chip. > > Oh boy. That complicates it. > >> The driver doesn't set DMA mask on the device to 32 bit. > > Is it set then to 45bit? > We were not explicitly setting the DMA mask. pci_alloc_coherent was always returning 32 bits but pci_map_single was returning a 34-bit address which we truncate by casting it to a uint32_t since the Tachyon's HBA register is only 32 bits. With swiotlb=3Dforce, both returned 32 bits without explicitly setting the DMA mask. Once we set the mask to 32 bits using pci_set_dma_mask, the NMIs stopped. However with iommu=3Dsoft (and no more swiotlb=3Dforce), we're still stuck with the abysmal I/O performance (same as when we had swiotlb=3Dforce). In pvops domU (xen-pcifront-0.8.2), what does iommu=3Dsoft do? What's the default if we don't specify it? Without it, we get no I/Os (it seems the interrupts and/or DMA don't work). Are there any profiling tools you can suggest for domU? I was able to apply Dulloor's xenoprofile patch to our dom0 kernel (2.6.32.25-pvops) but not to xen-pcifront-0.8.2. - Dante