From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Lin, Ray" <Ray.Lin@lsi.com>
Subject: RE: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops
	domU kernel with PCI passthrough
Date: Fri, 12 Nov 2010 15:57:01 -0700
Message-ID: <EB4C61A1A2501842A04B573FE42B14D601374FC079@cosmail02.lsi.com>
References: <20101112223333.GD26189@dumpdata.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Return-path: <xen-devel-bounces@lists.xensource.com>
In-Reply-To: <20101112223333.GD26189@dumpdata.com>
Content-Language: en-US
List-Unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Xen-devel <xen-devel@lists.xensource.com>, Dante Cinco <dantecinco@gmail.com>
List-Id: xen-devel@lists.xenproject.org

=20

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20
Sent: Friday, November 12, 2010 2:34 PM
To: Lin, Ray
Cc: Xen-devel; Dante Cinco
Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 pvo=
ps domU kernel with PCI passthrough

>> That sounds like the tachyon device is updating the wrong memory locatio=
n. How are you programming the memory location where thetachyon device is s=
uppose to touch? Are you using the value from pci_map_page or are you using=
 virt_to_phys? The virt_to_phys should be different from the pci_map_page..=
 unless you allocated a coherent DMA pool using pci_alloc_coherent in which=
 case the virt_to_phys() values for that pool should be the right MFNs.
>=20
> Our driver uses pci_map_single to get the physical addr to program the ch=
ip.

OK. Good.
>=20
>=20
> One way you can figure this is doing something like this to make sure you=
 got the right MFN:
>=20
> add these two:
> #include <xen/page.h>
> #include <asm/xen/page.h>
>=20
>           phys_addr_t phys =3D page_to_phys(mem->pages[i]);
> +               if (xen_pv_domain()) {
> +                       phys_addr_t xen_phys =3D PFN_PHYS(pfn_to_mfn(
> +                                       page_to_pfn(mem->pages[i])));
> +                       if (phys !=3D xen_phys) {
> +                               printk(KERN_ERR "Fixing up: (0x%lx->0x%lx=
)." \
> +                                       " CODE UNTESTED!\n",
> +                                       (unsigned long)phys,
> +                                       (unsigned long)xen_phys);
> +                               WARN_ON_ONCE(phys !=3D xen_phys);
> +                               phys =3D xen_phys;
> +                       }
> +               }
> 	and using the 'phys' value from now.
>=20
>=20
>=20
> If this sounds like black magic, here is a short writeup=20
> http://wiki.xensource.com/xenwiki/XenPVOPSDRM
>=20
> look at "Why those patches" section.
>=20
> Lastly, are you using unsigned long for or the phys_addr_t typedefs?
>=20
> The driver uses dma_addr_t for physical address.=20

Excellent.
>=20
> The more I think about your problem the more it sounds like a truncating =
issue. You said that it works just right (albeit slow) if you use 'swiotlb=
=3Dforce'. The slowness could be due to not using the pci_sync_* APIs to sy=
nc the DMA buffers.. But irregardless using bounce buffers will slow the DM=
A operations down.
>=20
> The driver do use pci_dma_sync_single_for_cpu or pci_dma_sync_single_for_=
device to sync the DMA buffers. Without these syncs, the driver would not w=
ork at all.

<nods> That makes sense.
>=20
> Using the bounce buffers limits the DMA operations to under 32-bit. So co=
uld it be that you are using some casting macro that casts a PFN to unsigne=
d long or vice-versa and we end up truncating it to 32-bit? (I've seen this=
 issue actually with InfiniBand drivers back in RHEL5 days..). Lastly, do y=
ou set your DMA mask on the device to 32BIT?
>=20
> The tachyon chip supports both 32-bit & 45-bit dma. Some features need to=
 set 32-bit physical addr to chip. Others need to set 45-bit physical addr =
to chip.=20

Oh boy. That complicates it.=20

> The driver doesn't set DMA mask on the device to 32 bit.

Is it set then to 45bit?
The driver doesn't use pci_set_dma_mask to set the HW_DMA_MASK.
The tachyon chip should support 64-bit dma transfer even though dma program=
mable address is limited to 32-bit/45-bit address. The chip should fill the=
 upper address with 0. I'm confirming it with their fae now. In the mean ti=
me, I try to manipulate pci_set_dma_mask to see it make the difference or n=
ot.=20