Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Dante Cinco <dantecinco@gmail.com>
Cc: Xen-devel <xen-devel@lists.xensource.com>
Subject: Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough
Date: Thu, 11 Nov 2010 11:04:59 -0500	[thread overview]
Message-ID: <20101111160459.GB25654@dumpdata.com> (raw)
In-Reply-To: <AANLkTi=q_ZgxEm9gYB6LvWOcJNnozW3cYqaySowTnD2y@mail.gmail.com>

On Wed, Nov 10, 2010 at 05:16:14PM -0800, Dante Cinco wrote:
> We have Fibre Channel HBA devices that we PCI passthrough to our pvops
> domU kernel. Without swiotlb=force in the domU's kernel command line,
> both domU and dom0 lock up after loading the kernel module drivers for
> the HBA devices. With swiotlb=force, the domU and dom0 are stable

Whoa. That is not good - what happens if you just pass in iommu=soft?
Does the PCI-DMA: Using.. show up if you don't pass in any of those parameters?
(I don't think it does, but just doing 'iommu=soft' should enable it).

> after loading the kernel module drivers but the I/O performance is at
> least an order of magnitude worse than what we were seeing with the
> HVM kernel. I see the following in /var/log/kern.log in the pvops
> domU:
> 
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> Placing 64MB software IO TLB between ffff880005800000 - ffff880009800000
> software IO TLB at phys 0x5800000 - 0x9800000
> 
> Is swiotlb=force responsible for the I/O performance degradation? I
> don't understand what swiotlb=force does so I would appreciate an
> explanation or a pointer.

So, you should only need to use 'iommu=soft'. It will enable the Linux kernel IOMMU
to translate the pseudo-PFNs to the real machine frame numbers (bus addresses).

If your card is 64-bit, then that is all it would do. If however your card is 32-bit
and your are DMA-ing data from above the 32-bit limit, it would copy the user-space page
to memory below 4GB, DMA that, and when done, copy it back to the where the user-space
page is. This is called bounce-buffering and this is why you would use a mix of
pci_map_page, pci_sync_single_for_[cpu|device] calls around your driver.

However, I think your cards are 64-bit, so you don't need this bounce-buffering. But
if you say 'swiotlb=force' it will force _all_ DMAs to go through the bounce-buffer.

So, try just 'iommu=soft' and see what happens.