Re: [PATCH v2 10/11] xen/arm: introduce phys/dma translations in xen_dma_sync_for_*

From: Stefano Stabellini <sstabellini@kernel.org>
To: Christoph Hellwig <hch@infradead.org>
Cc: Stefano Stabellini <sstabellini@kernel.org>,
	jgross@suse.com, boris.ostrovsky@oracle.com,
	konrad.wilk@oracle.com, xen-devel@lists.xenproject.org,
	linux-kernel@vger.kernel.org, tamas@tklengyel.com,
	roman@zededa.com,
	Stefano Stabellini <stefano.stabellini@xilinx.com>
Subject: Re: [PATCH v2 10/11] xen/arm: introduce phys/dma translations in xen_dma_sync_for_*
Date: Mon, 8 Jun 2020 17:38:28 -0700 (PDT)	[thread overview]
Message-ID: <alpine.DEB.2.21.2006081614530.2815@sstabellini-ThinkPad-T480s> (raw)
In-Reply-To: <20200608071221.GF15742@infradead.org>

On Mon, 8 Jun 2020, Christoph Hellwig wrote:
> On Wed, Jun 03, 2020 at 03:22:46PM -0700, Stefano Stabellini wrote:
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > 
> > xen_dma_sync_for_cpu, xen_dma_sync_for_device, xen_arch_need_swiotlb are
> > getting called passing dma addresses. On some platforms dma addresses
> > could be different from physical addresses. Before doing any operations
> > on these addresses we need to convert them back to physical addresses
> > using dma_to_phys.
> > 
> > Add dma_to_phys calls to xen_dma_sync_for_cpu, xen_dma_sync_for_device,
> > and xen_arch_need_swiotlb.
> > 
> > dma_cache_maint is fixed by the next patch.
> 
> The calling conventions because really weird now because
> xen_dma_sync_for_{device,cpu} already get both a phys_addr_t and
> a dma_addr_t.  
> 
> > 
> > -	if (pfn_valid(PFN_DOWN(handle)))
> > +	if (pfn_valid(PFN_DOWN(dma_to_phys(dev, handle))))
> 
> But here we translate the dma address to a phys addr
> 
> >  		arch_sync_dma_for_cpu(paddr, size, dir);
> 
> While this still uses the passed in paddr.  I think the uses of
> addresses in this code really needs a major rethink.

Yeah, the pfn_valid check is a bit weird by definition because we are
using it to understand whether the address belong to us or to another
VM. To do the pfn_valid check we need to translate the dma address into
something the CPU understands, hence, the dma_to_phys call.

Why can't we use the already-provided paddr? Because paddr has been
translated twice:
1) from dma to maybe-foreign phys address (could be ours, or another VM)
2) from maybe-foreign address to local (using our local mapping of the foreign page)

In fact, it would be clearer if we had all three addresses as parameters
of xen_dma_sync_for_cpu: the dma address, the maybe-foreign physical
address (we tend to call it xenbus address, baddr), the local physical
address. Something like:

void xen_dma_sync_for_cpu(struct device *dev, dma_addr_t handle,
			  phys_addr_t baddr, phys_addr_t paddr, size_t size,
			  enum dma_data_direction dir)
{
	if (pfn_valid(baddr))
		arch_sync_dma_for_cpu(paddr, size, dir);
	else if (dir != DMA_TO_DEVICE)
		dma_cache_maint(dev, handle, size, GNTTAB_CACHE_INVAL);
}