Re: [RFC + Queries] Flow of PCI passthrough in ARM

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Cc: anup.patel@linaro.org, Vijay Kilari <vijay.kilari@gmail.com>,
	Prasun Kapoor <prasun.Kapoor@caviumnetworks.com>,
	manish.jaggi@caviumnetworks.com,
	Julien Grall <julien.grall@linaro.org>,
	xen-devel <xen-devel@lists.xen.org>,
	psawargaonkar@linaro.org,
	manish jaggi <manishjaggi.oss@gmail.com>
Subject: Re: [RFC + Queries] Flow of PCI passthrough in ARM
Date: Mon, 22 Sep 2014 12:09:33 +0100	[thread overview]
Message-ID: <1411384173.18331.40.camel@kazak.uk.xensource.com> (raw)
In-Reply-To: <alpine.DEB.2.02.1409221057490.29862@kaball.uk.xensource.com>

On Mon, 2014-09-22 at 11:45 +0100, Stefano Stabellini wrote:
> On Thu, 18 Sep 2014, manish jaggi wrote:
> > Hi,
> > Below is the flow I am working on, Please provide your comments, I
> > have a couple of queries as well..
> > 
> > a) Device tree has smmu nodes and each smmu node has the mmu-master property.
> > In our Soc DT the mmu-master is a pcie node in device tree.
> 
> Do you mean that both the smmu nodes and the pcie node have the
> mmu-master property? The pcie node is the pcie root complex, right?
> 
> 
> > b) Xen parses the device tree and prepares a list which stores the pci
> > device tree node pointers. The order in device tree is mapped to
> > segment number in subsequent calls. For eg 1st pci node found is
> > segment 0, 2nd segment 1
> 
> What's a segment number? Something from the PCI spec?
> If you have several pci nodes on device tree, does that mean that you
> have several different pcie root complexes?
> 
> 
> > c) During SMMU init the pcie nodes in DT are saved as smmu masters.
> 
> At this point you should also be able to find via DT the stream-id range
> supported by each SMMU and program the SMMU with them, assigning
> everything to dom0.
> 
> 
> > d) Dom0 Enumerates PCI devices, calls hypercall PHYSDEVOP_pci_device_add.
> >  - In Xen the SMMU iommu_ops add_device is called. I have implemented
> > the add_device function.
> > - In the add_device function
> >  the segment number is used to locate the device tree node pointer of
> > the pcie node which helps to find out the corresponding smmu.
> > - In the same PHYSDEVOP the BAR regions are mapped to Dom0.
> > 
> > Note: The current SMMU driver maps the complete Domain's Address space
> > for the device in SMMU hardware.
> > 
> > The above flow works currently for us.
> 
> It would be nice to be able to skip d): in a system where all dma capable
> devices are behind smmus, we should be capable of booting dom0 without
> the 1:1 mapping hack. If we do that, it would be better to program the
> smmus before booting dom0. Otherwise there is a risk that dom0 is going
> to start using these devices and doing dma before we manage to secure
> the devices via smmus.

Without commenting on whether we should take this approach or not note
that the default before the call to pci_device_add should be to deny, so
there is no risk from a security perspective of doing things this way.

If the dom0 kernel tries to use a PCI device which it hasn't registered
then that would be a dom0 bug under this model. Probably an easily
avoided one since you would naturally want to call pci_device_add during
bus enumeration, I'd imagine, and a dom0 kernel which uses a device
before enumerating it would be pretty broken I think.

Ian.