Re: [RFC + Queries] Flow of PCI passthrough in ARM

[manish jaggi <manishjaggi.oss@gmail.com>]

On 8 October 2014 18:16, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> On Wed, Oct 08, 2014 at 05:16:53PM +0530, manish jaggi wrote:
>> On 7 October 2014 23:47, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > As the discussion is becoming Xen specific, reduce the CC list.
>> >
>> > On Mon, 6 Oct 2014, manish jaggi wrote:
>> >> On 6 October 2014 19:41, Stefano Stabellini
>> >> <stefano.stabellini@eu.citrix.com> wrote:
>> >> > On Mon, 6 Oct 2014, manish jaggi wrote:
>> >> >> Below is the Design Flow:
>> >> >>
>> >> >> =====
>> >> >> Xen PCI Passthrough support for ARM
>> >> >>
>> >> >> ----------------------------------------------
>> >> >>
>> >> >>
>> >> >> Nomenclature:
>> >> >>
>> >> >> SBDF - segment:bus:device.function.
>> >> >> Segment - Number of the PCIe RootComplex. (this is also referred to as
>> >> >> domain in linux terminology Domain:BDF)
>> >> >> domU sbdf - refers to the sbdf of the device assigned when enumerated
>> >> >> in domU. This is different from the actual sbdf.
>> >> >>
>> >> >>
>> >> >> What is the requirement
>> >> >>
>> >> >> 1. Any PCI device be assigned to any domU at runtime.
>> >> >> 2. The assignment should not be static, system admin must be able to
>> >> >> assign a PCI BDF at will.
>> >> >> 3. SMMU should be programmed for the device/domain pair
>> >> >> 4. MSI should be directly injected into domU.
>> >> >> 5. Access to GIC should be trapped in Xen to program MSI(LPI) in ITS
>> >> >> 6. FrontEnd - Backend communication between DomU-Dom0 must be limited
>> >> >> to PCI config read writes.
>> >> >>
>> >> >> What is support today
>> >> >>
>> >> >> 1. Non PCI passthrough using a device tree. A device tree node can be
>> >> >> passthrough to a domU.
>> >> >> 2. SMMU is programmed (devices' stream ID is allowed to access domU
>> >> >> guest memory)
>> >> >> 3. By default a device is assigned to dom0 if not done a pci-hide.
>> >> >> (SMMU for that device is programmed for dom0)
>> >> >>
>> >> >>
>> >> >> Proposed Flow:
>> >> >>
>> >> >> 1. Xen parses its device tree to find the pci nodes. There nodes
>> >> >> represent the PCIe Root Complexes
>> >> >>
>> >> >> 2. Xen parses its device tree to find smmu nodes which have mmu-master
>> >> >> property which should point to the PCIe RCs (pci nodes)
>> >> >> Note: The mapping between the pci node and the smmu node is based on:
>> >> >> which smmu which translates r/w requests from that pcie
>> >> >>
>> >> >> 3. All the pci nodes are assigned to the dom0.
>> >> >> 4. dom0 boots
>> >> >>
>> >> >> 5. dom0 enumerates all the PCI devices and calls xen hypercall
>> >> >> PHYSDEVOP_pci_device_add. This in-turn programs the smmu
>> >> >> Note: Also The MMIO regions for the device are assigned to the dom0.
>> >> >>
>> >> >> 6. dom0 boots domU.
>> >> >> Note: in domU, the pcifront bus has a msi-controller attached. It is
>> >> >> assumed that the domU device tree contains the 'gicv3-its' node.
>> >> >>
>> >> >> 7. dom0 calls xl pci-assignable-add <sbdf>
>> >> >>
>> >> >> 8. dom0 calls xl pci-attach <domU_id> '<sbdf>,permissive=1'
>> >> >> Note: XEN_DOMCTL_assign_device is called. Implemented assign_device in smmu.c
>> >> >> The MMIO regions for the device are assigned to the respective domU.
>> >> >>
>> >> >> 9. Front end driver is notified by xenwatch daemon and it starts
>> >> >> enumerating devices.
>> >> >> Note: the check of initial_domain in register_xen_pci_notifier() is removed.
>> >> >>
>> >> >>
>> >> >> 10. The device driver requests the msi using pcie_enable_msi, which in
>> >> >> turn calls the its driver which tries to issue commands like MAPD,
>> >> >> MAPVI. which are trapped by the Xen ITS emulation driver.
>> >> >> 10a. The MAPD command contains a device id, which is a stream ID
>> >> >> constructed using the sbdf. The sbdf is specific to domU for that
>> >> >> device.
>> >> >> 10b. Xen ITS driver has to replace the domU sbdf to the actual sdbf
>> >> >> and program the command in the command queue.
>> >> >>
>> >> >> Now there is a _problem_ that xen does not know the mapping between
>> >> >> domU streamid (sbdf) and actual sbdf.
>> >> >> For ex if two pci devices are assigned to domU, xen does not know that
>> >> >> which domU sbdf maps to which pci device
>> >> >> Thus the Xen ITS driver fails to program the MAPD command in command
>> >> >> queue, which results LPIs not programmed for that device
>> >> >>
>> >> >> At the time xl pci-attach is called the domU sbdf of the device is not
>> >> >> known, as the enumeration of the PCI device in domU has not started by
>> >> >> that time.
>> >> >> in xl pci-attach a virtual slot number can be provided but it is not
>> >> >> used in ITS commands in domU.
>> >> >>
>> >> >> If it is known somehow (__we need help on this__) then dom0 could
>> >> >> issue a hypercall to xen to map domU sbdf to actual in the following
>> >> >> format
>> >> >>
>> >> >> PHYSDEVOP_map_domU_sbdf{
>> >> >>
>> >> >> actual sBDF,
>> >> >> domU enumerated sBDF
>> >> >> and domU_ID.
>> >> >>
>> >> >> }
>> >> >
>> >> > Now the problem is much much clearer, thank you!
>> >> >
>> >> > Actually the xen-pcifront driver in the guest knows the real PCI sbdf
>> >> > for the assigned device, not just the virtual slot.
>> >> Could you please help in the data structure where it is stored.
>> >
>> > Actually I was wrong. It is true that the real sbdf is exposed to the
>> > guest via xenstore (see the dev-0 backend node) but it is not currently
>> > read by the guest and probably it shouldn't be readable in the first
>> > place. It's best not to rely on it.
>> >
>> > The virtual to physical sbdf mapping is done by pciback, see
>> > drivers/xen/xen-pciback/xenbus.c and drivers/xen/xen-pciback/vpci.c.
>> > Pciback should be one telling Xen what the mapping is.
>> > Unfortunately I think that you'll probably have to introduce a new
>> > hypercall to do it.
>> >
>> >
>> >
>> >> PS: in this mail thread you were averse to the fact that why guest
>> >> should know the real sbdf.
>> >
>> > I am averse to having to rely on real StreamIDs being exposed and used
>> > in the guest virtual ITS.
>> >
>> >
>> >> >On x86 xen-pcifront
>> >> > makes an hypercall to enable msi/msix on the device passing the real
>> >> > sbdf as argument:
>> >> On ARM since for non msi interrupts (SPI, SGI's) GIC access is
>> >> directly trapped in Xen, it makes sense to use traps for LPIs
>> >> The same logic is being proposed. Apart from PCI config space r/w the
>> >> proposal is not to use front-end back-end communication for MSIs.
>> >> In the long run, I plan to trap directly into Xen for PCI config space r/w.
>> >> Also in our previous mails we agreed on utilising arm trap and emulate for MSI.
>> >
>> > OK
>> >
>> >
>> >> >
>> >> > drivers/pci/xen-pcifront.c:pci_frontend_enable_msix
>> >> >
>> >> > Could we use the same hypercall to enable msi/msix on ARM? That would be
>> >> > ideal.
>> >> >
>> >> > Otherwise xen-pcifront could call a new hypercall to let Xen know the
>> >> > virtual sbdf to sbdf mapping. But I would prefer not to introduce a new
>> >> > hypercall and reuse the existing one.
>> >> We are proposing removal of pcifront back communication for MSIs. I do
>> >> not want to introduce a new hypercall
>> >> If we know the guest sbdf it can be done in the pci-attach DOMCTL itself.
>> >
>> > I agree that it would be nice to avoid a new hypercall but it might be
>> > the only way to do it. Dig through the pciback code and see what you can
>> > do, I am open to alternatives.
>> > pciback code and see what you can do.
>> Stefano, I need your help on this, can you point out someone who can help.
>> I am adding Ryan Wilson the author as well. Is he still active on Xen ?
>
> No, but what is the help you need?
Thanks for replying. As detailed in this thread, I need to create a
hypercall that would send the following information to Xen at the time
of PCI attach
{ sbdf , domU sbdf, domainId }.
I am not able to find a way to get the domU sbdf from dom0 at the time
of pci-attach.