Re: PCI Pass-through in Xen ARM - Draft 2.

From: Julien Grall <julien.grall@citrix.com>
To: Manish Jaggi <mjaggi@caviumnetworks.com>,
	Ian Campbell <ian.campbell@citrix.com>
Cc: Prasun Kapoor <Prasun.kapoor@caviumnetworks.com>,
	"Kumar, Vijaya" <Vijaya.Kumar@caviumnetworks.com>,
	Stefano Stabellini <stefano.stabellini@citrix.com>,
	"Kulkarni, Ganapatrao" <Ganapatrao.Kulkarni@caviumnetworks.com>,
	"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: PCI Pass-through in Xen ARM - Draft 2.
Date: Thu, 9 Jul 2015 15:57:20 +0200	[thread overview]
Message-ID: <559E7DC0.7070707@citrix.com> (raw)
In-Reply-To: <559E4D3C.2010805@caviumnetworks.com>

On 09/07/2015 12:30, Manish Jaggi wrote:
> On Thursday 09 July 2015 01:38 PM, Julien Grall wrote:
>> On 09/07/2015 08:13, Manish Jaggi wrote:
>>>>
>>>> If this was a domctl there might be scope for accepting an
>>>> implementation which made assumptions such as sbdf == deviceid. However
>>>> I'd still like to see this topic given proper treatment in the design
>>>> and not just glossed over with "this is how ThunderX does things".
>>> I got your point.
>>>> Or maybe the solution is simple and we should just do it now -- i.e.
>>>> can
>>>> we add a new field to the PHYSDEVOP_pci_host_bridge_add argument struct
>>>> which contains the base deviceid for that bridge
>>> deviceId would be same as sbdf. As we dont have a way to translate sbdf
>>> to deviceID.
>>
>> I think we have to be clear in this design document about the
>> different meaning.
>>
>> When the Device Tree is used, it's assumed that the deviceID will be
>> equal to the requester ID and not the sbdf.
> Does SMMU v2 has a concept of requesterID.
> I see requesterID term in SMMUv3

The requester ID is part of the PCI spec and not the SMMU.

The version of the SMMUv2 spec doesn't mention anything about PCI. I 
suspect this is because the spec has been written before the introduced 
PCI. And therefore this is implementation defined.

>>
>> Linux provides a function (pci_for_each_dma_alias) which will return a
>> requester ID for a given PCI device. It appears that the BDF (the 's'
>> of sBDF is only internal to Linux and not part of the hardware) is
>> equal to the requester ID on your platform but we can't assume it for
>> anyone else.
> so you mean requesterID = pci_for_each_dma_alias(sbdf)

Yes.

>>
>> When we have a PCI in hand, we have to find the requester ID for this
>> device.
> That is the question. How to map requesterID to sbdf

See above.

>> On
> Once ?

Yes.

>> we have it we can deduce the streamID and the deviceID. The way to do
>> it will depend on whether we use device tree or ACPI:
>>     - For device tree, the streamID, and deviceID will be equal to the
>> requester ID
> what do you think should be streamID when a device is PCI EP and is
> enumerated. Also per ARM SMMU 2.0 spec  StreamID is implementation
> specific.
> As per SMMUv3 specs
> "For PCI, it is intended that StreamID is generated from the PCI
> RequesterID. The generation function may be 1:1
> where one Root Complex is hosted by one SMMU"

I think my sentence "For device tree, the streamID, and deviceID will be 
equal to the requester ID" is pretty clear. FWIW, this is the solution 
chosen for Linux:

"Assume Stream ID == Requester ID for now. We need a way to describe the 
ID mappings in FDT" (see arm_smmu_add_pci_device in 
drivers/iommu/arm-smmu.c).

You can refer to my point below about ACPI tables. The solution would be 
exactly the same. If we have a requester ID in hand we can do pretty 
much everything.

The whole point of my previous email is to give insight about what we 
need and what we can deduce based on firmware tables. I didn't cover 
anything implementation details.

Regards,

-- 
Julien Grall