Re: Exploring the PCI EP Framework

Re: Exploring the PCI EP Framework

[Kishon Vijay Abraham I]

+linux-pci

Hi,

On 12/11/19 5:18 PM, Ismael Luceno Cortes wrote:
> Hello Kishon and Lorenzo,
> 
> I'm in the process of evaluating options to implement virtual devices,
> and I have a doubt that I couldn't solve by a quick look at the code.
> 
> I have some prototype boards with general purpose cores that can
> synthesize devices (sadly a custom driver, but I plan to replace it if I
> get approval).
> 
> The cores are for the most part equal, even in function, so the problem
> is that for whichever takes the role of endpoint I need a lightweight
> mechanism to signal an incoming transfer.
> 
> I didn't see any way to implement doorbell registers; is there such a
> thing?

The endpoint relies on polling a shared memory to detect any events. However if
the HW supports some sort of memory signaled interrupt (similar to what is
usually available in systems with RC) then we could use it for interrupting the
endpoint system. All we have to do is map the MSI address to a EP BAR region so
that host can write to a mapped BAR in order to raise MSI interrupt.

Even if the HW supports, the EP framework doesn't support to use MSI. I'll have
to see how that can be added.
> 
> I realize it may not exist because it would imply not yet existing
> mechanisms. I even don't know if such a thing is feasible with a
> standard IOMMU, but I'm trying to figure out just that.

Does the PCI EP controller also wired to IOMMU?
> 
> I'm tempted to try to glue together the EP framework with VFIO for the
> purpose.

That would help to create a user space endpoint function driver. Not sure if
it'll help for the doorbell.
> 
> Are you aware of any efforts along those lines?

Most of the efforts in EP right now are towards binding the epf to virtio.
https://lore.kernel.org/linux-pci/20190905161516.2845-1-haotian.wang@sifive.com/T/#m4092f14a49852425a00a9a9afa80b4d3b1b836d1

Thanks
Kishon

Re: Exploring the PCI EP Framework

[Ismael Luceno Cortes]

I'm adding Patrik, who was interested in the thread.

On 13/Nov/2019 15:36, Kishon Vijay Abraham I wrote:
<...> 
> The endpoint relies on polling a shared memory to detect any events. However if
> the HW supports some sort of memory signaled interrupt (similar to what is
> usually available in systems with RC) then we could use it for interrupting the
> endpoint system. All we have to do is map the MSI address to a EP BAR region so
> that host can write to a mapped BAR in order to raise MSI interrupt.

I was proposing exactly that internally here, but I'm not 100% happy
with having the RC-side driver trigger the interrupts this way as we
need to expose a hint about "how", but it's probably the only safe bet
(we know the endpoints in question will always support MSI-X).

> Even if the HW supports, the EP framework doesn't support to use MSI. I'll have
> to see how that can be added.

Thanks for the confirmation. Where's the polling happening? I can take a
look at that.

Patrik: Can you check if we could publish that part of our docs? It
would involve only the signalling.

> > I realize it may not exist because it would imply not yet existing
> > mechanisms. I even don't know if such a thing is feasible with a
> > standard IOMMU, but I'm trying to figure out just that.
> 
> Does the PCI EP controller also wired to IOMMU?

No, I'm just daydreaming: a programmable IOMMU could implement magic
pages which produce rewrites, hooked to the PPR mechanics. Even if it's
not all hardwired, I don't think I could get to play with that.

> > I'm tempted to try to glue together the EP framework with VFIO for the
> > purpose.
> 
> That would help to create a user space endpoint function driver. Not sure if
> it'll help for the doorbell.

Oh, now that I read it again, it's confusing, I meant I would like to
tie EP Framework with VFIO somehow, to implement the EPFs in user-space;
completely unrelated to the previous comments.

> > 
> > Are you aware of any efforts along those lines?
> 
> Most of the efforts in EP right now are towards binding the epf to virtio.
> https://lore.kernel.org/linux-pci/20190905161516.2845-1-haotian.wang@sifive.com/T/#m4092f14a49852425a00a9a9afa80b4d3b1b836d1

Very interesting, I didn't hit any limitations because our EPC driver is
a hack, but we're going to go with the EP Framework in the end, so good
to know (plus we're using virtio already in pretty much the same way).

Thank you very much.

Best regards.

-- 
Ismael Luceno <ismael.luceno@silicon-gears.com>
+34 934452260 ext. 139
Travessera de les Corts 302 Bajos, 08029 Barcelona, Spain

https://www.silicon-gears.com/ - https://www.tttech-auto.com/