RE: [PATCH v7 15/18] NTB: Add support for EPF PCI-Express Non-Transparent Bridge

[Sherry Sun <sherry.sun@nxp.com>]

Hi Kishon,

> Subject: Re: [PATCH v7 15/18] NTB: Add support for EPF PCI-Express Non-
> Transparent Bridge
> 
> Hi Sherry,
> 
> On 11/11/20 8:19 am, Sherry Sun wrote:
> > Hi Kishon,
> >
> >> Subject: Re: [PATCH v7 15/18] NTB: Add support for EPF PCI-Express
> >> Non- Transparent Bridge
> >>
> >> Hi Sherry, Arnd,
> >>
> >> On 10/11/20 8:29 pm, Arnd Bergmann wrote:
> >>> On Tue, Nov 10, 2020 at 3:20 PM Kishon Vijay Abraham I
> >>> <kishon@ti.com>
> >> wrote:
> >>>> On 10/11/20 7:55 am, Sherry Sun wrote:
> >>>
> >>>>> But for VOP, only two boards are needed(one board as host and one
> >>>>> board as card) to realize the communication between the two
> >>>>> systems,
> >> so my question is what are the advantages of using NTB?
> >>>>
> >>>> NTB is a bridge that facilitates communication between two
> >>>> different systems. So it by itself will not be source or sink of
> >>>> any data unlike a normal EP to RP system (or the VOP) which will be
> >>>> source or sink
> >> of data.
> >>>>
> >>>>> Because I think the architecture of NTB seems more complicated.
> >>>>> Many
> >> thanks!
> >>>>
> >>>> yeah, I think it enables a different use case all together.
> >>>> Consider you have two x86 HOST PCs (having RP) and they have to be
> >>>> communicate using PCIe. NTB can be used in such cases for the two
> >>>> x86 PCs to communicate with each other over PCIe, which wouldn't be
> >>>> possible
> >> without NTB.
> >>>
> >>> I think for VOP, we should have an abstraction that can work on
> >>> either NTB or directly on the endpoint framework but provide an
> >>> interface that then lets you create logical devices the same way.
> >>>
> >>> Doing VOP based on NTB plus the new NTB_EPF driver would also work
> >>> and just move the abstraction somewhere else, but I guess it would
> >>> complicate setting it up for those users that only care about the
> >>> simpler endpoint case.
> >>
> >> I'm not sure if you've got a chance to look at [1], where I added
> >> support for RP<->EP system both running Linux, with EP configured
> >> using Linux EP framework (as well as HOST ports connected to NTB
> >> switch, patches 20 and 21, that uses the Linux NTB framework) to
> >> communicate using virtio over PCIe.
> >>
> >
> > I saw your patches at [1], here you take a rpmsg as an example to
> > communicate between two SoCs using PCIe RC<->EP and HOST1-NTB-
> HOST2 for different usercases.
> > The VOP code works under the PCIe RC<->EP framework, which means that
> > we can also make VOP works under the Linux NTB framework, just like the
> rpmsg way you did here, right?
> 
> Does VOP really work with EP framework? At-least whatever is in upstream
> doesn't seem to indicate so.
> 

We did write a pci_epf driver to support VOP, looks like pci-epf-test.c, and it works well.
So certainly VOP can work with EP framework.
But it's a pity that the VOP related codes has been deleted before we send the pci_epf_vop driver patches to upstream.

> The NTB framework lets one host with RP port to communicate with another
> host with RP port.
> 
> The EP Framework lets one device with EP port to communicate with a host
> with RP port.
> 
> Rest of the trick should be how you tie them together.
> 
> PCIe framework creates "pci_device" for each of the devices it enumerates.
> NTB framework works on this pci_device to communicate with the remote
> host using PCIe bridge. The remote host will use NTB framework as well.
> 
> So depends on what interfaces VOP device provides you can use either NTB
> framework or EP framework. If it's going to connect two different devices in
> turn creating pci_device on each of the systems, then you can use NTB
> framework.
> 

Thanks for your detailed explanation! It is clear.
I think maybe VOP is suitable for the basic PCIe framework instead of NTB.

Best regards
Sherry

> Regards
> Kishon