Re: [PATCH V2 0/4] NTB function for PCIe RC to EP connection

[Zhi Li <lznuaa@gmail.com>]

On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Frank Li,
>
> On 22/02/22 9:53 pm, Frank Li wrote:
> > This implement NTB function for PCIe EP to RC connections.
> > The existed ntb epf need two PCI EPs and two PCI Host.
>
> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
> layer instead of trying to build on NTB layer (which is specific to RC<->RC
> communication).
>
> Are there any specific reasons for not taking that path?

1. EP side work as vHOST mode.  vHost suppose access all memory of virtual io.
But there are only map windows on the EP side to access RC side
memory. You have to move
map windows for each access.  It is quite low efficiency.

2. So far as I know, virtio is still not DMA yet.  CPU access PCI
can't generate longer PCI TLP,
So the speed is quite slow.  NTB already has DMA support.  If you use
system level DMA,
no change is needed at NTB level.  If we want to use a PCI controller
embedded DMA,  some small
changes need if based on my other Designware PCI eDMA patches, which
are under review.

3. All the major data transfer of NTB is using write.  Because TLP
write needn't wait for complete,  write
performance is better than reading.  On our platform,  write
performance is about 10% better than  read.

Frank

>
> Thanks,
> Kishon
>
> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
> >
> > This just need EP to RC connections.
> >
> >     ┌────────────┐         ┌─────────────────────────────────────┐
> >     │            │         │                                     │
> >     ├────────────┤         │                      ┌──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ NetDev     │         │                      │ NetDev       │
> >     ├────────────┤         │                      ├──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ Transfer   │         │                      │ Transfer     │
> >     ├────────────┤         │                      ├──────────────┤
> >     │            │         │                      │              │
> >     │  PCI NTB   │         │                      │              │
> >     │    EPF     │         │                      │              │
> >     │   Driver   │         │                      │ PCI Virtual  │
> >     │            │         ├───────────────┐      │ NTB Driver   │
> >     │            │         │ PCI EP NTB    │◄────►│              │
> >     │            │         │  FN Driver    │      │              │
> >     ├────────────┤         ├───────────────┤      ├──────────────┤
> >     │            │         │               │      │              │
> >     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> >     │            │  PCI    │               │      │     BUS      │
> >     └────────────┘         └───────────────┴──────┴──────────────┘
> >         PCI RC                        PCI EP
> >
> >
> >
> > Frank Li (4):
> >   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> >   NTB: epf: Allow more flexibility in the memory BAR map method
> >   PCI: endpoint: Support NTB transfer between RC and EP
> >   Documentation: PCI: Add specification for the PCI vNTB function device
> >
> >  Documentation/PCI/endpoint/index.rst          |    2 +
> >  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> >  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> >  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> >  drivers/pci/endpoint/functions/Kconfig        |   11 +
> >  drivers/pci/endpoint/functions/Makefile       |    1 +
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> >  8 files changed, 1775 insertions(+), 14 deletions(-)
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> >