RE: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell

[Frank Li <frank.li@nxp.com>]

> On Thu, Nov 24 2022 at 00:50, Frank Li wrote:
> > ┌────────────┐   ┌───────────────
> ────────────────────┐   ┌────────
> ────────┐
> > │            │   │                                   │   │                │
> > │            │   │ PCI Endpoint                      │   │ PCI Host       │
> > │            │   │                                   │   │                │
> > │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> > │            │   │                                   │   │                │
> > │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>
> │
> > │ Controller │   │   update doorbell register address│   │                │
> > │            │   │   for BAR                         │   │                │
> > │            │   │                                   │   │ 3. Write BAR<n>│
> > │            │◄──┼──────────────────────
> ─────────────┼───┤                │
> > │            │   │                                   │   │                │
> > │            ├──►│ 4.Irq Handle                      │   │                │
> > │            │   │                                   │   │                │
> > │            │   │                                   │   │                │
> > └────────────┘   └───────────────
> ────────────────────┘   └────────
> ────────┘
> >
> > Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
> 
> Can you please explain what the MSI controller is in this picture? MSI
> controller is not a term which is common in the interrupt handling
> landscape despite the fact that it's pretty wide spread in device tree
> bindings presumably through intensive copy & pasta cargo cult.
> 
> > Basic working follow as
> > 1. EP function driver call platform_msi_domain_alloc_irqs() alloc a
> > MSI irq from MSI controller with call back function write_msi_msg();
> > 2. write_msg_msg will config BAR and map to address defined in msi_msg;
> > 3. Host side trigger an IRQ at Endpoint by write to BAR region.
> 
> You're explaining what the code does, but fail to explain the underlying
> mechanisms.
> 
> Platform MSI is definitely the wrong mechanism here. Why?
> 
> This is about a PCIe endpoint, which is usually handled by a PCI/MSI
> interrupt domain. Obviously this usage does not fit into the way how the
> "global" PCI/MSI domains work.
> 
> There is upcoming work and at least the generic parts should show up in
> 6.2 which addresses exactly the problem you are trying to solve:
>
@tglx

I finally get a platform, ls1028, which use gic its and also support PCIe EP function.
I port my code into linux-next, (next-20230105) and understand your comments.

Its need a device ID to allocate msi irq for each devices.  

Epf device is dynamic created, where are field epc,  epc.parent point to PCIe EP
Controller, which have been probed by device-tree. 

I use a workaround to get msi irq number. 
	dev->of_node=  ntb->efp->epc->dev.parent->of_node;
	Platform_msi_domain_alloc_irqs(dev, ...);

Irq-gic-v3-its-pci-msi use of_node to get a ID. 

You said PCI/IMS can resolve this problem.  But I still not figure out it yet.
Can you give me sample code, which use PCI/IMS or provide a pseudo code?

Frank 

> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221124225331.464480443%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Q8jreVGG
> La2M4yhjGO7Njqwdm59XDC0GyLEwkr0k6B0%3D&amp;reserved=0
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221124230505.073418677%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Tc9pXNJ49
> 9ETFgNWQBNLViFk8D5GbvrrwYDlBW%2Bf2qg%3D&amp;reserved=0
> 
> plus the prove that the platform MSI mess can be replaced by this:
> 
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
> ernel.org%2Fall%2F20221121135653.208611233%40linutronix.de&amp;data=0
> 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C
> 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=R5KNVfcG
> qxoCam%2FYhY57ihsloWGhGLM3Kh9IkyME4lk%3D&amp;reserved=0
> 
> NTB in it's current form should never have happened, but that's water
> down the bridge.
> 
> What you really want is:
> 
>   1) Convert your platform to the new MSI parent model
> 
>   2) Utilize PCI/IMS which is giving you exactly what you need with
>      proper PCI semantics
> 
> Thanks,
> 
>         tglx