RE: Linux PCIe EP NTB function

RE: Linux PCIe EP NTB function

[Frank Li]

Sorry, correct Linux-pci mail list address.

> -----Original Message-----
> From: Frank Li
> Sent: Thursday, October 14, 2021 3:35 PM
> To: kishon@ti.com; kw@linux.com; Sherry Sun <sherry.sun@nxp.com>; Richard
> Zhu <hongxing.zhu@nxp.com>
> Cc: inux-pci@vger.kernel.org
> Subject: Linux PCIe EP NTB function
> 
> Kishon:
> 
> 	We use VOP(virtio over PCIe) communication between PCI RC and EP side.
> But VOP already removed and switch into NTB solution.
> 	I saw you submit patch and already accepted by community about pci-
> epf-ntb.
> 
> 	According to document, whole system work as Device1 (PCI host) -> EP1
> -> EP2 -> Device2(PCI host).
> 	But our user case is Device 1(RC Host) ->  Device 2(EP).
> 
> 	I am not sure how to apply ntb frame work into this user case.
> 
> 	I think we can modify pci-epf-ntb to register a ntb devices. But I am
> not sure that this is recommunicate method.  I think this user case is
> quite common. I don't want to implement a local solution.
> 
> 	Any suggestion?
> 
> 
> Best regards
> Frank Li

Re: Linux PCIe EP NTB function

[Kishon Vijay Abraham I]

Hi Frank,

On 15/10/21 2:10 am, Frank Li wrote:
> Sorry, correct Linux-pci mail list address.
> 
>> -----Original Message-----
>> From: Frank Li
>> Sent: Thursday, October 14, 2021 3:35 PM
>> To: kishon@ti.com; kw@linux.com; Sherry Sun <sherry.sun@nxp.com>; Richard
>> Zhu <hongxing.zhu@nxp.com>
>> Cc: inux-pci@vger.kernel.org
>> Subject: Linux PCIe EP NTB function
>>
>> Kishon:
>>
>> 	We use VOP(virtio over PCIe) communication between PCI RC and EP side.
>> But VOP already removed and switch into NTB solution.
>> 	I saw you submit patch and already accepted by community about pci-
>> epf-ntb.
>>
>> 	According to document, whole system work as Device1 (PCI host) -> EP1
>> -> EP2 -> Device2(PCI host).
>> 	But our user case is Device 1(RC Host) ->  Device 2(EP).
>>
>> 	I am not sure how to apply ntb frame work into this user case.

NTB by definition is PCIe RC-to-RC communication. For RC-to-EP, see
pci_endpoint_test.c (RC) and pci-epf-test.c (EP) that provides sample endpoint
usecase.

One more RC<->EP communication model was built in [1], but that is not yet
upstreamed.

[1] -> https://lore.kernel.org/r/20200702082143.25259-1-kishon@ti.com

Thanks,
Kishon

>>
>> 	I think we can modify pci-epf-ntb to register a ntb devices. But I am
>> not sure that this is recommunicate method.  I think this user case is
>> quite common. I don't want to implement a local solution.
>>
>> 	Any suggestion?
>>
>>
>> Best regards
>> Frank Li

RE: [EXT] Re: Linux PCIe EP NTB function

[Frank Li]

> -----Original Message-----
> From: Kishon Vijay Abraham I <kishon@ti.com>
> Sent: Sunday, October 17, 2021 11:18 PM
> To: Frank Li <frank.li@nxp.com>; kw@linux.com; Sherry Sun
> <sherry.sun@nxp.com>; Richard Zhu <hongxing.zhu@nxp.com>
> Cc: linux-pci@vger.kernel.org
> Subject: [EXT] Re: Linux PCIe EP NTB function
> 
> Caution: EXT Email
> 
> Hi Frank,
> 
> On 15/10/21 2:10 am, Frank Li wrote:
> > Sorry, correct Linux-pci mail list address.
> >
> >> -----Original Message-----
> >> From: Frank Li
> >> Sent: Thursday, October 14, 2021 3:35 PM
> >> To: kishon@ti.com; kw@linux.com; Sherry Sun <sherry.sun@nxp.com>;
> Richard
> >> Zhu <hongxing.zhu@nxp.com>
> >> Cc: inux-pci@vger.kernel.org
> >> Subject: Linux PCIe EP NTB function
> >>
> >> Kishon:
> >>
> >>      We use VOP(virtio over PCIe) communication between PCI RC and EP
> side.
> >> But VOP already removed and switch into NTB solution.
> >>      I saw you submit patch and already accepted by community about pci-
> >> epf-ntb.
> >>
> >>      According to document, whole system work as Device1 (PCI host) ->
> EP1
> >> -> EP2 -> Device2(PCI host).
> >>      But our user case is Device 1(RC Host) ->  Device 2(EP).
> >>
> >>      I am not sure how to apply ntb frame work into this user case.
> 
> NTB by definition is PCIe RC-to-RC communication. For RC-to-EP, see
> pci_endpoint_test.c (RC) and pci-epf-test.c (EP) that provides sample
> endpoint
> usecase.
> 
> One more RC<->EP communication model was built in [1], but that is not yet
> upstreamed.

Thank you very much. We already can run pci_endpoint_test, which
is too basic and just proof data can transfer between RC and EP.

We want to hook into standard interface such as net. I also find below patches by
Google. The patch set was sent out last years. What's reason why not continues such
Work. I just saw NTB solution is acceptable by community.

Previously we were working on VOP solution, kernel dropped VOP just when we prepare upstream
our work. We don't want to these happen again. So I want to check RC-EP communication direction.

So we can work at the same directions.   

Best regards
Frank Li

> 
> [1] ->
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kern
> el.org%2Fr%2F20200702082143.25259-1-
> kishon%40ti.com&amp;data=04%7C01%7Cfrank.li%40nxp.com%7C09c9aaaba663411c1bd
> a08d991ee3cd4%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6377012746822016
> 79%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> aWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=kJkAPZHywJfv9gwj5MhI9BUAG2veDyouMcBH%2B
> rx8xWY%3D&amp;reserved=0
> 
> Thanks,
> Kishon
> 
> >>
> >>      I think we can modify pci-epf-ntb to register a ntb devices. But I
> am
> >> not sure that this is recommunicate method.  I think this user case is
> >> quite common. I don't want to implement a local solution.
> >>
> >>      Any suggestion?
> >>
> >>
> >> Best regards
> >> Frank Li

Re: [EXT] Re: Linux PCIe EP NTB function

[Kishon Vijay Abraham I]

Hi Frank,

On 18/10/21 10:40 pm, Frank Li wrote:
> 
> 
>> -----Original Message-----
>> From: Kishon Vijay Abraham I <kishon@ti.com>
>> Sent: Sunday, October 17, 2021 11:18 PM
>> To: Frank Li <frank.li@nxp.com>; kw@linux.com; Sherry Sun
>> <sherry.sun@nxp.com>; Richard Zhu <hongxing.zhu@nxp.com>
>> Cc: linux-pci@vger.kernel.org
>> Subject: [EXT] Re: Linux PCIe EP NTB function
>>
>> Caution: EXT Email
>>
>> Hi Frank,
>>
>> On 15/10/21 2:10 am, Frank Li wrote:
>>> Sorry, correct Linux-pci mail list address.
>>>
>>>> -----Original Message-----
>>>> From: Frank Li
>>>> Sent: Thursday, October 14, 2021 3:35 PM
>>>> To: kishon@ti.com; kw@linux.com; Sherry Sun <sherry.sun@nxp.com>;
>> Richard
>>>> Zhu <hongxing.zhu@nxp.com>
>>>> Cc: inux-pci@vger.kernel.org
>>>> Subject: Linux PCIe EP NTB function
>>>>
>>>> Kishon:
>>>>
>>>>      We use VOP(virtio over PCIe) communication between PCI RC and EP
>> side.
>>>> But VOP already removed and switch into NTB solution.
>>>>      I saw you submit patch and already accepted by community about pci-
>>>> epf-ntb.
>>>>
>>>>      According to document, whole system work as Device1 (PCI host) ->
>> EP1
>>>> -> EP2 -> Device2(PCI host).
>>>>      But our user case is Device 1(RC Host) ->  Device 2(EP).
>>>>
>>>>      I am not sure how to apply ntb frame work into this user case.
>>
>> NTB by definition is PCIe RC-to-RC communication. For RC-to-EP, see
>> pci_endpoint_test.c (RC) and pci-epf-test.c (EP) that provides sample
>> endpoint
>> usecase.
>>
>> One more RC<->EP communication model was built in [1], but that is not yet
>> upstreamed.
> 
> Thank you very much. We already can run pci_endpoint_test, which
> is too basic and just proof data can transfer between RC and EP.
> 
> We want to hook into standard interface such as net. I also find below patches by
> Google. The patch set was sent out last years. What's reason why not continues such
> Work. I just saw NTB solution is acceptable by community.

I didn't have the bandwidth to pursue it. So the expectation was to update the
Virtio Spec to standardize PCIe RC to Configurable PCIe EP and then continue on
the patch series.

NTB provides a PCIe host side interface which cannot be used for PCIe endpoint.

Thanks,
Kishon

> 
> Previously we were working on VOP solution, kernel dropped VOP just when we prepare upstream
> our work. We don't want to these happen again. So I want to check RC-EP communication direction.
> 
> So we can work at the same directions.   
> 
> Best regards
> Frank Li
> 
>>
>> [1] ->
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kern
>> el.org%2Fr%2F20200702082143.25259-1-
>> kishon%40ti.com&amp;data=04%7C01%7Cfrank.li%40nxp.com%7C09c9aaaba663411c1bd
>> a08d991ee3cd4%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6377012746822016
>> 79%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
>> aWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=kJkAPZHywJfv9gwj5MhI9BUAG2veDyouMcBH%2B
>> rx8xWY%3D&amp;reserved=0
>>
>> Thanks,
>> Kishon
>>
>>>>
>>>>      I think we can modify pci-epf-ntb to register a ntb devices. But I
>> am
>>>> not sure that this is recommunicate method.  I think this user case is
>>>> quite common. I don't want to implement a local solution.
>>>>
>>>>      Any suggestion?
>>>>
>>>>
>>>> Best regards
>>>> Frank Li