Re: PCIe EPF

[Kishon Vijay Abraham I <kishon@ti.com>]

Hi Prabhakar,

On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
> Hi Kishon,
> 
> On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
> <prabhakar.csengg@gmail.com> wrote:
>>
>> Hi Kishon,
>>
>> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
>> <prabhakar.csengg@gmail.com> wrote:
>>>
>>> Hi Kishon,
>>>
>>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>
>>>> Hi Prabhakar,
>>>>
>>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
>>>>> Hi Kishon,
>>>>>
>>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>
>>>>>> Hi Prabhakar,
>>>>>>
>>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>>>>>> Hi Kishon,
>>>>>>>
>>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>>>>>> support for streaming DMA API support, with this  read/write/copy
>>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
>>>>>>> reverted the streaming DMA API patch
>>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>>>>>> again.
>>>>>>>
>>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>>>>>> for read/write/copy pass as expected.
>>>>>>>
>>>>>>> Could you please through some light why this could be happening.
>>>>>>
>>>>>> Do you see any differences in the address returned by dma_map_single() like is
>>>>>> it 32-bit address or 64-bit address?
>>>>>>
>>>>> Both return 32 bit address, debugging further I see that with
>>>>> GFP_KERNEL flag for small buffer
>>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
>>>>> related to caching probably.
>>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
>>>>> am using PIO transfers.
>>>>> Any thoughts on how we tackle it ?
>>>>>
>>>>> # With GFP_KERNEL flag
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [   46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes):           NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [   51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes):           OKAY
>>>>
>>>> Here one of the read test is passing and the other is failing.
>>>> For the 1st case dma:7e99d000, address is aligned to 4K
>>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
>>>>
>>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
>>>> EP has any restrictions? (like the address should be aligned to the size?).
>>>>
>>> There isn't any  restriction for outbound ATU on ep,  Although I tried
>>> alignment from
>>> SZ_1 - SZ_256K and each failed at several points.
>>>
>>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
>>> is not aligned to 256 but still read passes.
>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>  [  186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>> READ (  16384 bytes):           OKAY
>>>
>>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
>>> everything works as expected,
>>>
>>> So how about a patch for pci_endpoint_test.c, where flags are passed
>>> as  part of driver_data and it defaults to just GFP_KERNEL ?
>>>
>> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
>>
> Correct me if I am wrong here, streaming DMA API should be used with
> dma (-d) option so that root device
> makes sure the data is synced when data is transferred whereas
> previously with dma_alloc_coherent()
> we didn't have to care about cache issues. Also for a non-dma (-d)
> option we don't have a handle to dma
> in rootpport device so that we can call a sync operation. I say this
> because on my platform  with streaming
> DMA api it works for small size buffers but it doesn't work with large
> size buffers.

Streaming DMA API and DMA support in endpoint can be treated independently.
dma_alloc_coherent() will give you coherent memory, so you don't have to flush
or invalidate. This memory is usually limited in a platform.
The other option was to use streaming DMA APIs which doesn't give coherent
memory but SW has to take care of flush and invalidate.

> 
> Could you please confirm with streaming DMA api without DMA (-d)
> option for large buffers read/write/copy
> still passes for you.

root@j7-evm:~# ./pcitest -r
READ ( 102400 bytes):           OKAY
root@j7-evm:~# ./pcitest -r -s 1024000
READ (1024000 bytes):           OKAY
root@j7-evm:~# ./pcitest -w -s 1024000
WRITE (1024000 bytes):          OKAY
root@j7-evm:~# ./pcitest -c -s 1024000
COPY (1024000 bytes):           OKAY
root@j7-evm:~# ./pcitest -c -s 10240000
COPY (10240000 bytes):          OKAY
root@j7-evm:~# ./pcitest -r -s 10240000
READ (10240000 bytes):          OKAY
root@j7-evm:~# ./pcitest -w -s 10240000
WRITE (10240000 bytes):         OKAY
> 
> Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
> kzalloc  on my platform fixes everything.

Which host do you use? If this is only a host side limitation, you could try
using a different host.

Thanks
Kishon

> 
> Cheers,
> --Prabhakar
> 
> 
>> Cheers,
>> --Prabhakar
>>
>>> Cheers,
>>> --Prabhakar
>>>
>>>> Thanks
>>>> Kishon
>>>>
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [   53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes):           NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [   55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes):           NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [   57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes):           NOT OKAY
>>>>>
>>>>> # GFP_KERNEL | GFP_DMA
>>>>>
>>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
>>>>> [  174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
>>>>> READ (1024001 bytes):           OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>> [  186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>> READ (  16384 bytes):           OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
>>>>> [  190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
>>>>> READ (   8192 bytes):           OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 128
>>>>> [  199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
>>>>> READ (    128 bytes):           OKAY
>>>>> root@hihope-rzg2m:~#
>>>>>
>>>>> Cheers,
>>>>> --Prabhakar
>>>>>
>>>>>> Thanks
>>>>>> Kishon