Re: PCIe EPF

From: Kishon Vijay Abraham I <kishon@ti.com>
To: "Lad, Prabhakar" <prabhakar.csengg@gmail.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	linux-pci <linux-pci@vger.kernel.org>
Subject: Re: PCIe EPF
Date: Tue, 24 Mar 2020 07:28:21 +0530	[thread overview]
Message-ID: <83024641-7bd3-b47f-cd2c-0d831279086d@ti.com> (raw)
In-Reply-To: <CA+V-a8ssdO9R_wHbJM8RinzP5d7YX5KWES20G-TV0XnCx4SUeA@mail.gmail.com>

Hi Prabhakar,

On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> Hi Kishon,
> 
> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>
>> Hi Prabhakar,
>>
>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>> Hi Kishon,
>>>
>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>> support for streaming DMA API support, with this  read/write/copy
>>> tests failed, to make sure nothing hasn't changed on my driver I
>>> reverted the streaming DMA API patch
>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>> again.
>>>
>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>> for read/write/copy pass as expected.
>>>
>>> Could you please through some light why this could be happening.
>>
>> Do you see any differences in the address returned by dma_map_single() like is
>> it 32-bit address or 64-bit address?
>>
> Both return 32 bit address, debugging further I see that with
> GFP_KERNEL flag for small buffer
> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> related to caching probably.
> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> am using PIO transfers.
> Any thoughts on how we tackle it ?
> 
> # With GFP_KERNEL flag
> root@hihope-rzg2m:~# pcitest -r
> [   46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> READ ( 102400 bytes):           NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [   51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> READ ( 102400 bytes):           OKAY

Here one of the read test is passing and the other is failing.
For the 1st case dma:7e99d000, address is aligned to 4K
For the 2nd case dma:7e9c0000, address is aligned to 256K

I'm suspecting this could be an alignment issue. Does the outbound ATU of your
EP has any restrictions? (like the address should be aligned to the size?).

Thanks
Kishon

> root@hihope-rzg2m:~# pcitest -r
> [   53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> READ ( 102400 bytes):           NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [   55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> READ ( 102400 bytes):           NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [   57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> READ ( 102400 bytes):           NOT OKAY
> 
> # GFP_KERNEL | GFP_DMA
> 
> root@hihope-rzg2m:~# pcitest -r -s 1024001
> [  174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> READ (1024001 bytes):           OKAY
> root@hihope-rzg2m:~# pcitest -r -s 16384
> [  186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> READ (  16384 bytes):           OKAY
> root@hihope-rzg2m:~# pcitest -r -s 8192
> [  190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> READ (   8192 bytes):           OKAY
> root@hihope-rzg2m:~# pcitest -r -s 128
> [  199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> READ (    128 bytes):           OKAY
> root@hihope-rzg2m:~#
> 
> Cheers,
> --Prabhakar
> 
>> Thanks
>> Kishon