From: Kishon Vijay Abraham I <kishon@ti.com>
To: "Lad, Prabhakar" <prabhakar.csengg@gmail.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
linux-pci <linux-pci@vger.kernel.org>
Subject: Re: PCIe EPF
Date: Mon, 30 Mar 2020 18:50:59 +0530 [thread overview]
Message-ID: <e34a54f2-af3a-b760-c7d2-1da836e8fb4d@ti.com> (raw)
In-Reply-To: <CA+V-a8uxAD5-BovZPrKi_a6DPJVJPpez4V45C7YY-Rh3QjN8ag@mail.gmail.com>
Prabhakar,
On 3/30/2020 6:39 PM, Lad, Prabhakar wrote:
> Hi Kishon,
>
> On Mon, Mar 30, 2020 at 12:59 PM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>
>> Hi Prabhakar,
>>
>> On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
>>> Hi Kishon,
>>>
>>> On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
>>> <prabhakar.csengg@gmail.com> wrote:
>>>>
>>>> Hi Kishon,
>>>>
>>>> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
>>>> <prabhakar.csengg@gmail.com> wrote:
>>>>>
>>>>> Hi Kishon,
>>>>>
>>>>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>
>>>>>> Hi Prabhakar,
>>>>>>
>>>>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
>>>>>>> Hi Kishon,
>>>>>>>
>>>>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>>>
>>>>>>>> Hi Prabhakar,
>>>>>>>>
>>>>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>>>>>>>> Hi Kishon,
>>>>>>>>>
>>>>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>>>>>>>> support for streaming DMA API support, with this read/write/copy
>>>>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
>>>>>>>>> reverted the streaming DMA API patch
>>>>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>>>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>>>>>>>> again.
>>>>>>>>>
>>>>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>>>>>>>> for read/write/copy pass as expected.
>>>>>>>>>
>>>>>>>>> Could you please through some light why this could be happening.
>>>>>>>>
>>>>>>>> Do you see any differences in the address returned by dma_map_single() like is
>>>>>>>> it 32-bit address or 64-bit address?
>>>>>>>>
>>>>>>> Both return 32 bit address, debugging further I see that with
>>>>>>> GFP_KERNEL flag for small buffer
>>>>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
>>>>>>> related to caching probably.
>>>>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
>>>>>>> am using PIO transfers.
>>>>>>> Any thoughts on how we tackle it ?
>>>>>>>
>>>>>>> # With GFP_KERNEL flag
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): OKAY
>>>>>>
>>>>>> Here one of the read test is passing and the other is failing.
>>>>>> For the 1st case dma:7e99d000, address is aligned to 4K
>>>>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
>>>>>>
>>>>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
>>>>>> EP has any restrictions? (like the address should be aligned to the size?).
>>>>>>
>>>>> There isn't any restriction for outbound ATU on ep, Although I tried
>>>>> alignment from
>>>>> SZ_1 - SZ_256K and each failed at several points.
>>>>>
>>>>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
>>>>> is not aligned to 256 but still read passes.
>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>> READ ( 16384 bytes): OKAY
>>>>>
>>>>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
>>>>> everything works as expected,
>>>>>
>>>>> So how about a patch for pci_endpoint_test.c, where flags are passed
>>>>> as part of driver_data and it defaults to just GFP_KERNEL ?
>>>>>
>>>> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
>>>>
>>> Correct me if I am wrong here, streaming DMA API should be used with
>>> dma (-d) option so that root device
>>> makes sure the data is synced when data is transferred whereas
>>> previously with dma_alloc_coherent()
>>> we didn't have to care about cache issues. Also for a non-dma (-d)
>>> option we don't have a handle to dma
>>> in rootpport device so that we can call a sync operation. I say this
>>> because on my platform with streaming
>>> DMA api it works for small size buffers but it doesn't work with large
>>> size buffers.
>>
>> Streaming DMA API and DMA support in endpoint can be treated independently.
>> dma_alloc_coherent() will give you coherent memory, so you don't have to flush
>> or invalidate. This memory is usually limited in a platform.
>> The other option was to use streaming DMA APIs which doesn't give coherent
>> memory but SW has to take care of flush and invalidate.
>>
> Agreed. But we don't flush in SW when -d option is not specified I am
> assuming when we us
> -d dma engine takes care of flushing it.
The -d option switch doesn't change anything on the SW that runs on the host
side (misc/pci-endpoint-test.c). That only tells the EP to use DMA.
When you use streaming APIs, dma_map_single(), dmap_unmap_single() takes care
of flushing or invalidating memory based on the platform. (Platforms which have
coherent memory will have dma-coherent property,
dma_map_single()/dmap_unmap_single() will not do flush or invalidate.
Did you try to probe the failure further by comparing the hexdumps? Where does
the mismatch happen?
>
>>>
>>> Could you please confirm with streaming DMA api without DMA (-d)
>>> option for large buffers read/write/copy
>>> still passes for you.
>>
>> root@j7-evm:~# ./pcitest -r
>> READ ( 102400 bytes): OKAY
>> root@j7-evm:~# ./pcitest -r -s 1024000
>> READ (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -w -s 1024000
>> WRITE (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -c -s 1024000
>> COPY (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -c -s 10240000
>> COPY (10240000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -r -s 10240000
>> READ (10240000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -w -s 10240000
>> WRITE (10240000 bytes): OKAY
>
> Thank you for testing is this on Jacinto ?
>
>>>
>>> Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
>>> kzalloc on my platform fixes everything.
>>
>> Which host do you use? If this is only a host side limitation, you could try
>> using a different host.
>>
> I am trying this on Renesas RZ/G2N as host and RZ/G2E as an endpoint.
> ATM I can only test this on
> Renesas platforms only and all of them have same PCIe controller :(
Does the host controller work well with other standard PCIe cards (NVMe, USB
etc.,)?
>
> How about adding flags as part of driver data and defaulting it GFP_KERNEL ?
Nope. Not before we clearly understand the reason for the failure.
Thanks
Kishon
>
> Cheers,
> --Prabhakar
>
>> Thanks
>> Kishon
>>
>>>
>>> Cheers,
>>> --Prabhakar
>>>
>>>
>>>> Cheers,
>>>> --Prabhakar
>>>>
>>>>> Cheers,
>>>>> --Prabhakar
>>>>>
>>>>>> Thanks
>>>>>> Kishon
>>>>>>
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>>
>>>>>>> # GFP_KERNEL | GFP_DMA
>>>>>>>
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
>>>>>>> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
>>>>>>> READ (1024001 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>>>> READ ( 16384 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
>>>>>>> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
>>>>>>> READ ( 8192 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 128
>>>>>>> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
>>>>>>> READ ( 128 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~#
>>>>>>>
>>>>>>> Cheers,
>>>>>>> --Prabhakar
>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Kishon
next prev parent reply other threads:[~2020-03-30 13:21 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-18 11:37 PCIe EPF Lad, Prabhakar
2020-03-20 5:28 ` Kishon Vijay Abraham I
2020-03-21 22:49 ` Lad, Prabhakar
2020-03-24 1:58 ` Kishon Vijay Abraham I
2020-03-24 14:41 ` Lad, Prabhakar
2020-03-28 18:44 ` Lad, Prabhakar
2020-03-29 14:04 ` Lad, Prabhakar
2020-03-30 11:59 ` Kishon Vijay Abraham I
2020-03-30 13:09 ` Lad, Prabhakar
2020-03-30 13:20 ` Kishon Vijay Abraham I [this message]
2020-03-30 13:46 ` Lad, Prabhakar
2020-03-31 7:10 ` Yoshihiro Shimoda
2020-03-31 17:25 ` Lad, Prabhakar
2020-04-01 1:25 ` Yoshihiro Shimoda
2020-04-01 8:00 ` Lad, Prabhakar
2020-04-01 8:50 ` Yoshihiro Shimoda
2020-04-01 9:24 ` Lad, Prabhakar
2020-04-02 1:22 ` Yoshihiro Shimoda
2020-04-02 4:58 ` Yoshihiro Shimoda
2020-04-02 17:44 ` Lad, Prabhakar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e34a54f2-af3a-b760-c7d2-1da836e8fb4d@ti.com \
--to=kishon@ti.com \
--cc=linux-pci@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=prabhakar.csengg@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).