Re: PCIe EPF

["Lad, Prabhakar" <prabhakar.csengg@gmail.com>]

Hi Kishon,

On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
<prabhakar.csengg@gmail.com> wrote:
>
> Hi Kishon,
>
> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >
> > Hi Prabhakar,
> >
> > On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> > > Hi Kishon,
> > >
> > > On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> > >>
> > >> Hi Prabhakar,
> > >>
> > >> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> > >>> Hi Kishon,
> > >>>
> > >>> I rebased my rcar-endpoint patches on endpoint branch, which has
> > >>> support for streaming DMA API support, with this  read/write/copy
> > >>> tests failed, to make sure nothing hasn't changed on my driver I
> > >>> reverted the streaming DMA API patch
> > >>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> > >>> streaming DMA APIs for buffer allocation" and tests began to pass
> > >>> again.
> > >>>
> > >>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> > >>> for read/write/copy pass as expected.
> > >>>
> > >>> Could you please through some light why this could be happening.
> > >>
> > >> Do you see any differences in the address returned by dma_map_single() like is
> > >> it 32-bit address or 64-bit address?
> > >>
> > > Both return 32 bit address, debugging further I see that with
> > > GFP_KERNEL flag for small buffer
> > > sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> > > related to caching probably.
> > > Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> > > am using PIO transfers.
> > > Any thoughts on how we tackle it ?
> > >
> > > # With GFP_KERNEL flag
> > > root@hihope-rzg2m:~# pcitest -r
> > > [   46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes):           NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [   51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes):           OKAY
> >
> > Here one of the read test is passing and the other is failing.
> > For the 1st case dma:7e99d000, address is aligned to 4K
> > For the 2nd case dma:7e9c0000, address is aligned to 256K
> >
> > I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> > EP has any restrictions? (like the address should be aligned to the size?).
> >
> There isn't any  restriction for outbound ATU on ep,  Although I tried
> alignment from
> SZ_1 - SZ_256K and each failed at several points.
>
> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
> is not aligned to 256 but still read passes.
> root@hihope-rzg2m:~# pcitest -r -s 16384
>  [  186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> READ (  16384 bytes):           OKAY
>
> And I have verified with GFP_KERNEL | GFP_DMA on my platform
> everything works as expected,
>
> So how about a patch for pci_endpoint_test.c, where flags are passed
> as  part of driver_data and it defaults to just GFP_KERNEL ?
>
Any thoughts on the above ? I intended to get the endpoint driver for v5.7.

Cheers,
--Prabhakar

> Cheers,
> --Prabhakar
>
> > Thanks
> > Kishon
> >
> > > root@hihope-rzg2m:~# pcitest -r
> > > [   53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes):           NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [   55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes):           NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [   57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes):           NOT OKAY
> > >
> > > # GFP_KERNEL | GFP_DMA
> > >
> > > root@hihope-rzg2m:~# pcitest -r -s 1024001
> > > [  174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> > > READ (1024001 bytes):           OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 16384
> > > [  186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> > > READ (  16384 bytes):           OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 8192
> > > [  190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> > > READ (   8192 bytes):           OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 128
> > > [  199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> > > READ (    128 bytes):           OKAY
> > > root@hihope-rzg2m:~#
> > >
> > > Cheers,
> > > --Prabhakar
> > >
> > >> Thanks
> > >> Kishon