* PCIe EPF
@ 2020-03-18 11:37 Lad, Prabhakar
2020-03-20 5:28 ` Kishon Vijay Abraham I
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-18 11:37 UTC (permalink / raw)
To: Kishon Vijay Abraham I; +Cc: Lorenzo Pieralisi, linux-pci
Hi Kishon,
I rebased my rcar-endpoint patches on endpoint branch, which has
support for streaming DMA API support, with this read/write/copy
tests failed, to make sure nothing hasn't changed on my driver I
reverted the streaming DMA API patch
74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
streaming DMA APIs for buffer allocation" and tests began to pass
again.
If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
for read/write/copy pass as expected.
Could you please through some light why this could be happening.
Cheers,
--Prabhakar Lad
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-18 11:37 PCIe EPF Lad, Prabhakar
@ 2020-03-20 5:28 ` Kishon Vijay Abraham I
2020-03-21 22:49 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Kishon Vijay Abraham I @ 2020-03-20 5:28 UTC (permalink / raw)
To: Lad, Prabhakar; +Cc: Lorenzo Pieralisi, linux-pci
Hi Prabhakar,
On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> Hi Kishon,
>
> I rebased my rcar-endpoint patches on endpoint branch, which has
> support for streaming DMA API support, with this read/write/copy
> tests failed, to make sure nothing hasn't changed on my driver I
> reverted the streaming DMA API patch
> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> streaming DMA APIs for buffer allocation" and tests began to pass
> again.
>
> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> for read/write/copy pass as expected.
>
> Could you please through some light why this could be happening.
Do you see any differences in the address returned by dma_map_single() like is
it 32-bit address or 64-bit address?
Thanks
Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-20 5:28 ` Kishon Vijay Abraham I
@ 2020-03-21 22:49 ` Lad, Prabhakar
2020-03-24 1:58 ` Kishon Vijay Abraham I
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-21 22:49 UTC (permalink / raw)
To: Kishon Vijay Abraham I; +Cc: Lorenzo Pieralisi, linux-pci
Hi Kishon,
On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Prabhakar,
>
> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> > Hi Kishon,
> >
> > I rebased my rcar-endpoint patches on endpoint branch, which has
> > support for streaming DMA API support, with this read/write/copy
> > tests failed, to make sure nothing hasn't changed on my driver I
> > reverted the streaming DMA API patch
> > 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> > streaming DMA APIs for buffer allocation" and tests began to pass
> > again.
> >
> > If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> > for read/write/copy pass as expected.
> >
> > Could you please through some light why this could be happening.
>
> Do you see any differences in the address returned by dma_map_single() like is
> it 32-bit address or 64-bit address?
>
Both return 32 bit address, debugging further I see that with
GFP_KERNEL flag for small buffer
sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
related to caching probably.
Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
am using PIO transfers.
Any thoughts on how we tackle it ?
# With GFP_KERNEL flag
root@hihope-rzg2m:~# pcitest -r
[ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
READ ( 102400 bytes): NOT OKAY
root@hihope-rzg2m:~# pcitest -r
[ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
READ ( 102400 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r
[ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
READ ( 102400 bytes): NOT OKAY
root@hihope-rzg2m:~# pcitest -r
[ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
READ ( 102400 bytes): NOT OKAY
root@hihope-rzg2m:~# pcitest -r
[ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
READ ( 102400 bytes): NOT OKAY
# GFP_KERNEL | GFP_DMA
root@hihope-rzg2m:~# pcitest -r -s 1024001
[ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
READ (1024001 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r -s 16384
[ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
READ ( 16384 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r -s 8192
[ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
READ ( 8192 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r -s 128
[ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
READ ( 128 bytes): OKAY
root@hihope-rzg2m:~#
Cheers,
--Prabhakar
> Thanks
> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-21 22:49 ` Lad, Prabhakar
@ 2020-03-24 1:58 ` Kishon Vijay Abraham I
2020-03-24 14:41 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Kishon Vijay Abraham I @ 2020-03-24 1:58 UTC (permalink / raw)
To: Lad, Prabhakar; +Cc: Lorenzo Pieralisi, linux-pci
Hi Prabhakar,
On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> Hi Kishon,
>
> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>
>> Hi Prabhakar,
>>
>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>> Hi Kishon,
>>>
>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>> support for streaming DMA API support, with this read/write/copy
>>> tests failed, to make sure nothing hasn't changed on my driver I
>>> reverted the streaming DMA API patch
>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>> again.
>>>
>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>> for read/write/copy pass as expected.
>>>
>>> Could you please through some light why this could be happening.
>>
>> Do you see any differences in the address returned by dma_map_single() like is
>> it 32-bit address or 64-bit address?
>>
> Both return 32 bit address, debugging further I see that with
> GFP_KERNEL flag for small buffer
> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> related to caching probably.
> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> am using PIO transfers.
> Any thoughts on how we tackle it ?
>
> # With GFP_KERNEL flag
> root@hihope-rzg2m:~# pcitest -r
> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> READ ( 102400 bytes): NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> READ ( 102400 bytes): OKAY
Here one of the read test is passing and the other is failing.
For the 1st case dma:7e99d000, address is aligned to 4K
For the 2nd case dma:7e9c0000, address is aligned to 256K
I'm suspecting this could be an alignment issue. Does the outbound ATU of your
EP has any restrictions? (like the address should be aligned to the size?).
Thanks
Kishon
> root@hihope-rzg2m:~# pcitest -r
> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> READ ( 102400 bytes): NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> READ ( 102400 bytes): NOT OKAY
> root@hihope-rzg2m:~# pcitest -r
> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> READ ( 102400 bytes): NOT OKAY
>
> # GFP_KERNEL | GFP_DMA
>
> root@hihope-rzg2m:~# pcitest -r -s 1024001
> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> READ (1024001 bytes): OKAY
> root@hihope-rzg2m:~# pcitest -r -s 16384
> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> READ ( 16384 bytes): OKAY
> root@hihope-rzg2m:~# pcitest -r -s 8192
> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> READ ( 8192 bytes): OKAY
> root@hihope-rzg2m:~# pcitest -r -s 128
> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> READ ( 128 bytes): OKAY
> root@hihope-rzg2m:~#
>
> Cheers,
> --Prabhakar
>
>> Thanks
>> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-24 1:58 ` Kishon Vijay Abraham I
@ 2020-03-24 14:41 ` Lad, Prabhakar
2020-03-28 18:44 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-24 14:41 UTC (permalink / raw)
To: Kishon Vijay Abraham I; +Cc: Lorenzo Pieralisi, linux-pci
Hi Kishon,
On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Prabhakar,
>
> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> > Hi Kishon,
> >
> > On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>
> >> Hi Prabhakar,
> >>
> >> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> >>> Hi Kishon,
> >>>
> >>> I rebased my rcar-endpoint patches on endpoint branch, which has
> >>> support for streaming DMA API support, with this read/write/copy
> >>> tests failed, to make sure nothing hasn't changed on my driver I
> >>> reverted the streaming DMA API patch
> >>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> >>> streaming DMA APIs for buffer allocation" and tests began to pass
> >>> again.
> >>>
> >>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> >>> for read/write/copy pass as expected.
> >>>
> >>> Could you please through some light why this could be happening.
> >>
> >> Do you see any differences in the address returned by dma_map_single() like is
> >> it 32-bit address or 64-bit address?
> >>
> > Both return 32 bit address, debugging further I see that with
> > GFP_KERNEL flag for small buffer
> > sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> > related to caching probably.
> > Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> > am using PIO transfers.
> > Any thoughts on how we tackle it ?
> >
> > # With GFP_KERNEL flag
> > root@hihope-rzg2m:~# pcitest -r
> > [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> > READ ( 102400 bytes): NOT OKAY
> > root@hihope-rzg2m:~# pcitest -r
> > [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> > READ ( 102400 bytes): OKAY
>
> Here one of the read test is passing and the other is failing.
> For the 1st case dma:7e99d000, address is aligned to 4K
> For the 2nd case dma:7e9c0000, address is aligned to 256K
>
> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> EP has any restrictions? (like the address should be aligned to the size?).
>
There isn't any restriction for outbound ATU on ep, Although I tried
alignment from
SZ_1 - SZ_256K and each failed at several points.
With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
is not aligned to 256 but still read passes.
root@hihope-rzg2m:~# pcitest -r -s 16384
[ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
READ ( 16384 bytes): OKAY
And I have verified with GFP_KERNEL | GFP_DMA on my platform
everything works as expected,
So how about a patch for pci_endpoint_test.c, where flags are passed
as part of driver_data and it defaults to just GFP_KERNEL ?
Cheers,
--Prabhakar
> Thanks
> Kishon
>
> > root@hihope-rzg2m:~# pcitest -r
> > [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> > READ ( 102400 bytes): NOT OKAY
> > root@hihope-rzg2m:~# pcitest -r
> > [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> > READ ( 102400 bytes): NOT OKAY
> > root@hihope-rzg2m:~# pcitest -r
> > [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> > READ ( 102400 bytes): NOT OKAY
> >
> > # GFP_KERNEL | GFP_DMA
> >
> > root@hihope-rzg2m:~# pcitest -r -s 1024001
> > [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> > READ (1024001 bytes): OKAY
> > root@hihope-rzg2m:~# pcitest -r -s 16384
> > [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> > READ ( 16384 bytes): OKAY
> > root@hihope-rzg2m:~# pcitest -r -s 8192
> > [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> > READ ( 8192 bytes): OKAY
> > root@hihope-rzg2m:~# pcitest -r -s 128
> > [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> > READ ( 128 bytes): OKAY
> > root@hihope-rzg2m:~#
> >
> > Cheers,
> > --Prabhakar
> >
> >> Thanks
> >> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-24 14:41 ` Lad, Prabhakar
@ 2020-03-28 18:44 ` Lad, Prabhakar
2020-03-29 14:04 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-28 18:44 UTC (permalink / raw)
To: Kishon Vijay Abraham I, Lorenzo Pieralisi; +Cc: linux-pci
Hi Kishon,
On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
<prabhakar.csengg@gmail.com> wrote:
>
> Hi Kishon,
>
> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >
> > Hi Prabhakar,
> >
> > On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> > > Hi Kishon,
> > >
> > > On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> > >>
> > >> Hi Prabhakar,
> > >>
> > >> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> > >>> Hi Kishon,
> > >>>
> > >>> I rebased my rcar-endpoint patches on endpoint branch, which has
> > >>> support for streaming DMA API support, with this read/write/copy
> > >>> tests failed, to make sure nothing hasn't changed on my driver I
> > >>> reverted the streaming DMA API patch
> > >>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> > >>> streaming DMA APIs for buffer allocation" and tests began to pass
> > >>> again.
> > >>>
> > >>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> > >>> for read/write/copy pass as expected.
> > >>>
> > >>> Could you please through some light why this could be happening.
> > >>
> > >> Do you see any differences in the address returned by dma_map_single() like is
> > >> it 32-bit address or 64-bit address?
> > >>
> > > Both return 32 bit address, debugging further I see that with
> > > GFP_KERNEL flag for small buffer
> > > sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> > > related to caching probably.
> > > Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> > > am using PIO transfers.
> > > Any thoughts on how we tackle it ?
> > >
> > > # With GFP_KERNEL flag
> > > root@hihope-rzg2m:~# pcitest -r
> > > [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes): NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes): OKAY
> >
> > Here one of the read test is passing and the other is failing.
> > For the 1st case dma:7e99d000, address is aligned to 4K
> > For the 2nd case dma:7e9c0000, address is aligned to 256K
> >
> > I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> > EP has any restrictions? (like the address should be aligned to the size?).
> >
> There isn't any restriction for outbound ATU on ep, Although I tried
> alignment from
> SZ_1 - SZ_256K and each failed at several points.
>
> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
> is not aligned to 256 but still read passes.
> root@hihope-rzg2m:~# pcitest -r -s 16384
> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> READ ( 16384 bytes): OKAY
>
> And I have verified with GFP_KERNEL | GFP_DMA on my platform
> everything works as expected,
>
> So how about a patch for pci_endpoint_test.c, where flags are passed
> as part of driver_data and it defaults to just GFP_KERNEL ?
>
Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
Cheers,
--Prabhakar
> Cheers,
> --Prabhakar
>
> > Thanks
> > Kishon
> >
> > > root@hihope-rzg2m:~# pcitest -r
> > > [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes): NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes): NOT OKAY
> > > root@hihope-rzg2m:~# pcitest -r
> > > [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> > > READ ( 102400 bytes): NOT OKAY
> > >
> > > # GFP_KERNEL | GFP_DMA
> > >
> > > root@hihope-rzg2m:~# pcitest -r -s 1024001
> > > [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> > > READ (1024001 bytes): OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 16384
> > > [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> > > READ ( 16384 bytes): OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 8192
> > > [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> > > READ ( 8192 bytes): OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 128
> > > [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> > > READ ( 128 bytes): OKAY
> > > root@hihope-rzg2m:~#
> > >
> > > Cheers,
> > > --Prabhakar
> > >
> > >> Thanks
> > >> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-28 18:44 ` Lad, Prabhakar
@ 2020-03-29 14:04 ` Lad, Prabhakar
2020-03-30 11:59 ` Kishon Vijay Abraham I
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-29 14:04 UTC (permalink / raw)
To: Kishon Vijay Abraham I, Lorenzo Pieralisi; +Cc: linux-pci
Hi Kishon,
On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
<prabhakar.csengg@gmail.com> wrote:
>
> Hi Kishon,
>
> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
> <prabhakar.csengg@gmail.com> wrote:
> >
> > Hi Kishon,
> >
> > On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> > >
> > > Hi Prabhakar,
> > >
> > > On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> > > > Hi Kishon,
> > > >
> > > > On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> > > >>
> > > >> Hi Prabhakar,
> > > >>
> > > >> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> > > >>> Hi Kishon,
> > > >>>
> > > >>> I rebased my rcar-endpoint patches on endpoint branch, which has
> > > >>> support for streaming DMA API support, with this read/write/copy
> > > >>> tests failed, to make sure nothing hasn't changed on my driver I
> > > >>> reverted the streaming DMA API patch
> > > >>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> > > >>> streaming DMA APIs for buffer allocation" and tests began to pass
> > > >>> again.
> > > >>>
> > > >>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> > > >>> for read/write/copy pass as expected.
> > > >>>
> > > >>> Could you please through some light why this could be happening.
> > > >>
> > > >> Do you see any differences in the address returned by dma_map_single() like is
> > > >> it 32-bit address or 64-bit address?
> > > >>
> > > > Both return 32 bit address, debugging further I see that with
> > > > GFP_KERNEL flag for small buffer
> > > > sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> > > > related to caching probably.
> > > > Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> > > > am using PIO transfers.
> > > > Any thoughts on how we tackle it ?
> > > >
> > > > # With GFP_KERNEL flag
> > > > root@hihope-rzg2m:~# pcitest -r
> > > > [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> > > > READ ( 102400 bytes): NOT OKAY
> > > > root@hihope-rzg2m:~# pcitest -r
> > > > [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> > > > READ ( 102400 bytes): OKAY
> > >
> > > Here one of the read test is passing and the other is failing.
> > > For the 1st case dma:7e99d000, address is aligned to 4K
> > > For the 2nd case dma:7e9c0000, address is aligned to 256K
> > >
> > > I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> > > EP has any restrictions? (like the address should be aligned to the size?).
> > >
> > There isn't any restriction for outbound ATU on ep, Although I tried
> > alignment from
> > SZ_1 - SZ_256K and each failed at several points.
> >
> > With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
> > is not aligned to 256 but still read passes.
> > root@hihope-rzg2m:~# pcitest -r -s 16384
> > [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> > READ ( 16384 bytes): OKAY
> >
> > And I have verified with GFP_KERNEL | GFP_DMA on my platform
> > everything works as expected,
> >
> > So how about a patch for pci_endpoint_test.c, where flags are passed
> > as part of driver_data and it defaults to just GFP_KERNEL ?
> >
> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
>
Correct me if I am wrong here, streaming DMA API should be used with
dma (-d) option so that root device
makes sure the data is synced when data is transferred whereas
previously with dma_alloc_coherent()
we didn't have to care about cache issues. Also for a non-dma (-d)
option we don't have a handle to dma
in rootpport device so that we can call a sync operation. I say this
because on my platform with streaming
DMA api it works for small size buffers but it doesn't work with large
size buffers.
Could you please confirm with streaming DMA api without DMA (-d)
option for large buffers read/write/copy
still passes for you.
Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
kzalloc on my platform fixes everything.
Cheers,
--Prabhakar
> Cheers,
> --Prabhakar
>
> > Cheers,
> > --Prabhakar
> >
> > > Thanks
> > > Kishon
> > >
> > > > root@hihope-rzg2m:~# pcitest -r
> > > > [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> > > > READ ( 102400 bytes): NOT OKAY
> > > > root@hihope-rzg2m:~# pcitest -r
> > > > [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> > > > READ ( 102400 bytes): NOT OKAY
> > > > root@hihope-rzg2m:~# pcitest -r
> > > > [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> > > > READ ( 102400 bytes): NOT OKAY
> > > >
> > > > # GFP_KERNEL | GFP_DMA
> > > >
> > > > root@hihope-rzg2m:~# pcitest -r -s 1024001
> > > > [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> > > > READ (1024001 bytes): OKAY
> > > > root@hihope-rzg2m:~# pcitest -r -s 16384
> > > > [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> > > > READ ( 16384 bytes): OKAY
> > > > root@hihope-rzg2m:~# pcitest -r -s 8192
> > > > [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> > > > READ ( 8192 bytes): OKAY
> > > > root@hihope-rzg2m:~# pcitest -r -s 128
> > > > [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> > > > READ ( 128 bytes): OKAY
> > > > root@hihope-rzg2m:~#
> > > >
> > > > Cheers,
> > > > --Prabhakar
> > > >
> > > >> Thanks
> > > >> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-29 14:04 ` Lad, Prabhakar
@ 2020-03-30 11:59 ` Kishon Vijay Abraham I
2020-03-30 13:09 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Kishon Vijay Abraham I @ 2020-03-30 11:59 UTC (permalink / raw)
To: Lad, Prabhakar, Lorenzo Pieralisi; +Cc: linux-pci
Hi Prabhakar,
On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
> Hi Kishon,
>
> On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
> <prabhakar.csengg@gmail.com> wrote:
>>
>> Hi Kishon,
>>
>> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
>> <prabhakar.csengg@gmail.com> wrote:
>>>
>>> Hi Kishon,
>>>
>>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>
>>>> Hi Prabhakar,
>>>>
>>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
>>>>> Hi Kishon,
>>>>>
>>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>
>>>>>> Hi Prabhakar,
>>>>>>
>>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>>>>>> Hi Kishon,
>>>>>>>
>>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>>>>>> support for streaming DMA API support, with this read/write/copy
>>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
>>>>>>> reverted the streaming DMA API patch
>>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>>>>>> again.
>>>>>>>
>>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>>>>>> for read/write/copy pass as expected.
>>>>>>>
>>>>>>> Could you please through some light why this could be happening.
>>>>>>
>>>>>> Do you see any differences in the address returned by dma_map_single() like is
>>>>>> it 32-bit address or 64-bit address?
>>>>>>
>>>>> Both return 32 bit address, debugging further I see that with
>>>>> GFP_KERNEL flag for small buffer
>>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
>>>>> related to caching probably.
>>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
>>>>> am using PIO transfers.
>>>>> Any thoughts on how we tackle it ?
>>>>>
>>>>> # With GFP_KERNEL flag
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes): NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes): OKAY
>>>>
>>>> Here one of the read test is passing and the other is failing.
>>>> For the 1st case dma:7e99d000, address is aligned to 4K
>>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
>>>>
>>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
>>>> EP has any restrictions? (like the address should be aligned to the size?).
>>>>
>>> There isn't any restriction for outbound ATU on ep, Although I tried
>>> alignment from
>>> SZ_1 - SZ_256K and each failed at several points.
>>>
>>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
>>> is not aligned to 256 but still read passes.
>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>> READ ( 16384 bytes): OKAY
>>>
>>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
>>> everything works as expected,
>>>
>>> So how about a patch for pci_endpoint_test.c, where flags are passed
>>> as part of driver_data and it defaults to just GFP_KERNEL ?
>>>
>> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
>>
> Correct me if I am wrong here, streaming DMA API should be used with
> dma (-d) option so that root device
> makes sure the data is synced when data is transferred whereas
> previously with dma_alloc_coherent()
> we didn't have to care about cache issues. Also for a non-dma (-d)
> option we don't have a handle to dma
> in rootpport device so that we can call a sync operation. I say this
> because on my platform with streaming
> DMA api it works for small size buffers but it doesn't work with large
> size buffers.
Streaming DMA API and DMA support in endpoint can be treated independently.
dma_alloc_coherent() will give you coherent memory, so you don't have to flush
or invalidate. This memory is usually limited in a platform.
The other option was to use streaming DMA APIs which doesn't give coherent
memory but SW has to take care of flush and invalidate.
>
> Could you please confirm with streaming DMA api without DMA (-d)
> option for large buffers read/write/copy
> still passes for you.
root@j7-evm:~# ./pcitest -r
READ ( 102400 bytes): OKAY
root@j7-evm:~# ./pcitest -r -s 1024000
READ (1024000 bytes): OKAY
root@j7-evm:~# ./pcitest -w -s 1024000
WRITE (1024000 bytes): OKAY
root@j7-evm:~# ./pcitest -c -s 1024000
COPY (1024000 bytes): OKAY
root@j7-evm:~# ./pcitest -c -s 10240000
COPY (10240000 bytes): OKAY
root@j7-evm:~# ./pcitest -r -s 10240000
READ (10240000 bytes): OKAY
root@j7-evm:~# ./pcitest -w -s 10240000
WRITE (10240000 bytes): OKAY
>
> Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
> kzalloc on my platform fixes everything.
Which host do you use? If this is only a host side limitation, you could try
using a different host.
Thanks
Kishon
>
> Cheers,
> --Prabhakar
>
>
>> Cheers,
>> --Prabhakar
>>
>>> Cheers,
>>> --Prabhakar
>>>
>>>> Thanks
>>>> Kishon
>>>>
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes): NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes): NOT OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>
>>>>> # GFP_KERNEL | GFP_DMA
>>>>>
>>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
>>>>> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
>>>>> READ (1024001 bytes): OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>> READ ( 16384 bytes): OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
>>>>> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
>>>>> READ ( 8192 bytes): OKAY
>>>>> root@hihope-rzg2m:~# pcitest -r -s 128
>>>>> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
>>>>> READ ( 128 bytes): OKAY
>>>>> root@hihope-rzg2m:~#
>>>>>
>>>>> Cheers,
>>>>> --Prabhakar
>>>>>
>>>>>> Thanks
>>>>>> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-30 11:59 ` Kishon Vijay Abraham I
@ 2020-03-30 13:09 ` Lad, Prabhakar
2020-03-30 13:20 ` Kishon Vijay Abraham I
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-30 13:09 UTC (permalink / raw)
To: Kishon Vijay Abraham I; +Cc: Lorenzo Pieralisi, linux-pci
Hi Kishon,
On Mon, Mar 30, 2020 at 12:59 PM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Prabhakar,
>
> On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
> > Hi Kishon,
> >
> > On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
> > <prabhakar.csengg@gmail.com> wrote:
> >>
> >> Hi Kishon,
> >>
> >> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
> >> <prabhakar.csengg@gmail.com> wrote:
> >>>
> >>> Hi Kishon,
> >>>
> >>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>>>
> >>>> Hi Prabhakar,
> >>>>
> >>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> >>>>> Hi Kishon,
> >>>>>
> >>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>>>>>
> >>>>>> Hi Prabhakar,
> >>>>>>
> >>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> >>>>>>> Hi Kishon,
> >>>>>>>
> >>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
> >>>>>>> support for streaming DMA API support, with this read/write/copy
> >>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
> >>>>>>> reverted the streaming DMA API patch
> >>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> >>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
> >>>>>>> again.
> >>>>>>>
> >>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> >>>>>>> for read/write/copy pass as expected.
> >>>>>>>
> >>>>>>> Could you please through some light why this could be happening.
> >>>>>>
> >>>>>> Do you see any differences in the address returned by dma_map_single() like is
> >>>>>> it 32-bit address or 64-bit address?
> >>>>>>
> >>>>> Both return 32 bit address, debugging further I see that with
> >>>>> GFP_KERNEL flag for small buffer
> >>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> >>>>> related to caching probably.
> >>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> >>>>> am using PIO transfers.
> >>>>> Any thoughts on how we tackle it ?
> >>>>>
> >>>>> # With GFP_KERNEL flag
> >>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> >>>>> READ ( 102400 bytes): NOT OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> >>>>> READ ( 102400 bytes): OKAY
> >>>>
> >>>> Here one of the read test is passing and the other is failing.
> >>>> For the 1st case dma:7e99d000, address is aligned to 4K
> >>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
> >>>>
> >>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> >>>> EP has any restrictions? (like the address should be aligned to the size?).
> >>>>
> >>> There isn't any restriction for outbound ATU on ep, Although I tried
> >>> alignment from
> >>> SZ_1 - SZ_256K and each failed at several points.
> >>>
> >>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
> >>> is not aligned to 256 but still read passes.
> >>> root@hihope-rzg2m:~# pcitest -r -s 16384
> >>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> >>> READ ( 16384 bytes): OKAY
> >>>
> >>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
> >>> everything works as expected,
> >>>
> >>> So how about a patch for pci_endpoint_test.c, where flags are passed
> >>> as part of driver_data and it defaults to just GFP_KERNEL ?
> >>>
> >> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
> >>
> > Correct me if I am wrong here, streaming DMA API should be used with
> > dma (-d) option so that root device
> > makes sure the data is synced when data is transferred whereas
> > previously with dma_alloc_coherent()
> > we didn't have to care about cache issues. Also for a non-dma (-d)
> > option we don't have a handle to dma
> > in rootpport device so that we can call a sync operation. I say this
> > because on my platform with streaming
> > DMA api it works for small size buffers but it doesn't work with large
> > size buffers.
>
> Streaming DMA API and DMA support in endpoint can be treated independently.
> dma_alloc_coherent() will give you coherent memory, so you don't have to flush
> or invalidate. This memory is usually limited in a platform.
> The other option was to use streaming DMA APIs which doesn't give coherent
> memory but SW has to take care of flush and invalidate.
>
Agreed. But we don't flush in SW when -d option is not specified I am
assuming when we us
-d dma engine takes care of flushing it.
> >
> > Could you please confirm with streaming DMA api without DMA (-d)
> > option for large buffers read/write/copy
> > still passes for you.
>
> root@j7-evm:~# ./pcitest -r
> READ ( 102400 bytes): OKAY
> root@j7-evm:~# ./pcitest -r -s 1024000
> READ (1024000 bytes): OKAY
> root@j7-evm:~# ./pcitest -w -s 1024000
> WRITE (1024000 bytes): OKAY
> root@j7-evm:~# ./pcitest -c -s 1024000
> COPY (1024000 bytes): OKAY
> root@j7-evm:~# ./pcitest -c -s 10240000
> COPY (10240000 bytes): OKAY
> root@j7-evm:~# ./pcitest -r -s 10240000
> READ (10240000 bytes): OKAY
> root@j7-evm:~# ./pcitest -w -s 10240000
> WRITE (10240000 bytes): OKAY
Thank you for testing is this on Jacinto ?
> >
> > Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
> > kzalloc on my platform fixes everything.
>
> Which host do you use? If this is only a host side limitation, you could try
> using a different host.
>
I am trying this on Renesas RZ/G2N as host and RZ/G2E as an endpoint.
ATM I can only test this on
Renesas platforms only and all of them have same PCIe controller :(
How about adding flags as part of driver data and defaulting it GFP_KERNEL ?
Cheers,
--Prabhakar
> Thanks
> Kishon
>
> >
> > Cheers,
> > --Prabhakar
> >
> >
> >> Cheers,
> >> --Prabhakar
> >>
> >>> Cheers,
> >>> --Prabhakar
> >>>
> >>>> Thanks
> >>>> Kishon
> >>>>
> >>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> >>>>> READ ( 102400 bytes): NOT OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> >>>>> READ ( 102400 bytes): NOT OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> >>>>> READ ( 102400 bytes): NOT OKAY
> >>>>>
> >>>>> # GFP_KERNEL | GFP_DMA
> >>>>>
> >>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
> >>>>> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> >>>>> READ (1024001 bytes): OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
> >>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> >>>>> READ ( 16384 bytes): OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
> >>>>> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> >>>>> READ ( 8192 bytes): OKAY
> >>>>> root@hihope-rzg2m:~# pcitest -r -s 128
> >>>>> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> >>>>> READ ( 128 bytes): OKAY
> >>>>> root@hihope-rzg2m:~#
> >>>>>
> >>>>> Cheers,
> >>>>> --Prabhakar
> >>>>>
> >>>>>> Thanks
> >>>>>> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-30 13:09 ` Lad, Prabhakar
@ 2020-03-30 13:20 ` Kishon Vijay Abraham I
2020-03-30 13:46 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Kishon Vijay Abraham I @ 2020-03-30 13:20 UTC (permalink / raw)
To: Lad, Prabhakar; +Cc: Lorenzo Pieralisi, linux-pci
Prabhakar,
On 3/30/2020 6:39 PM, Lad, Prabhakar wrote:
> Hi Kishon,
>
> On Mon, Mar 30, 2020 at 12:59 PM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>
>> Hi Prabhakar,
>>
>> On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
>>> Hi Kishon,
>>>
>>> On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
>>> <prabhakar.csengg@gmail.com> wrote:
>>>>
>>>> Hi Kishon,
>>>>
>>>> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
>>>> <prabhakar.csengg@gmail.com> wrote:
>>>>>
>>>>> Hi Kishon,
>>>>>
>>>>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>
>>>>>> Hi Prabhakar,
>>>>>>
>>>>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
>>>>>>> Hi Kishon,
>>>>>>>
>>>>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>>>>>>
>>>>>>>> Hi Prabhakar,
>>>>>>>>
>>>>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
>>>>>>>>> Hi Kishon,
>>>>>>>>>
>>>>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
>>>>>>>>> support for streaming DMA API support, with this read/write/copy
>>>>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
>>>>>>>>> reverted the streaming DMA API patch
>>>>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
>>>>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
>>>>>>>>> again.
>>>>>>>>>
>>>>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
>>>>>>>>> for read/write/copy pass as expected.
>>>>>>>>>
>>>>>>>>> Could you please through some light why this could be happening.
>>>>>>>>
>>>>>>>> Do you see any differences in the address returned by dma_map_single() like is
>>>>>>>> it 32-bit address or 64-bit address?
>>>>>>>>
>>>>>>> Both return 32 bit address, debugging further I see that with
>>>>>>> GFP_KERNEL flag for small buffer
>>>>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
>>>>>>> related to caching probably.
>>>>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
>>>>>>> am using PIO transfers.
>>>>>>> Any thoughts on how we tackle it ?
>>>>>>>
>>>>>>> # With GFP_KERNEL flag
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): OKAY
>>>>>>
>>>>>> Here one of the read test is passing and the other is failing.
>>>>>> For the 1st case dma:7e99d000, address is aligned to 4K
>>>>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
>>>>>>
>>>>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
>>>>>> EP has any restrictions? (like the address should be aligned to the size?).
>>>>>>
>>>>> There isn't any restriction for outbound ATU on ep, Although I tried
>>>>> alignment from
>>>>> SZ_1 - SZ_256K and each failed at several points.
>>>>>
>>>>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
>>>>> is not aligned to 256 but still read passes.
>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>> READ ( 16384 bytes): OKAY
>>>>>
>>>>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
>>>>> everything works as expected,
>>>>>
>>>>> So how about a patch for pci_endpoint_test.c, where flags are passed
>>>>> as part of driver_data and it defaults to just GFP_KERNEL ?
>>>>>
>>>> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
>>>>
>>> Correct me if I am wrong here, streaming DMA API should be used with
>>> dma (-d) option so that root device
>>> makes sure the data is synced when data is transferred whereas
>>> previously with dma_alloc_coherent()
>>> we didn't have to care about cache issues. Also for a non-dma (-d)
>>> option we don't have a handle to dma
>>> in rootpport device so that we can call a sync operation. I say this
>>> because on my platform with streaming
>>> DMA api it works for small size buffers but it doesn't work with large
>>> size buffers.
>>
>> Streaming DMA API and DMA support in endpoint can be treated independently.
>> dma_alloc_coherent() will give you coherent memory, so you don't have to flush
>> or invalidate. This memory is usually limited in a platform.
>> The other option was to use streaming DMA APIs which doesn't give coherent
>> memory but SW has to take care of flush and invalidate.
>>
> Agreed. But we don't flush in SW when -d option is not specified I am
> assuming when we us
> -d dma engine takes care of flushing it.
The -d option switch doesn't change anything on the SW that runs on the host
side (misc/pci-endpoint-test.c). That only tells the EP to use DMA.
When you use streaming APIs, dma_map_single(), dmap_unmap_single() takes care
of flushing or invalidating memory based on the platform. (Platforms which have
coherent memory will have dma-coherent property,
dma_map_single()/dmap_unmap_single() will not do flush or invalidate.
Did you try to probe the failure further by comparing the hexdumps? Where does
the mismatch happen?
>
>>>
>>> Could you please confirm with streaming DMA api without DMA (-d)
>>> option for large buffers read/write/copy
>>> still passes for you.
>>
>> root@j7-evm:~# ./pcitest -r
>> READ ( 102400 bytes): OKAY
>> root@j7-evm:~# ./pcitest -r -s 1024000
>> READ (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -w -s 1024000
>> WRITE (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -c -s 1024000
>> COPY (1024000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -c -s 10240000
>> COPY (10240000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -r -s 10240000
>> READ (10240000 bytes): OKAY
>> root@j7-evm:~# ./pcitest -w -s 10240000
>> WRITE (10240000 bytes): OKAY
>
> Thank you for testing is this on Jacinto ?
>
>>>
>>> Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
>>> kzalloc on my platform fixes everything.
>>
>> Which host do you use? If this is only a host side limitation, you could try
>> using a different host.
>>
> I am trying this on Renesas RZ/G2N as host and RZ/G2E as an endpoint.
> ATM I can only test this on
> Renesas platforms only and all of them have same PCIe controller :(
Does the host controller work well with other standard PCIe cards (NVMe, USB
etc.,)?
>
> How about adding flags as part of driver data and defaulting it GFP_KERNEL ?
Nope. Not before we clearly understand the reason for the failure.
Thanks
Kishon
>
> Cheers,
> --Prabhakar
>
>> Thanks
>> Kishon
>>
>>>
>>> Cheers,
>>> --Prabhakar
>>>
>>>
>>>> Cheers,
>>>> --Prabhakar
>>>>
>>>>> Cheers,
>>>>> --Prabhakar
>>>>>
>>>>>> Thanks
>>>>>> Kishon
>>>>>>
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r
>>>>>>> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
>>>>>>> READ ( 102400 bytes): NOT OKAY
>>>>>>>
>>>>>>> # GFP_KERNEL | GFP_DMA
>>>>>>>
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
>>>>>>> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
>>>>>>> READ (1024001 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
>>>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
>>>>>>> READ ( 16384 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
>>>>>>> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
>>>>>>> READ ( 8192 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~# pcitest -r -s 128
>>>>>>> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
>>>>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
>>>>>>> READ ( 128 bytes): OKAY
>>>>>>> root@hihope-rzg2m:~#
>>>>>>>
>>>>>>> Cheers,
>>>>>>> --Prabhakar
>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-30 13:20 ` Kishon Vijay Abraham I
@ 2020-03-30 13:46 ` Lad, Prabhakar
2020-03-31 7:10 ` Yoshihiro Shimoda
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-30 13:46 UTC (permalink / raw)
To: Kishon Vijay Abraham I, Yoshihiro Shimoda; +Cc: Lorenzo Pieralisi, linux-pci
Hi Kishon,
On Mon, Mar 30, 2020 at 2:21 PM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Prabhakar,
>
> On 3/30/2020 6:39 PM, Lad, Prabhakar wrote:
> > Hi Kishon,
> >
> > On Mon, Mar 30, 2020 at 12:59 PM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>
> >> Hi Prabhakar,
> >>
> >> On 3/29/2020 7:34 PM, Lad, Prabhakar wrote:
> >>> Hi Kishon,
> >>>
> >>> On Sat, Mar 28, 2020 at 6:44 PM Lad, Prabhakar
> >>> <prabhakar.csengg@gmail.com> wrote:
> >>>>
> >>>> Hi Kishon,
> >>>>
> >>>> On Tue, Mar 24, 2020 at 2:41 PM Lad, Prabhakar
> >>>> <prabhakar.csengg@gmail.com> wrote:
> >>>>>
> >>>>> Hi Kishon,
> >>>>>
> >>>>> On Tue, Mar 24, 2020 at 1:58 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>>>>>
> >>>>>> Hi Prabhakar,
> >>>>>>
> >>>>>> On 3/22/2020 4:19 AM, Lad, Prabhakar wrote:
> >>>>>>> Hi Kishon,
> >>>>>>>
> >>>>>>> On Fri, Mar 20, 2020 at 5:28 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Prabhakar,
> >>>>>>>>
> >>>>>>>> On 3/18/2020 5:07 PM, Lad, Prabhakar wrote:
> >>>>>>>>> Hi Kishon,
> >>>>>>>>>
> >>>>>>>>> I rebased my rcar-endpoint patches on endpoint branch, which has
> >>>>>>>>> support for streaming DMA API support, with this read/write/copy
> >>>>>>>>> tests failed, to make sure nothing hasn't changed on my driver I
> >>>>>>>>> reverted the streaming DMA API patch
> >>>>>>>>> 74b9b4da84c71418ceeaaeb78dc790376df92fea "misc: pci_endpoint_test: Use
> >>>>>>>>> streaming DMA APIs for buffer allocation" and tests began to pass
> >>>>>>>>> again.
> >>>>>>>>>
> >>>>>>>>> If add a GFP_DMA flag for kzalloc (with streaming DMA), the test cases
> >>>>>>>>> for read/write/copy pass as expected.
> >>>>>>>>>
> >>>>>>>>> Could you please through some light why this could be happening.
> >>>>>>>>
> >>>>>>>> Do you see any differences in the address returned by dma_map_single() like is
> >>>>>>>> it 32-bit address or 64-bit address?
> >>>>>>>>
> >>>>>>> Both return 32 bit address, debugging further I see that with
> >>>>>>> GFP_KERNEL flag for small buffer
> >>>>>>> sizes the read/write/copy tests pass(upto 4k), so I am suspecting its
> >>>>>>> related to caching probably.
> >>>>>>> Also adding wmb()/rmb() just with GFP_KERNEL flag didn't help. Note I
> >>>>>>> am using PIO transfers.
> >>>>>>> Any thoughts on how we tackle it ?
> >>>>>>>
> >>>>>>> # With GFP_KERNEL flag
> >>>>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>>>> [ 46.210649] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff0004b4ae0000 dma:7e99d000 align:ffff0004b4ae0000
> >>>>>>> READ ( 102400 bytes): NOT OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>>>> [ 51.880063] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9c0000 align:ffff0004b4ae0000
> >>>>>>> READ ( 102400 bytes): OKAY
> >>>>>>
> >>>>>> Here one of the read test is passing and the other is failing.
> >>>>>> For the 1st case dma:7e99d000, address is aligned to 4K
> >>>>>> For the 2nd case dma:7e9c0000, address is aligned to 256K
> >>>>>>
> >>>>>> I'm suspecting this could be an alignment issue. Does the outbound ATU of your
> >>>>>> EP has any restrictions? (like the address should be aligned to the size?).
> >>>>>>
> >>>>> There isn't any restriction for outbound ATU on ep, Although I tried
> >>>>> alignment from
> >>>>> SZ_1 - SZ_256K and each failed at several points.
> >>>>>
> >>>>> With GFP_KERNEL | GFP_DMA, as in my previous dump here the address too
> >>>>> is not aligned to 256 but still read passes.
> >>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
> >>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> >>>>> READ ( 16384 bytes): OKAY
> >>>>>
> >>>>> And I have verified with GFP_KERNEL | GFP_DMA on my platform
> >>>>> everything works as expected,
> >>>>>
> >>>>> So how about a patch for pci_endpoint_test.c, where flags are passed
> >>>>> as part of driver_data and it defaults to just GFP_KERNEL ?
> >>>>>
> >>>> Any thoughts on the above ? I intended to get the endpoint driver for v5.7.
> >>>>
> >>> Correct me if I am wrong here, streaming DMA API should be used with
> >>> dma (-d) option so that root device
> >>> makes sure the data is synced when data is transferred whereas
> >>> previously with dma_alloc_coherent()
> >>> we didn't have to care about cache issues. Also for a non-dma (-d)
> >>> option we don't have a handle to dma
> >>> in rootpport device so that we can call a sync operation. I say this
> >>> because on my platform with streaming
> >>> DMA api it works for small size buffers but it doesn't work with large
> >>> size buffers.
> >>
> >> Streaming DMA API and DMA support in endpoint can be treated independently.
> >> dma_alloc_coherent() will give you coherent memory, so you don't have to flush
> >> or invalidate. This memory is usually limited in a platform.
> >> The other option was to use streaming DMA APIs which doesn't give coherent
> >> memory but SW has to take care of flush and invalidate.
> >>
> > Agreed. But we don't flush in SW when -d option is not specified I am
> > assuming when we us
> > -d dma engine takes care of flushing it.
>
> The -d option switch doesn't change anything on the SW that runs on the host
> side (misc/pci-endpoint-test.c). That only tells the EP to use DMA.
>
> When you use streaming APIs, dma_map_single(), dmap_unmap_single() takes care
> of flushing or invalidating memory based on the platform. (Platforms which have
> coherent memory will have dma-coherent property,
> dma_map_single()/dmap_unmap_single() will not do flush or invalidate.
>
My bad, I thought dma_sync*() calls did it.
Shimoda-san do you see any platform restrictions while using streaming DMA api
instead of coherent memory. Note I tried this enabling/disabling ipmmu
too but the
results are the same.
> Did you try to probe the failure further by comparing the hexdumps? Where does
> the mismatch happen?
>
I shall dump the memory and check the values, but basically crc is failing.
> >
> >>>
> >>> Could you please confirm with streaming DMA api without DMA (-d)
> >>> option for large buffers read/write/copy
> >>> still passes for you.
> >>
> >> root@j7-evm:~# ./pcitest -r
> >> READ ( 102400 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -r -s 1024000
> >> READ (1024000 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -w -s 1024000
> >> WRITE (1024000 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -c -s 1024000
> >> COPY (1024000 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -c -s 10240000
> >> COPY (10240000 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -r -s 10240000
> >> READ (10240000 bytes): OKAY
> >> root@j7-evm:~# ./pcitest -w -s 10240000
> >> WRITE (10240000 bytes): OKAY
> >
> > Thank you for testing is this on Jacinto ?
> >
> >>>
> >>> Although I am not sure why adding GFP_KERNEL | GFP_DMA flag for
> >>> kzalloc on my platform fixes everything.
> >>
> >> Which host do you use? If this is only a host side limitation, you could try
> >> using a different host.
> >>
> > I am trying this on Renesas RZ/G2N as host and RZ/G2E as an endpoint.
> > ATM I can only test this on
> > Renesas platforms only and all of them have same PCIe controller :(
>
> Does the host controller work well with other standard PCIe cards (NVMe, USB
> etc.,)?
It does work Ethernet/USB if I am not wrong there is a issue with NVMe.
> >
> > How about adding flags as part of driver data and defaulting it GFP_KERNEL ?
>
> Nope. Not before we clearly understand the reason for the failure.
>
I shall do further debugging.
Cheers,
--Prabhakar
> Thanks
> Kishon
>
> >
> > Cheers,
> > --Prabhakar
> >
> >> Thanks
> >> Kishon
> >>
> >>>
> >>> Cheers,
> >>> --Prabhakar
> >>>
> >>>
> >>>> Cheers,
> >>>> --Prabhakar
> >>>>
> >>>>> Cheers,
> >>>>> --Prabhakar
> >>>>>
> >>>>>> Thanks
> >>>>>> Kishon
> >>>>>>
> >>>>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>>>> [ 53.354830] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff0004b4ae0000 dma:7e9e2000 align:ffff0004b4ae0000
> >>>>>>> READ ( 102400 bytes): NOT OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>>>> [ 55.307236] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea04000 align:ffff0004b4ae0000
> >>>>>>> READ ( 102400 bytes): NOT OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r
> >>>>>>> [ 57.098626] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff0004b4ae0000 dma:7ea23000 align:ffff0004b4ae0000
> >>>>>>> READ ( 102400 bytes): NOT OKAY
> >>>>>>>
> >>>>>>> # GFP_KERNEL | GFP_DMA
> >>>>>>>
> >>>>>>> root@hihope-rzg2m:~# pcitest -r -s 1024001
> >>>>>>> [ 174.562071] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff00003b900000 dma:7b900000 align:ffff00003b900000
> >>>>>>> READ (1024001 bytes): OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r -s 16384
> >>>>>>> [ 186.629347] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff00003b848000 dma:7b848000 align:ffff00003b848000
> >>>>>>> READ ( 16384 bytes): OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r -s 8192
> >>>>>>> [ 190.578335] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff00003b840000 dma:7b840000 align:ffff00003b840000
> >>>>>>> READ ( 8192 bytes): OKAY
> >>>>>>> root@hihope-rzg2m:~# pcitest -r -s 128
> >>>>>>> [ 199.428021] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> >>>>>>> kzalloc:ffff00003b800000 dma:7b800000 align:ffff00003b800000
> >>>>>>> READ ( 128 bytes): OKAY
> >>>>>>> root@hihope-rzg2m:~#
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> --Prabhakar
> >>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Kishon
^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: PCIe EPF
2020-03-30 13:46 ` Lad, Prabhakar
@ 2020-03-31 7:10 ` Yoshihiro Shimoda
2020-03-31 17:25 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Yoshihiro Shimoda @ 2020-03-31 7:10 UTC (permalink / raw)
To: Lad, Prabhakar, Kishon Vijay Abraham I; +Cc: Lorenzo Pieralisi, linux-pci
Hi Prabhakar-san,
> From: Lad, Prabhakar, Sent: Monday, March 30, 2020 10:47 PM
<snip>
> > > Agreed. But we don't flush in SW when -d option is not specified I am
> > > assuming when we us
> > > -d dma engine takes care of flushing it.
> >
> > The -d option switch doesn't change anything on the SW that runs on the host
> > side (misc/pci-endpoint-test.c). That only tells the EP to use DMA.
> >
> > When you use streaming APIs, dma_map_single(), dmap_unmap_single() takes care
> > of flushing or invalidating memory based on the platform. (Platforms which have
> > coherent memory will have dma-coherent property,
> > dma_map_single()/dmap_unmap_single() will not do flush or invalidate.
> >
> My bad, I thought dma_sync*() calls did it.
>
> Shimoda-san do you see any platform restrictions while using streaming DMA api
> instead of coherent memory. Note I tried this enabling/disabling ipmmu
> too but the
> results are the same.
(Un)fortunately, I have never replaced coherent memory API with stream DMA API.
> > Did you try to probe the failure further by comparing the hexdumps? Where does
> > the mismatch happen?
> >
> I shall dump the memory and check the values, but basically crc is failing.
I'm also interesting about comparing the hexdump between host and ep.
This is my gut feeling though, I'm guessing this is a timing issue
because using coherent memory API will be on non-cache even if CPU access,
but using steaming DMA API will be on cache if CPU access by
dma_unmap_single(DMA_FROM_DEVICE) and get_random_bytes() in pci_endpoint_test_write().
If my guess is correct, almost all hexdumps are the same, but last
some bytes (I'm not sure how much bytes though) are not match.
Best regards,
Yoshihiro Shimoda
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-03-31 7:10 ` Yoshihiro Shimoda
@ 2020-03-31 17:25 ` Lad, Prabhakar
2020-04-01 1:25 ` Yoshihiro Shimoda
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-03-31 17:25 UTC (permalink / raw)
To: Yoshihiro Shimoda, Kishon Vijay Abraham I
Cc: Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Shimoda-san,
On Tue, Mar 31, 2020 at 8:10 AM Yoshihiro Shimoda
<yoshihiro.shimoda.uh@renesas.com> wrote:
>
> Hi Prabhakar-san,
>
> > From: Lad, Prabhakar, Sent: Monday, March 30, 2020 10:47 PM
> <snip>
> > > > Agreed. But we don't flush in SW when -d option is not specified I am
> > > > assuming when we us
> > > > -d dma engine takes care of flushing it.
> > >
> > > The -d option switch doesn't change anything on the SW that runs on the host
> > > side (misc/pci-endpoint-test.c). That only tells the EP to use DMA.
> > >
> > > When you use streaming APIs, dma_map_single(), dmap_unmap_single() takes care
> > > of flushing or invalidating memory based on the platform. (Platforms which have
> > > coherent memory will have dma-coherent property,
> > > dma_map_single()/dmap_unmap_single() will not do flush or invalidate.
> > >
> > My bad, I thought dma_sync*() calls did it.
> >
> > Shimoda-san do you see any platform restrictions while using streaming DMA api
> > instead of coherent memory. Note I tried this enabling/disabling ipmmu
> > too but the
> > results are the same.
>
> (Un)fortunately, I have never replaced coherent memory API with stream DMA API.
>
> > > Did you try to probe the failure further by comparing the hexdumps? Where does
> > > the mismatch happen?
> > >
> > I shall dump the memory and check the values, but basically crc is failing.
>
> I'm also interesting about comparing the hexdump between host and ep.
> This is my gut feeling though, I'm guessing this is a timing issue
> because using coherent memory API will be on non-cache even if CPU access,
> but using steaming DMA API will be on cache if CPU access by
> dma_unmap_single(DMA_FROM_DEVICE) and get_random_bytes() in pci_endpoint_test_write().
>
> If my guess is correct, almost all hexdumps are the same, but last
> some bytes (I'm not sure how much bytes though) are not match.
>
So I did some experiments only focusing on read operation for now
1: I tried printing the crc value in pci_epf_test_write() function
with below code snippet:
@@ -472,6 +474,9 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
*/
usleep_range(1000, 2000);
+ dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
+ phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
+
err_dma_map:
kfree(buf);
But with this I get :
[ 873.081520] WRITE => Size: 17 bytes DMA: NO Time:
0.000002333 seconds Rate: 7115 KB/s
[ 873.092461] Unable to handle kernel paging request at virtual
address ffff800014400001
[ 873.100481] Mem abort info:
[ 873.103441] ESR = 0x96000021
[ 873.106598] EC = 0x25: DABT (current EL), IL = 32 bits
[ 873.112064] SET = 0, FnV = 0
[ 873.115207] EA = 0, S1PTW = 0
[ 873.118500] Data abort info:
[ 873.121473] ISV = 0, ISS = 0x00000021
[ 873.125444] CM = 0, WnR = 0
[ 873.128437] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000048e3e000
[ 873.135272] [ffff800014400001] pgd=00000000b9fff003,
pud=00000000b9ffe003, pmd=00000000b2ddc003, pte=00680000fe100707
[ 873.146710] Internal error: Oops: 96000021 [#1] PREEMPT SMP
[ 873.152326] CPU: 0 PID: 393 Comm: kworker/0:1H Not tainted
5.6.0-rc1-arm64-renesas-g842da098c217-dirty #215
[ 873.162085] Hardware name: Silicon Linux RZ/G2E evaluation kit
EK874 (CAT874 + CAT875) (DT)
[ 873.170478] Workqueue: kpcitest pci_epf_test_cmd_handler
[ 873.175804] pstate: 20000005 (nzCv daif -PAN -UAO)
[ 873.180614] pc : crc32_le+0x28/0xe0
[ 873.184109] lr : pci_epf_test_cmd_handler+0x4ec/0xb50
[ 873.189169] sp : ffff800011cf3cb0
[ 873.192488] x29: ffff800011cf3cb0 x28: ffff800011bff000
[ 873.197812] x27: 0000000000000000 x26: ffff800014400000
[ 873.203136] x25: ffff0000762e0ac8 x24: ffff000072f87000
[ 873.208459] x23: ffff000072ce6480 x22: ffff0000764ee000
[ 873.213783] x21: ffff800011bff000 x20: ffff0000762e0a80
[ 873.219107] x19: ffff800010f0f000 x18: ffffffffffffffff
[ 873.224431] x17: 00000000e1b480cf x16: 000000004efa1cf1
[ 873.229755] x15: ffff800010f0f908 x14: ffff800091cf39a7
[ 873.235078] x13: ffff800011cf39b5 x12: 0000000000000001
[ 873.240401] x11: 0000000000000002 x10: 0000000000000050
[ 873.245725] x9 : ffff000079bd4fd8 x8 : ffff800014400001
[ 873.251048] x7 : 0000000000000011 x6 : 0000000000000000
[ 873.256371] x5 : 0000000000000001 x4 : dfdfdfdfdfdfdfdf
[ 873.261694] x3 : dfdfdfdfdfdfdfdf x2 : 0000000000000000
[ 873.267018] x1 : ffff800014400011 x0 : 00000000ffffffff
[ 873.272342] Call trace:
[ 873.274795] crc32_le+0x28/0xe0
[ 873.277950] process_one_work+0x298/0x710
[ 873.281968] worker_thread+0x40/0x468
[ 873.285639] kthread+0x11c/0x120
[ 873.288877] ret_from_fork+0x10/0x18
[ 873.292464] Code: 92400ce8 a9401023 8b010108 8b070021 (a9401905)
[ 873.298575] ---[ end trace 64f8c9b1e3bdfa49 ]---
2: Instead of get_random_bytes() in pci_epf_test_write() I used
memset() with the fixed value
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -431,6 +431,8 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
}
get_random_bytes(buf, reg->size);
+ memset(buf, 0xDF, reg->size);
reg->checksum = crc32_le(~0, buf, reg->size);
And later for the dst buffer I did memset for dst_buffer
@@ -472,6 +474,10 @@ static int pci_epf_test_write(struct pci_epf_test
*epf_test)
*/
usleep_range(1000, 2000);
+ memset(dst_addr, 0xDF, reg->size);
+ dev_err(dev, "%s %llx %llx csum: %x\n", __func__, reg->dst_addr,
+ phys_addr, reg->checksum);
+
err_dma_map:
kfree(buf);
But with this I get similar issue as above:
[ 102.576482] WRITE => Size: 151551 bytes DMA: NO Time:
0.002141125 seconds Rate: 69122 KB/s
[ 102.590369] Unable to handle kernel paging request at virtual
address ffff800012424fef
[ 102.598717] Mem abort info:
[ 102.601873] ESR = 0x96000061
[ 102.605094] EC = 0x25: DABT (current EL), IL = 32 bits
[ 102.610492] SET = 0, FnV = 0
[ 102.613645] EA = 0, S1PTW = 0
[ 102.616855] Data abort info:
[ 102.619834] ISV = 0, ISS = 0x00000061
[ 102.623742] CM = 0, WnR = 1
[ 102.626733] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000048e2e000
[ 102.633528] [ffff800012424fef] pgd=00000000b9fff003,
pud=00000000b9ffe003, pmd=00000000b2d16003, pte=00680000fe124707
[ 102.644870] Internal error: Oops: 96000061 [#1] PREEMPT SMP
[ 102.650485] CPU: 0 PID: 392 Comm: kworker/0:1H Not tainted
5.6.0-rc1-arm64-renesas-g842da098c217-dirty #216
[ 102.660245] Hardware name: Silicon Linux RZ/G2E evaluation kit
EK874 (CAT874 + CAT875) (DT)
[ 102.668639] Workqueue: kpcitest pci_epf_test_cmd_handler
[ 102.673967] pstate: 00000005 (nzcv daif -PAN -UAO)
[ 102.678776] pc : __memset+0x90/0x188
[ 102.682359] lr : pci_epf_test_cmd_handler+0x4e4/0xb40
[ 102.687419] sp : ffff8000122f3cb0
[ 102.690738] x29: ffff8000122f3cb0 x28: ffff000074b44ac8
[ 102.696062] x27: 0000000000000000 x26: ffff800012400000
[ 102.701386] x25: ffff800011bed000 x24: ffff000072fc3800
[ 102.706709] x23: ffff000071300000 x22: ffff0000764f3000
[ 102.712033] x21: ffff800011bed000 x20: ffff000074b44a80
[ 102.717356] x19: ffff800010eff000 x18: 0000000000000000
[ 102.722680] x17: 606069623b383631 x16: 0000000000000001
[ 102.728003] x15: 0000000000000036 x14: 000000177c58c470
[ 102.733326] x13: 0000000000000284 x12: 0000000000000001
[ 102.738650] x11: 0000000000000002 x10: 0000000000000050
[ 102.743973] x9 : ffff000079bd4fd8 x8 : ffff800012424fff
[ 102.749297] x7 : dfdfdfdfdfdfdfdf x6 : 0000000000000000
[ 102.754620] x5 : 0000000000000001 x4 : 0000000000000000
[ 102.759943] x3 : 0000000000000030 x2 : 000000000000000f
[ 102.765267] x1 : 00000000000000df x0 : ffff800012400000
[ 102.770592] Call trace:
[ 102.773043] __memset+0x90/0x188
[ 102.776285] process_one_work+0x298/0x710
[ 102.780303] worker_thread+0x40/0x468
[ 102.783976] kthread+0x11c/0x120
[ 102.787214] ret_from_fork+0x10/0x18
[ 102.790800] Code: a8811d07 f2400c42 b4000062 8b020108 (a93f1d07)
[ 102.796912] ---[ end trace d3b2b57e50f065f8 ]--
Note: Also changing DMA_FROM_DEVICE to DMA_BIDIRECTIONAL in
pci_endpoint_test_read() resulted same behaviour for both 1 & 2.
3: After making sure -s was aligned to 128 bytes memset and crc32_le
in pci_epf_test_write() didnt hit above issues,
and now when I compared the crc value of both the buffers this
matched on endpoint!
4: Now focusing more in rootdevice, I added memset(buf, 0xDF,
reg->size); in pci_epf_test_write() function
and was dumping the contents of buffer in pci_endpoint_test_read()
with below function:
static void print_buffer(struct device *dev, void *buff_addr, size_t size)
{
size_t i;
u64 *addr = buff_addr;
for(i = 0; i < size; i += sizeof(addr))
dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
dev_err(dev, "\n\n\n\n");
}
Below is the dump from pci_endpoint_test_read() of one passing case
and other no-passing case:
The contents of buffer match as expected, but the crc32_le is
differrent and I am clue-less why is this
being caused when using non-coherent memory.
root@hihope-rzg2m:~# pcitest -r -s 2176
[ 526.726444] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 526.732942] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 526.739434] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 526.746021] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 526.752587] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 526.759157] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 526.765728] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 526.772307] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 526.778872] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 526.785438] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 526.792001] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 526.798569] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 526.805132] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 526.811697] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 526.818372] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 526.825028] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 526.831687] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 526.838342] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 526.844989] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 526.851645] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 526.858298] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 526.864967] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 526.871623] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 526.878277] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 526.884923] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 526.891576] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 526.898225] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 526.904876] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 526.911525] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 526.918178] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 526.924825] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 526.931489] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 526.938140] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 526.944789] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 526.951439] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 526.958089] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 526.964737] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 526.971390] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 526.978043] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 526.984712] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 526.991371] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 526.998030] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 527.004678] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 527.011330] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 527.017979] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 527.024633] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 527.031291] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 527.037945] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 527.044594] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 527.051246] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 527.057895] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 527.064546] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 527.071194] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 527.077926] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 527.084598] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 527.091262] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 527.097914] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 527.104566] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 527.111214] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 527.117864] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 527.124511] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 527.131177] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 527.137827] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 527.144478] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 527.151127] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 527.157776] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 527.164425] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 527.171076] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 527.177723] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 527.184379] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 527.191028] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 527.197683] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 527.204347] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 527.211007] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 527.217656] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 527.224306] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 527.230964] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 527.237616] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 527.244265] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 527.250916] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 527.257563] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 527.264214] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 527.270864] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 527.277515] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 527.284162] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 527.290819] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 527.297468] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 527.304121] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 527.310770] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 527.317426] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 527.324090] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 527.330788] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 527.337443] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 527.344096] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 527.350745] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 527.357407] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 527.364058] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 527.370707] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 527.377353] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 527.384008] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 527.390669] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 527.397321] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 527.403969] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 527.410621] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 527.417270] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 527.423921] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 527.430579] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 527.437236] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 527.443902] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 527.450562] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 527.457210] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 527.463862] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 527.470510] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 527.477160] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 527.483810] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 527.490462] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 527.497111] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 527.503762] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 527.510410] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 527.517059] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 527.523705] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 527.530369] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 527.537018] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 527.543672] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 527.550322] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 527.556980] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 527.563729] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 527.570476] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 527.577212] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 527.583954] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 527.590691] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 527.597427] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 527.604163] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 527.610900] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 527.617636] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 527.624372] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 527.631119] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 527.637858] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 527.644594] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 527.651331] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 527.658067] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 527.664803] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 527.671537] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 527.678279] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 527.685031] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 527.691778] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 527.698515] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 527.705249] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 527.711984] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 527.718721] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 527.725457] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 527.732205] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 527.738941] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 527.745679] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 527.752414] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 527.759150] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 527.765886] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 527.772625] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 527.779359] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 527.786100] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 527.792841] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 527.799598] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 527.806342] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 527.813095] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 527.819833] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 527.826566] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 527.833316] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 527.840052] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 527.846791] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 527.853527] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 527.860261] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 527.866996] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 527.873733] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 527.880466] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 527.887206] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 527.893942] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 527.900679] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 527.907415] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 527.914156] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 527.920906] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 527.927653] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 527.934398] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 527.941138] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 527.947873] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 527.954611] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 527.961346] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 527.968081] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 527.974817] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 527.981555] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 527.988290] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 527.995027] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 528.001762] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 528.008497] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 528.015232] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 528.021969] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 528.028711] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 528.035470] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 528.042219] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 528.048957] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 528.055693] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 528.062432] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 528.069167] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 528.075904] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 528.082638] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 528.089375] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 528.096131] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 528.102872] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 528.109608] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 528.116345] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 528.123080] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 528.129815] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 528.136560] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 528.143299] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 528.150037] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 528.156788] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 528.163534] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 528.170275] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 528.177008] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 528.183749] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 528.190485] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 528.197222] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 528.203955] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 528.210692] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 528.217427] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 528.224163] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 528.230896] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 528.237646] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 528.244382] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 528.251119] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 528.257856] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 528.264597] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 528.271344] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 528.278089] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 528.284825] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 528.291567] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 528.298303] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 528.305037] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 528.311771] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 528.318508] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 528.325240] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 528.331979] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 528.338721] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 528.345459] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 528.352194] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 528.358933] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 528.365666] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 528.372404] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 528.379137] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 528.385880] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 528.392632] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 528.399379] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 528.406116] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 528.412853] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 528.419588] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 528.426326] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 528.433060] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 528.439811] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 528.446548] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 528.453285] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 528.460020] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 528.466756] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 528.473492] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 528.480229] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 528.486963] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 528.493702] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 528.500439] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 528.507191] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 528.513936] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 528.520679] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 528.527412] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 528.534152] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 528.540893] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 528.548254] pci-endpoint-test 0000:01:00.0:
[ 528.548254]
[ 528.548254]
[ 528.548254]
[ 528.556991] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
ffff0004b61f9000 7e951000 910c690d=910c690d
READ ( 2176 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r -s 2176
[ 531.627350] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 531.633879] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 531.640377] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 531.646965] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 531.653531] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 531.660102] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 531.666666] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 531.673230] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 531.679792] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 531.686360] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 531.692924] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 531.699492] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 531.706054] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 531.712618] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 531.719266] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 531.725917] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 531.732565] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 531.739230] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 531.745881] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 531.752536] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 531.759200] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 531.765857] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 531.772508] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 531.779160] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 531.785807] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 531.792460] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 531.799108] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 531.805759] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 531.812406] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 531.819058] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 531.825708] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 531.832359] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 531.839016] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 531.845671] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 531.852320] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 531.858972] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 531.865621] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 531.872275] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 531.878936] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 531.885597] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 531.892243] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 531.898898] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 531.905548] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 531.912199] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 531.918848] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 531.925496] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 531.932146] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 531.938840] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 531.945492] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 531.952146] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 531.958796] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 531.965445] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 531.972094] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 531.978746] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 531.985400] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 531.992058] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 531.998723] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 532.005387] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 532.012035] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 532.018691] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 532.025339] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 532.031989] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 532.038648] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 532.045301] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 532.051949] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 532.058605] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 532.065252] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 532.071908] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 532.078559] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 532.085210] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 532.091857] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 532.098511] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 532.105161] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 532.111816] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 532.118478] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 532.125140] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 532.131787] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 532.138451] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 532.145101] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 532.151753] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 532.158399] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 532.165052] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 532.171702] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 532.178352] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 532.185000] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 532.191653] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 532.198302] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 532.204952] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 532.211601] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 532.218251] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 532.224899] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 532.231550] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 532.238214] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 532.244884] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 532.251546] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 532.258200] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 532.264849] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 532.271501] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 532.278150] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 532.284800] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 532.291446] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 532.298098] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 532.304746] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 532.311397] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 532.318045] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 532.324696] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 532.331344] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 532.338006] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 532.344658] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 532.351322] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 532.357978] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 532.364649] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 532.371311] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 532.377968] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 532.384615] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 532.391267] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 532.397917] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 532.404567] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 532.411217] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 532.417866] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 532.424513] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 532.431167] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 532.437829] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 532.444482] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 532.451131] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 532.457782] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 532.464520] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 532.471260] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 532.477999] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 532.484756] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 532.491502] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 532.498246] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 532.504982] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 532.511720] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 532.518457] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 532.525195] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 532.531933] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 532.538680] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 532.545418] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 532.552155] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 532.558891] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 532.565629] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 532.572365] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 532.579104] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 532.585839] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 532.592583] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 532.599332] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 532.606081] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 532.612818] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 532.619557] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 532.626292] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 532.633027] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 532.639773] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 532.646511] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 532.653247] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 532.659984] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 532.666719] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 532.673454] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 532.680188] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 532.686927] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 532.693661] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 532.700402] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 532.707137] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 532.713876] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 532.720630] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 532.727378] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 532.734114] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 532.740861] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 532.747600] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 532.754338] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 532.761117] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 532.767872] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 532.774621] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 532.781360] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 532.788095] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 532.794838] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 532.801573] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 532.808309] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 532.815045] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 532.821783] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 532.828517] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 532.835255] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 532.842004] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 532.848745] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 532.855480] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 532.862218] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 532.868952] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 532.875691] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 532.882428] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 532.889184] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 532.895928] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 532.902668] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 532.909403] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 532.916142] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 532.922877] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 532.929614] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 532.936348] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 532.943096] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 532.949834] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 532.956572] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 532.963306] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 532.970048] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 532.976784] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 532.983524] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 532.990259] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 532.997002] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 533.003737] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 533.010477] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 533.017228] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 533.023980] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 533.030714] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 533.037454] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 533.044198] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 533.050939] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 533.057674] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 533.064409] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 533.071146] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 533.077883] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 533.084617] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 533.091356] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 533.098092] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 533.104829] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 533.111565] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 533.118303] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 533.125039] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 533.131777] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 533.138517] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 533.145282] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 533.152032] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 533.158772] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 533.165508] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 533.172243] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 533.178978] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 533.185713] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 533.192446] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 533.199184] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 533.205920] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 533.212656] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 533.219391] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 533.226127] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 533.232861] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 533.239603] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 533.246347] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 533.253098] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 533.259836] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 533.266587] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 533.273332] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 533.280072] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 533.286805] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 533.293546] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 533.300281] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 533.307018] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 533.313754] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 533.320491] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 533.327226] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 533.333962] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 533.340696] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 533.347443] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 533.354181] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 533.360917] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 533.367653] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 533.374391] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 533.381129] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 533.387881] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 533.394629] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 533.401368] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 533.408105] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 533.414842] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 533.421576] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 533.428312] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 533.435046] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 533.441785] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 533.449184] pci-endpoint-test 0000:01:00.0:
[ 533.449184]
[ 533.449184]
[ 533.449184]
[ 533.457921] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
ffff0004b61f9000 7e959800 ce535039=910c690d
READ ( 2176 bytes): NOT OKAY
Note: for failure case the crc is always ce535039
Cheers,
--Prabhakar
> Best regards,
> Yoshihiro Shimoda
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: PCIe EPF
2020-03-31 17:25 ` Lad, Prabhakar
@ 2020-04-01 1:25 ` Yoshihiro Shimoda
2020-04-01 8:00 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Yoshihiro Shimoda @ 2020-04-01 1:25 UTC (permalink / raw)
To: Lad, Prabhakar, Kishon Vijay Abraham I
Cc: Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Prabhakar-san,
> From: Lad, Prabhakar, Sent: Wednesday, April 1, 2020 2:26 AM
<snip>
> > > > Did you try to probe the failure further by comparing the hexdumps? Where does
> > > > the mismatch happen?
> > > >
> > > I shall dump the memory and check the values, but basically crc is failing.
> >
> > I'm also interesting about comparing the hexdump between host and ep.
> > This is my gut feeling though, I'm guessing this is a timing issue
> > because using coherent memory API will be on non-cache even if CPU access,
> > but using steaming DMA API will be on cache if CPU access by
> > dma_unmap_single(DMA_FROM_DEVICE) and get_random_bytes() in pci_endpoint_test_write().
> >
> > If my guess is correct, almost all hexdumps are the same, but last
> > some bytes (I'm not sure how much bytes though) are not match.
> >
> So I did some experiments only focusing on read operation for now
>
> 1: I tried printing the crc value in pci_epf_test_write() function
> with below code snippet:
>
> @@ -472,6 +474,9 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
> */
> usleep_range(1000, 2000);
>
> + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
> +
> err_dma_map:
> kfree(buf);
>
> But with this I get :
I don't know why this happen for now.
<snip>
> 2: Instead of get_random_bytes() in pci_epf_test_write() I used
> memset() with the fixed value
<snip>
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -431,6 +431,8 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
> }
>
> get_random_bytes(buf, reg->size);
> + memset(buf, 0xDF, reg->size);
> reg->checksum = crc32_le(~0, buf, reg->size);
>
> And later for the dst buffer I did memset for dst_buffer
>
> @@ -472,6 +474,10 @@ static int pci_epf_test_write(struct pci_epf_test
> *epf_test)
> */
> usleep_range(1000, 2000);
>
> + memset(dst_addr, 0xDF, reg->size);
I think this memset is not needed because this function calls memcpy_toio(dst_addr, buf,..).
> + dev_err(dev, "%s %llx %llx csum: %x\n", __func__, reg->dst_addr,
> + phys_addr, reg->checksum);
> +
> err_dma_map:
> kfree(buf);
>
> But with this I get similar issue as above:
I don't know why like the case 1 for now...
<snip>
>
> 3: After making sure -s was aligned to 128 bytes memset and crc32_le
> in pci_epf_test_write() didnt hit above issues,
> and now when I compared the crc value of both the buffers this
> matched on endpoint!
OK.
> 4: Now focusing more in rootdevice, I added memset(buf, 0xDF,
> reg->size); in pci_epf_test_write() function
> and was dumping the contents of buffer in pci_endpoint_test_read()
> with below function:
>
> static void print_buffer(struct device *dev, void *buff_addr, size_t size)
> {
> size_t i;
> u64 *addr = buff_addr;
>
> for(i = 0; i < size; i += sizeof(addr))
> dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
>
> dev_err(dev, "\n\n\n\n");
> }
>
> Below is the dump from pci_endpoint_test_read() of one passing case
> and other no-passing case:
> The contents of buffer match as expected, but the crc32_le is
> differrent and I am clue-less why is this
> being caused when using non-coherent memory.
>
> root@hihope-rzg2m:~# pcitest -r -s 2176
<snip>
> [ 528.556991] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> ffff0004b61f9000 7e951000 910c690d=910c690d
I'd like to know how to print these crc values. Your report on the case 1
mentioned on pci-epf-test.c like below though.
> + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
Also, as I mentioned on previous email, this is possible to cause timing issue.
So, I'd like to where you added the hexdump on pci_endpoint_test.c.
Perhaps, copy and pasting your whole debug diff is useful to understand about it.
> READ ( 2176 bytes): OKAY
> root@hihope-rzg2m:~# pcitest -r -s 2176
<snip>
> [ 533.457921] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> ffff0004b61f9000 7e959800 ce535039=910c690d
> READ ( 2176 bytes): NOT OKAY
>
> Note: for failure case the crc is always ce535039
The value of ce535039 is from pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM)?
If so, it's strange because all 0xdf values with 2176 bytes should be 910c690d by crc32_le().
Best regards,
Yoshihiro Shimoda
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-04-01 1:25 ` Yoshihiro Shimoda
@ 2020-04-01 8:00 ` Lad, Prabhakar
2020-04-01 8:50 ` Yoshihiro Shimoda
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-04-01 8:00 UTC (permalink / raw)
To: Yoshihiro Shimoda
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Shimoda-san,
On Wed, Apr 1, 2020 at 2:25 AM Yoshihiro Shimoda
<yoshihiro.shimoda.uh@renesas.com> wrote:
>
> Hi Prabhakar-san,
>
> > From: Lad, Prabhakar, Sent: Wednesday, April 1, 2020 2:26 AM
> <snip>
> > > > > Did you try to probe the failure further by comparing the hexdumps? Where does
> > > > > the mismatch happen?
> > > > >
> > > > I shall dump the memory and check the values, but basically crc is failing.
> > >
> > > I'm also interesting about comparing the hexdump between host and ep.
> > > This is my gut feeling though, I'm guessing this is a timing issue
> > > because using coherent memory API will be on non-cache even if CPU access,
> > > but using steaming DMA API will be on cache if CPU access by
> > > dma_unmap_single(DMA_FROM_DEVICE) and get_random_bytes() in pci_endpoint_test_write().
> > >
> > > If my guess is correct, almost all hexdumps are the same, but last
> > > some bytes (I'm not sure how much bytes though) are not match.
> > >
> > So I did some experiments only focusing on read operation for now
> >
> > 1: I tried printing the crc value in pci_epf_test_write() function
> > with below code snippet:
> >
> > @@ -472,6 +474,9 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
> > */
> > usleep_range(1000, 2000);
> >
> > + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> > + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
> > +
> > err_dma_map:
> > kfree(buf);
> >
> > But with this I get :
>
> I don't know why this happen for now.
>
> <snip>
> > 2: Instead of get_random_bytes() in pci_epf_test_write() I used
> > memset() with the fixed value
> <snip>
> > +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> > @@ -431,6 +431,8 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test)
> > }
> >
> > get_random_bytes(buf, reg->size);
> > + memset(buf, 0xDF, reg->size);
> > reg->checksum = crc32_le(~0, buf, reg->size);
> >
> > And later for the dst buffer I did memset for dst_buffer
> >
> > @@ -472,6 +474,10 @@ static int pci_epf_test_write(struct pci_epf_test
> > *epf_test)
> > */
> > usleep_range(1000, 2000);
> >
> > + memset(dst_addr, 0xDF, reg->size);
>
> I think this memset is not needed because this function calls memcpy_toio(dst_addr, buf,..).
>
> > + dev_err(dev, "%s %llx %llx csum: %x\n", __func__, reg->dst_addr,
> > + phys_addr, reg->checksum);
> > +
> > err_dma_map:
> > kfree(buf);
> >
> > But with this I get similar issue as above:
>
> I don't know why like the case 1 for now...
>
> <snip>
> >
> > 3: After making sure -s was aligned to 128 bytes memset and crc32_le
> > in pci_epf_test_write() didnt hit above issues,
> > and now when I compared the crc value of both the buffers this
> > matched on endpoint!
>
> OK.
>
> > 4: Now focusing more in rootdevice, I added memset(buf, 0xDF,
> > reg->size); in pci_epf_test_write() function
> > and was dumping the contents of buffer in pci_endpoint_test_read()
> > with below function:
> >
> > static void print_buffer(struct device *dev, void *buff_addr, size_t size)
> > {
> > size_t i;
> > u64 *addr = buff_addr;
> >
> > for(i = 0; i < size; i += sizeof(addr))
> > dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
> >
> > dev_err(dev, "\n\n\n\n");
> > }
> >
> > Below is the dump from pci_endpoint_test_read() of one passing case
> > and other no-passing case:
> > The contents of buffer match as expected, but the crc32_le is
> > differrent and I am clue-less why is this
> > being caused when using non-coherent memory.
> >
> > root@hihope-rzg2m:~# pcitest -r -s 2176
> <snip>
> > [ 528.556991] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > ffff0004b61f9000 7e951000 910c690d=910c690d
>
> I'd like to know how to print these crc values. Your report on the case 1
> mentioned on pci-epf-test.c like below though.
>
> > + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> > + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
>
> Also, as I mentioned on previous email, this is possible to cause timing issue.
> So, I'd like to where you added the hexdump on pci_endpoint_test.c.
> Perhaps, copy and pasting your whole debug diff is useful to understand about it.
>
Following is the complete diff:
diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
index 3c49514b4813..e7bf58a1fee8 100644
--- a/drivers/misc/pci_endpoint_test.c
+++ b/drivers/misc/pci_endpoint_test.c
@@ -561,6 +561,23 @@ static bool pci_endpoint_test_write(struct
pci_endpoint_test *test,
return ret;
}
+static void print_buffer(struct device *dev, void *buff_addr, size_t size)
+{
+ size_t i;
+ u64 *addr = buff_addr;
+
+ for(i = 0; i < size; i += sizeof(addr))
+ dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
+
+ for(i = 0; i < size; i +=1) {
+ if (0x910c690d == crc32_le(~0, buff_addr, i))
+ dev_err(dev, "%x\n",
+ crc32_le(~0, buff_addr, i));
+ }
+
+ dev_err(dev, "\n\n\n\n");
+}
+
static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
unsigned long arg)
{
@@ -608,7 +625,7 @@ static bool pci_endpoint_test_read(struct
pci_endpoint_test *test,
}
orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
- DMA_FROM_DEVICE);
+ DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, orig_phys_addr)) {
dev_err(dev, "failed to map source buffer address\n");
ret = false;
@@ -640,12 +657,17 @@ static bool pci_endpoint_test_read(struct
pci_endpoint_test *test,
wait_for_completion(&test->irq_raised);
dma_unmap_single(dev, orig_phys_addr, size + alignment,
- DMA_FROM_DEVICE);
+ DMA_BIDIRECTIONAL);
crc32 = crc32_le(~0, addr, size);
if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
ret = true;
+ print_buffer(dev, addr, size);
+ dev_err(dev, "%s %px %llx %x=%x\n", __func__, orig_addr,
+ orig_phys_addr, crc32,
+ pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM));
+
err_phys_addr:
kfree(orig_addr);
err:
Note: I have added DMA_BIDIRECTIONAL that is because I am also
printing the buffer on ep
and calulating the crc
> > READ ( 2176 bytes): OKAY
> > root@hihope-rzg2m:~# pcitest -r -s 2176
> <snip>
> > [ 533.457921] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > ffff0004b61f9000 7e959800 ce535039=910c690d
> > READ ( 2176 bytes): NOT OKAY
> >
> > Note: for failure case the crc is always ce535039
>
> The value of ce535039 is from pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM)?
> If so, it's strange because all 0xdf values with 2176 bytes should be 910c690d by crc32_le().
>
The value ce535039 is the one which is calculated from the buffer and
value 910c690d is the one
which is passed from the BAR memory which is correct.
Cheers,
--Prabhakar
^ permalink raw reply related [flat|nested] 20+ messages in thread
* RE: PCIe EPF
2020-04-01 8:00 ` Lad, Prabhakar
@ 2020-04-01 8:50 ` Yoshihiro Shimoda
2020-04-01 9:24 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Yoshihiro Shimoda @ 2020-04-01 8:50 UTC (permalink / raw)
To: Lad, Prabhakar
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Prabhakar-san,
> From: Lad, Prabhakar, Sent: Wednesday, April 1, 2020 5:00 PM
<snip>
> > > root@hihope-rzg2m:~# pcitest -r -s 2176
> > <snip>
> > > [ 528.556991] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > ffff0004b61f9000 7e951000 910c690d=910c690d
> >
> > I'd like to know how to print these crc values. Your report on the case 1
> > mentioned on pci-epf-test.c like below though.
> >
> > > + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> > > + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
> >
> > Also, as I mentioned on previous email, this is possible to cause timing issue.
> > So, I'd like to where you added the hexdump on pci_endpoint_test.c.
> > Perhaps, copy and pasting your whole debug diff is useful to understand about it.
> >
>
> Following is the complete diff:
Thanks!
> diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> index 3c49514b4813..e7bf58a1fee8 100644
> --- a/drivers/misc/pci_endpoint_test.c
> +++ b/drivers/misc/pci_endpoint_test.c
> @@ -561,6 +561,23 @@ static bool pci_endpoint_test_write(struct
> pci_endpoint_test *test,
> return ret;
> }
>
> +static void print_buffer(struct device *dev, void *buff_addr, size_t size)
> +{
> + size_t i;
> + u64 *addr = buff_addr;
> +
> + for(i = 0; i < size; i += sizeof(addr))
> + dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
> +
> + for(i = 0; i < size; i +=1) {
> + if (0x910c690d == crc32_le(~0, buff_addr, i))
> + dev_err(dev, "%x\n",
> + crc32_le(~0, buff_addr, i));
> + }
> +
> + dev_err(dev, "\n\n\n\n");
> +}
> +
> static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
> unsigned long arg)
> {
> @@ -608,7 +625,7 @@ static bool pci_endpoint_test_read(struct
> pci_endpoint_test *test,
> }
I'd like to allocate temporary buffer here to copy data later...
void *tmp;
...
tmp = kzalloc(size + alignment, GFP_KERNEL);
> orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
> - DMA_FROM_DEVICE);
> + DMA_BIDIRECTIONAL);
> if (dma_mapping_error(dev, orig_phys_addr)) {
> dev_err(dev, "failed to map source buffer address\n");
> ret = false;
> @@ -640,12 +657,17 @@ static bool pci_endpoint_test_read(struct
> pci_endpoint_test *test,
> wait_for_completion(&test->irq_raised);
>
> dma_unmap_single(dev, orig_phys_addr, size + alignment,
> - DMA_FROM_DEVICE);
> + DMA_BIDIRECTIONAL);
And then, I'd like to copy addr buffer to the tmp here.
memcpy(tmp, addr, size);
> crc32 = crc32_le(~0, addr, size);
And addr is replaced with tmp;
> if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
> ret = true;
>
> + print_buffer(dev, addr, size);
This addr is also replaced with tmp;
> + dev_err(dev, "%s %px %llx %x=%x\n", __func__, orig_addr,
> + orig_phys_addr, crc32,
> + pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM));
The crc32 value was calculated before print the buffer.
This means that timing is difference between crc32_le(addr) and print_buffer(addr).
So, I'd like to copy the addr buffer to tmp and use the tmp for crc32_le() and print_buffer().
> +
> err_phys_addr:
> kfree(orig_addr);
> err:
>
> Note: I have added DMA_BIDIRECTIONAL that is because I am also
> printing the buffer on ep
> and calulating the crc
>
> > > READ ( 2176 bytes): OKAY
> > > root@hihope-rzg2m:~# pcitest -r -s 2176
> > <snip>
> > > [ 533.457921] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > ffff0004b61f9000 7e959800 ce535039=910c690d
> > > READ ( 2176 bytes): NOT OKAY
> > >
> > > Note: for failure case the crc is always ce535039
> >
> > The value of ce535039 is from pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM)?
> > If so, it's strange because all 0xdf values with 2176 bytes should be 910c690d by crc32_le().
> >
> The value ce535039 is the one which is calculated from the buffer and
> value 910c690d is the one
> which is passed from the BAR memory which is correct.
I got it. If my guess is correct, using the tmp buffer above can print
the buffer with wrong value(s).
By the way, your PCIe host environment (RZ/G2N) output the following log?
[ 0.000000] software IO TLB: mapped [mem 0x7bfff000-0x7ffff000] (64MB)
If so, I guess this issue is related to this software IO TLB behavior.
IIUC, if we use coherent buffer or GPF_DMA buffer, the kernel will not
use software IO TLB for these buffers.
Best regards,
Yoshihiro Shimoda
> Cheers,
> --Prabhakar
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-04-01 8:50 ` Yoshihiro Shimoda
@ 2020-04-01 9:24 ` Lad, Prabhakar
2020-04-02 1:22 ` Yoshihiro Shimoda
0 siblings, 1 reply; 20+ messages in thread
From: Lad, Prabhakar @ 2020-04-01 9:24 UTC (permalink / raw)
To: Yoshihiro Shimoda
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
HI Shimoda-san,
On Wed, Apr 1, 2020 at 9:50 AM Yoshihiro Shimoda
<yoshihiro.shimoda.uh@renesas.com> wrote:
>
> Hi Prabhakar-san,
>
> > From: Lad, Prabhakar, Sent: Wednesday, April 1, 2020 5:00 PM
> <snip>
> > > > root@hihope-rzg2m:~# pcitest -r -s 2176
> > > <snip>
> > > > [ 528.556991] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > ffff0004b61f9000 7e951000 910c690d=910c690d
> > >
> > > I'd like to know how to print these crc values. Your report on the case 1
> > > mentioned on pci-epf-test.c like below though.
> > >
> > > > + dev_err(dev, "%s %llx %llx csum: %x = %x\n", __func__, reg->dst_addr,
> > > > + phys_addr, reg->checksum, crc32_le(~0, dst_addr, reg->size));
> > >
> > > Also, as I mentioned on previous email, this is possible to cause timing issue.
> > > So, I'd like to where you added the hexdump on pci_endpoint_test.c.
> > > Perhaps, copy and pasting your whole debug diff is useful to understand about it.
> > >
> >
> > Following is the complete diff:
>
> Thanks!
>
> > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > index 3c49514b4813..e7bf58a1fee8 100644
> > --- a/drivers/misc/pci_endpoint_test.c
> > +++ b/drivers/misc/pci_endpoint_test.c
> > @@ -561,6 +561,23 @@ static bool pci_endpoint_test_write(struct
> > pci_endpoint_test *test,
> > return ret;
> > }
> >
> > +static void print_buffer(struct device *dev, void *buff_addr, size_t size)
> > +{
> > + size_t i;
> > + u64 *addr = buff_addr;
> > +
> > + for(i = 0; i < size; i += sizeof(addr))
> > + dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
> > +
> > + for(i = 0; i < size; i +=1) {
> > + if (0x910c690d == crc32_le(~0, buff_addr, i))
> > + dev_err(dev, "%x\n",
> > + crc32_le(~0, buff_addr, i));
> > + }
> > +
> > + dev_err(dev, "\n\n\n\n");
> > +}
> > +
> > static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
> > unsigned long arg)
> > {
> > @@ -608,7 +625,7 @@ static bool pci_endpoint_test_read(struct
> > pci_endpoint_test *test,
> > }
>
> I'd like to allocate temporary buffer here to copy data later...
> void *tmp;
> ...
> tmp = kzalloc(size + alignment, GFP_KERNEL);
>
> > orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
> > - DMA_FROM_DEVICE);
> > + DMA_BIDIRECTIONAL);
> > if (dma_mapping_error(dev, orig_phys_addr)) {
> > dev_err(dev, "failed to map source buffer address\n");
> > ret = false;
> > @@ -640,12 +657,17 @@ static bool pci_endpoint_test_read(struct
> > pci_endpoint_test *test,
> > wait_for_completion(&test->irq_raised);
> >
> > dma_unmap_single(dev, orig_phys_addr, size + alignment,
> > - DMA_FROM_DEVICE);
> > + DMA_BIDIRECTIONAL);
>
> And then, I'd like to copy addr buffer to the tmp here.
> memcpy(tmp, addr, size);
>
> > crc32 = crc32_le(~0, addr, size);
>
> And addr is replaced with tmp;
>
> > if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
> > ret = true;
> >
> > + print_buffer(dev, addr, size);
>
> This addr is also replaced with tmp;
>
> > + dev_err(dev, "%s %px %llx %x=%x\n", __func__, orig_addr,
> > + orig_phys_addr, crc32,
> > + pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM));
>
> The crc32 value was calculated before print the buffer.
> This means that timing is difference between crc32_le(addr) and print_buffer(addr).
> So, I'd like to copy the addr buffer to tmp and use the tmp for crc32_le() and print_buffer().
>
Following is the diff of suggested changes:
diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
index 3c49514b4813..b782134838d1 100644
--- a/drivers/misc/pci_endpoint_test.c
+++ b/drivers/misc/pci_endpoint_test.c
@@ -561,6 +561,23 @@ static bool pci_endpoint_test_write(struct
pci_endpoint_test *test,
return ret;
}
+static void print_buffer(struct device *dev, void *buff_addr, size_t size)
+{
+ size_t i;
+ u64 *addr = buff_addr;
+
+ for(i = 0; i < size; i += sizeof(addr))
+ dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
+
+ for(i = 0; i < size; i +=1) {
+ if (0x910c690d == crc32_le(~0, buff_addr, i))
+ dev_err(dev, "%x\n",
+ crc32_le(~0, buff_addr, i));
+ }
+
+ dev_err(dev, "\n\n\n\n");
+}
+
static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
unsigned long arg)
{
@@ -574,6 +591,7 @@ static bool pci_endpoint_test_read(struct
pci_endpoint_test *test,
struct pci_dev *pdev = test->pdev;
struct device *dev = &pdev->dev;
void *orig_addr;
+ void *tmp_buffer;
dma_addr_t orig_phys_addr;
size_t offset;
size_t alignment = test->alignment;
@@ -601,14 +619,19 @@ static bool pci_endpoint_test_read(struct
pci_endpoint_test *test,
}
orig_addr = kzalloc(size + alignment, GFP_KERNEL);
- if (!orig_addr) {
+ tmp_buffer = kzalloc(size + alignment, GFP_KERNEL);
+ if (!orig_addr || !tmp_buffer) {
dev_err(dev, "Failed to allocate destination address\n");
ret = false;
+ if (orig_addr)
+ kfree(orig_addr);
+ if (tmp_buffer)
+ kfree(tmp_buffer);
goto err;
}
orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
- DMA_FROM_DEVICE);
+ DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, orig_phys_addr)) {
dev_err(dev, "failed to map source buffer address\n");
ret = false;
@@ -640,14 +663,24 @@ static bool pci_endpoint_test_read(struct
pci_endpoint_test *test,
wait_for_completion(&test->irq_raised);
dma_unmap_single(dev, orig_phys_addr, size + alignment,
- DMA_FROM_DEVICE);
+ DMA_BIDIRECTIONAL);
+
+ memcpy(tmp_buffer, addr, size);
crc32 = crc32_le(~0, addr, size);
if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
ret = true;
+ print_buffer(dev, addr, size);
+ print_buffer(dev, tmp_buffer, size);
+ dev_err(dev, "%s kzalloc:%px dma handle:%llx buffer CRC:%x BAR
CRC:%x tmp buffer crc: %x\n",
+ __func__, orig_addr, orig_phys_addr, crc32,
+ pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM),
+ crc32_le(~0, tmp_buffer, size));
+
err_phys_addr:
kfree(orig_addr);
+ kfree(tmp_buffer);
err:
return ret;
}
> > +
> > err_phys_addr:
> > kfree(orig_addr);
> > err:
> >
> > Note: I have added DMA_BIDIRECTIONAL that is because I am also
> > printing the buffer on ep
> > and calulating the crc
> >
> > > > READ ( 2176 bytes): OKAY
> > > > root@hihope-rzg2m:~# pcitest -r -s 2176
> > > <snip>
> > > > [ 533.457921] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> > > > ffff0004b61f9000 7e959800 ce535039=910c690d
> > > > READ ( 2176 bytes): NOT OKAY
> > > >
> > > > Note: for failure case the crc is always ce535039
> > >
> > > The value of ce535039 is from pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM)?
> > > If so, it's strange because all 0xdf values with 2176 bytes should be 910c690d by crc32_le().
> > >
> > The value ce535039 is the one which is calculated from the buffer and
> > value 910c690d is the one
> > which is passed from the BAR memory which is correct.
>
> I got it. If my guess is correct, using the tmp buffer above can print
> the buffer with wrong value(s).
>
>
> By the way, your PCIe host environment (RZ/G2N) output the following log?
>
> [ 0.000000] software IO TLB: mapped [mem 0x7bfff000-0x7ffff000] (64MB)
>
> If so, I guess this issue is related to this software IO TLB behavior.
> IIUC, if we use coherent buffer or GPF_DMA buffer, the kernel will not
> use software IO TLB for these buffers.
>
root@hihope-rzg2m:~# dmesg | grep TLB
[ 0.000000] software IO TLB: mapped [mem 0x7bfff000-0x7ffff000] (64MB)
[ 0.149808] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.149825] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.149834] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.149843] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
root@hihope-rzg2m:~#
root@hihope-rzg2m:~#
root@hihope-rzg2m:~#
root@hihope-rzg2m:~#
root@hihope-rzg2m:~# pcitest -i1
SET IRQ TYPE TO MSI: OKAY
root@hihope-rzg2m:~#
root@hihope-rzg2m:~#
root@hihope-rzg2m:~# cat read.sh
pcitest -i 1
while [ 1 ]; do pcitest -r -s 2176; done
root@hihope-rzg2m:~# pcitest -r -s 2176
[ 77.863719] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 77.870299] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 77.876800] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 77.883563] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 77.890241] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 77.896837] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 77.903406] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 77.909971] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 77.916534] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 77.923110] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 77.929672] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 77.936242] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 77.942808] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 77.949374] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 77.956023] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 77.962675] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 77.969323] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 77.975977] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 77.982626] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 77.989281] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 77.995944] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 78.002602] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 78.009253] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 78.015905] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 78.022563] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 78.029214] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 78.035863] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 78.042515] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 78.049165] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 78.055818] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 78.062466] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 78.069116] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 78.075768] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 78.082418] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 78.089066] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 78.095720] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 78.102371] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 78.109025] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 78.115686] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 78.122356] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 78.129003] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 78.135658] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 78.142306] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 78.148958] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 78.155607] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 78.162257] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 78.168905] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 78.175554] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 78.182204] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 78.188854] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 78.195504] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 78.202154] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 78.208804] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 78.215457] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 78.222113] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 78.228769] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 78.235428] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 78.242089] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 78.248741] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 78.255394] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 78.262042] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 78.268692] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 78.275342] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 78.281992] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 78.288639] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 78.295291] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 78.301939] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 78.308587] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 78.315234] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 78.321895] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 78.328545] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 78.335195] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 78.341843] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 78.348496] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 78.355156] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 78.361815] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 78.368462] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 78.375117] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 78.381764] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 78.388416] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 78.395063] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 78.401717] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 78.408366] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 78.415023] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 78.421677] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 78.428329] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 78.434977] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 78.441630] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 78.448279] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 78.454929] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 78.461578] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 78.468231] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 78.474890] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 78.481548] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 78.488199] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 78.494852] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 78.501500] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 78.508150] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 78.514797] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 78.521470] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 78.528122] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 78.534774] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 78.541423] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 78.548070] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 78.554720] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 78.561367] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 78.568017] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 78.574667] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 78.581317] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 78.587964] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 78.594623] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 78.601286] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 78.607946] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 78.614596] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 78.621246] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 78.627902] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 78.634553] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 78.641203] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 78.647852] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 78.654499] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 78.661148] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 78.667794] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 78.674445] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 78.681094] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 78.687746] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 78.694395] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 78.701133] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 78.707870] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 78.714613] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 78.721363] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 78.728119] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 78.734856] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 78.741595] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 78.748330] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 78.755069] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 78.761806] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 78.768543] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 78.775277] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 78.782015] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 78.788749] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 78.795487] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 78.802221] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 78.808957] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 78.815692] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 78.822428] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 78.829173] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 78.835925] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 78.842670] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 78.849531] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 78.856287] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 78.863031] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 78.869767] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 78.876506] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 78.883240] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 78.889983] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 78.896719] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 78.903459] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 78.910194] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 78.916932] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 78.923667] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 78.930414] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 78.937149] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 78.943892] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 78.950628] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 78.957367] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 78.964102] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 78.970846] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 78.977597] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 78.984342] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 78.991078] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 78.997815] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 79.004550] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 79.011288] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 79.018023] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 79.024762] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 79.031504] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 79.038243] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 79.044978] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 79.051715] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 79.058450] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 79.065187] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 79.071922] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 79.078662] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 79.085397] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 79.092137] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 79.098874] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 79.105628] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 79.112375] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 79.119116] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 79.125850] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 79.132599] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 79.139336] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 79.146072] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 79.152808] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 79.159545] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 79.166279] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 79.173015] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 79.179754] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 79.186494] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 79.193230] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 79.199968] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 79.206703] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 79.213442] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 79.220214] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 79.226952] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 79.233700] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 79.240436] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 79.247173] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 79.253907] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 79.260642] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 79.267376] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 79.274110] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 79.280845] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 79.287580] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 79.294315] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 79.301050] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 79.307786] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 79.314524] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 79.321260] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 79.327998] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 79.334742] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 79.341490] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 79.348239] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 79.354979] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 79.361718] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 79.368454] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 79.375192] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 79.381928] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 79.388663] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 79.395399] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 79.402133] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 79.408868] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 79.415606] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 79.422340] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 79.429075] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 79.435814] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 79.442551] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 79.449289] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 79.456027] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 79.462769] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 79.469524] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 79.476272] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 79.483017] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 79.489754] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 79.496490] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 79.503224] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 79.509960] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 79.516696] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 79.523434] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 79.530172] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 79.536915] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 79.543650] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 79.550387] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 79.557121] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 79.563857] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 79.570592] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 79.577326] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 79.584061] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 79.590803] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 79.597553] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 79.604303] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 79.611041] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 79.617780] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 79.624514] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 79.631252] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 79.637992] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 79.644729] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 79.651464] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 79.658202] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 79.664939] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 79.671678] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 79.678415] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 79.685812] pci-endpoint-test 0000:01:00.0:
[ 79.685812]
[ 79.685812]
[ 79.685812]
[ 79.694542] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 79.701018] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 79.707504] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 79.714066] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 79.720629] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 79.727189] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 79.733753] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 79.740319] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 79.746885] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 79.753461] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 79.760026] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 79.766590] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 79.773154] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 79.779717] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 79.786369] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 79.793017] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 79.799667] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 79.806318] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 79.812985] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 79.819643] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 79.826296] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 79.832942] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 79.839670] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 79.846336] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 79.852996] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 79.859644] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 79.866297] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 79.872946] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 79.879602] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 79.886253] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 79.892902] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 79.899550] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 79.906199] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 79.912850] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 79.919502] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 79.926150] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 79.932801] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 79.939455] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 79.946107] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 79.952755] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 79.959406] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 79.966060] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 79.972723] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 79.979379] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 79.986028] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 79.992675] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 79.999328] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 80.005976] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 80.012627] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 80.019277] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 80.025928] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 80.032577] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 80.039236] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 80.045885] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 80.052535] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 80.059185] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 80.065837] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 80.072487] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 80.079136] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 80.085786] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 80.092451] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 80.099110] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 80.105764] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 80.112413] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 80.119068] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 80.125718] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 80.132368] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 80.139021] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 80.145670] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 80.152319] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 80.158968] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 80.165616] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 80.172265] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 80.178915] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 80.185564] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 80.192212] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 80.198862] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 80.205512] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 80.212179] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 80.218839] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 80.225493] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 80.232144] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 80.238801] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 80.245463] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 80.252113] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 80.258759] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 80.265411] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 80.272060] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 80.278710] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 80.285359] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 80.292010] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 80.298659] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 80.305308] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 80.311958] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 80.318606] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 80.325257] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 80.331919] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 80.338586] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 80.345241] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 80.351890] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 80.358541] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 80.365189] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 80.371838] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 80.378489] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 80.385138] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 80.391788] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 80.398439] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 80.405087] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 80.411737] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 80.418388] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 80.425039] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 80.431688] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 80.438346] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 80.444997] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 80.451661] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 80.458321] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 80.464972] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 80.471620] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 80.478267] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 80.484918] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 80.491567] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 80.498215] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 80.504863] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 80.511510] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 80.518160] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 80.524807] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 80.531550] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 80.538291] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 80.545028] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 80.551765] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 80.558503] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 80.565243] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 80.571995] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 80.578742] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 80.585493] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 80.592230] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 80.598968] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 80.605704] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 80.612441] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 80.619177] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 80.625912] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 80.632648] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 80.639394] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 80.646129] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 80.652864] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 80.659599] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 80.666336] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 80.673070] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 80.679806] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 80.686544] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 80.693297] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 80.700044] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 80.706783] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 80.713519] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 80.720255] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 80.726994] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 80.733728] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 80.740468] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 80.747206] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 80.753941] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 80.760677] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 80.767415] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 80.774151] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 80.780883] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 80.787624] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 80.794358] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 80.801103] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 80.807851] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 80.814600] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 80.821339] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 80.828079] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 80.834870] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 80.841635] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 80.848386] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 80.855127] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 80.861864] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 80.868603] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 80.875337] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 80.882075] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 80.888810] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 80.895549] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 80.902290] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 80.909029] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 80.915765] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 80.922501] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 80.929238] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 80.935974] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 80.942716] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 80.949466] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 80.956203] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 80.962953] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 80.969694] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 80.976431] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 80.983166] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 80.989903] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 80.996638] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 81.003373] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 81.010107] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 81.016846] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 81.023582] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 81.030318] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 81.037054] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 81.043799] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 81.050534] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 81.057270] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 81.064003] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 81.070744] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 81.077483] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 81.084239] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 81.090985] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 81.097722] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 81.104459] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 81.111195] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 81.117930] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 81.124667] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 81.131403] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 81.138138] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 81.144878] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 81.151616] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 81.158352] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 81.165087] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 81.171824] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 81.178560] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 81.185293] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 81.192033] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 81.198770] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 81.205522] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 81.212272] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 81.219015] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 81.226317] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 81.233133] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 81.239882] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 81.246644] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 81.253382] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 81.260121] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 81.266856] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 81.273593] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 81.280331] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 81.287070] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 81.293805] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 81.300545] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 81.307280] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 81.314016] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 81.320750] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 81.327488] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 81.334221] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 81.340965] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 81.347722] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 81.354468] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 81.361205] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 81.367945] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 81.374679] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 81.381417] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 81.388150] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 81.394891] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 81.401624] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 81.408362] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 81.415096] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 81.421833] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 81.428570] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 81.435309] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 81.442045] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 81.448790] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 81.455527] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 81.462270] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 81.469020] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 81.475770] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 81.482505] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 81.489245] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 81.495981] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 81.502719] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 81.509464] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 81.516824] pci-endpoint-test 0000:01:00.0:
[ 81.516824]
[ 81.516824]
[ 81.516824]
[ 81.525562] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b6392000 dma handle:7e92f000 buffer CRC:910c690d BAR
CRC:910c690d tmp buffer crc: 910c690d
READ ( 2176 bytes): OKAY
root@hihope-rzg2m:~# pcitest -r -s 2176
[ 85.302372] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 85.308869] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 85.315361] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 85.321949] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 85.328517] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 85.335087] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 85.341649] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 85.348216] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 85.354778] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 85.361341] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 85.367902] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 85.374465] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 85.381028] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 85.387593] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 85.394241] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 85.400892] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 85.407547] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 85.414198] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 85.420846] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 85.427499] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 85.434152] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 85.440820] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 85.447475] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 85.454123] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 85.460773] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 85.467428] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 85.474076] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 85.480729] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 85.487377] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 85.494030] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 85.500678] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 85.507335] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 85.513985] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 85.520636] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 85.527285] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 85.533937] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 85.540586] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 85.547239] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 85.553891] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 85.560556] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 85.567214] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 85.573872] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 85.580520] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 85.587171] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 85.593818] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 85.600469] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 85.607124] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 85.613775] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 85.620425] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 85.627078] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 85.633724] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 85.640374] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 85.647025] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 85.653676] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 85.660324] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 85.666976] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 85.673626] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 85.680293] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 85.686950] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 85.693607] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 85.700257] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 85.706914] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 85.713563] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 85.720213] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 85.726861] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 85.733509] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 85.740384] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 85.747076] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 85.753738] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 85.760395] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 85.767043] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 85.773698] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 85.780346] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 85.786996] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 85.793643] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 85.800292] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 85.806951] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 85.813601] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 85.820249] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 85.826973] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 85.833639] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 85.840303] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 85.846956] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 85.853611] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 85.860261] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 85.866910] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 85.873558] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 85.880206] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 85.886855] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 85.893505] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 85.900154] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 85.906812] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 85.913471] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 85.920124] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 85.926774] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 85.933420] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 85.940072] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 85.946731] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 85.953395] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 85.960057] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 85.966707] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 85.973356] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 85.980006] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 85.986655] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 85.993301] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 85.999952] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 86.006700] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 86.013372] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 86.020105] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 86.026777] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 86.033434] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 86.040091] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 86.046752] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 86.053405] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 86.060055] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 86.066706] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 86.073355] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 86.080003] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 86.086650] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 86.093302] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 86.099948] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 86.106611] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 86.113262] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 86.119915] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 86.126563] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 86.133212] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 86.139949] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 86.146690] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 86.153440] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 86.160197] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 86.166933] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 86.173676] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 86.180413] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 86.187150] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 86.193886] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 86.200626] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 86.207367] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 86.214106] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 86.220842] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 86.227579] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 86.234316] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 86.241054] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 86.247886] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 86.254636] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 86.261372] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 86.268111] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 86.274849] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 86.281604] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 86.288348] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 86.295092] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 86.301828] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 86.308574] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 86.315310] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 86.322045] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 86.328779] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 86.335517] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 86.342254] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 86.348988] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 86.355723] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 86.362460] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 86.369194] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 86.375936] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 86.382672] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 86.389410] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 86.396147] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 86.402892] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 86.409819] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 86.416585] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 86.423320] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 86.430065] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 86.436800] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 86.443542] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 86.450277] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 86.457014] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 86.463749] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 86.470485] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 86.477220] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 86.483956] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 86.490689] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 86.497427] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 86.504163] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 86.510911] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 86.517647] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 86.524388] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 86.531135] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 86.537881] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 86.544614] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 86.551357] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 86.558095] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 86.564831] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 86.571565] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 86.578302] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 86.585036] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 86.591774] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 86.598511] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 86.605246] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 86.611986] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 86.618723] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 86.625468] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 86.632205] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 86.638940] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 86.645683] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 86.652434] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 86.659181] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 86.665916] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 86.672656] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 86.679392] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 86.686129] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 86.692866] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 86.699602] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 86.706338] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 86.713079] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 86.719815] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 86.726552] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 86.733287] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 86.740025] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 86.746759] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 86.753496] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 86.760232] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 86.766980] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 86.773725] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 86.780464] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 86.787198] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 86.793936] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 86.800672] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 86.807409] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 86.814149] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 86.820888] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 86.827625] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 86.834364] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 86.841102] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 86.847895] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 86.854647] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 86.861395] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 86.868131] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 86.874873] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 86.881608] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 86.888345] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 86.895078] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 86.901816] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 86.908549] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 86.915295] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 86.922031] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 86.928770] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 86.935505] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 86.942243] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 86.948978] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 86.955715] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 86.962449] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 86.969189] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 86.975935] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 86.982682] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 86.989416] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 86.996157] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 87.002891] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 87.009628] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 87.016367] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 87.023106] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 87.029841] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 87.036577] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 87.043331] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 87.050072] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 87.056804] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 87.063544] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 87.070279] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 87.077016] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 87.083753] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 87.090494] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 87.097245] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 87.103994] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 87.110730] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 87.117488] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 87.124845] pci-endpoint-test 0000:01:00.0:
[ 87.124845]
[ 87.124845]
[ 87.124845]
[ 87.133581] pci-endpoint-test 0000:01:00.0: buf[0] : dfdfdfdfdfdfdfdf
[ 87.140059] pci-endpoint-test 0000:01:00.0: buf[8] : dfdfdfdfdfdfdfdf
[ 87.146537] pci-endpoint-test 0000:01:00.0: buf[16] : dfdfdfdfdfdfdfdf
[ 87.153099] pci-endpoint-test 0000:01:00.0: buf[24] : dfdfdfdfdfdfdfdf
[ 87.159663] pci-endpoint-test 0000:01:00.0: buf[32] : dfdfdfdfdfdfdfdf
[ 87.166226] pci-endpoint-test 0000:01:00.0: buf[40] : dfdfdfdfdfdfdfdf
[ 87.172787] pci-endpoint-test 0000:01:00.0: buf[48] : dfdfdfdfdfdfdfdf
[ 87.179348] pci-endpoint-test 0000:01:00.0: buf[56] : dfdfdfdfdfdfdfdf
[ 87.185912] pci-endpoint-test 0000:01:00.0: buf[64] : dfdfdfdfdfdfdfdf
[ 87.192477] pci-endpoint-test 0000:01:00.0: buf[72] : dfdfdfdfdfdfdfdf
[ 87.199042] pci-endpoint-test 0000:01:00.0: buf[80] : dfdfdfdfdfdfdfdf
[ 87.205606] pci-endpoint-test 0000:01:00.0: buf[88] : dfdfdfdfdfdfdfdf
[ 87.212186] pci-endpoint-test 0000:01:00.0: buf[96] : dfdfdfdfdfdfdfdf
[ 87.218765] pci-endpoint-test 0000:01:00.0: buf[104] : dfdfdfdfdfdfdfdf
[ 87.225420] pci-endpoint-test 0000:01:00.0: buf[112] : dfdfdfdfdfdfdfdf
[ 87.232069] pci-endpoint-test 0000:01:00.0: buf[120] : dfdfdfdfdfdfdfdf
[ 87.238720] pci-endpoint-test 0000:01:00.0: buf[128] : dfdfdfdfdfdfdfdf
[ 87.245370] pci-endpoint-test 0000:01:00.0: buf[136] : dfdfdfdfdfdfdfdf
[ 87.252021] pci-endpoint-test 0000:01:00.0: buf[144] : dfdfdfdfdfdfdfdf
[ 87.258669] pci-endpoint-test 0000:01:00.0: buf[152] : dfdfdfdfdfdfdfdf
[ 87.265319] pci-endpoint-test 0000:01:00.0: buf[160] : dfdfdfdfdfdfdfdf
[ 87.271969] pci-endpoint-test 0000:01:00.0: buf[168] : dfdfdfdfdfdfdfdf
[ 87.278618] pci-endpoint-test 0000:01:00.0: buf[176] : dfdfdfdfdfdfdfdf
[ 87.285266] pci-endpoint-test 0000:01:00.0: buf[184] : dfdfdfdfdfdfdfdf
[ 87.291919] pci-endpoint-test 0000:01:00.0: buf[192] : dfdfdfdfdfdfdfdf
[ 87.298568] pci-endpoint-test 0000:01:00.0: buf[200] : dfdfdfdfdfdfdfdf
[ 87.305217] pci-endpoint-test 0000:01:00.0: buf[208] : dfdfdfdfdfdfdfdf
[ 87.311866] pci-endpoint-test 0000:01:00.0: buf[216] : dfdfdfdfdfdfdfdf
[ 87.318523] pci-endpoint-test 0000:01:00.0: buf[224] : dfdfdfdfdfdfdfdf
[ 87.325178] pci-endpoint-test 0000:01:00.0: buf[232] : dfdfdfdfdfdfdfdf
[ 87.331843] pci-endpoint-test 0000:01:00.0: buf[240] : dfdfdfdfdfdfdfdf
[ 87.338500] pci-endpoint-test 0000:01:00.0: buf[248] : dfdfdfdfdfdfdfdf
[ 87.345152] pci-endpoint-test 0000:01:00.0: buf[256] : dfdfdfdfdfdfdfdf
[ 87.351800] pci-endpoint-test 0000:01:00.0: buf[264] : dfdfdfdfdfdfdfdf
[ 87.358451] pci-endpoint-test 0000:01:00.0: buf[272] : dfdfdfdfdfdfdfdf
[ 87.365099] pci-endpoint-test 0000:01:00.0: buf[280] : dfdfdfdfdfdfdfdf
[ 87.371748] pci-endpoint-test 0000:01:00.0: buf[288] : dfdfdfdfdfdfdfdf
[ 87.378396] pci-endpoint-test 0000:01:00.0: buf[296] : dfdfdfdfdfdfdfdf
[ 87.385046] pci-endpoint-test 0000:01:00.0: buf[304] : dfdfdfdfdfdfdfdf
[ 87.391696] pci-endpoint-test 0000:01:00.0: buf[312] : dfdfdfdfdfdfdfdf
[ 87.398345] pci-endpoint-test 0000:01:00.0: buf[320] : dfdfdfdfdfdfdfdf
[ 87.404994] pci-endpoint-test 0000:01:00.0: buf[328] : dfdfdfdfdfdfdfdf
[ 87.411645] pci-endpoint-test 0000:01:00.0: buf[336] : dfdfdfdfdfdfdfdf
[ 87.418299] pci-endpoint-test 0000:01:00.0: buf[344] : dfdfdfdfdfdfdfdf
[ 87.424948] pci-endpoint-test 0000:01:00.0: buf[352] : dfdfdfdfdfdfdfdf
[ 87.431597] pci-endpoint-test 0000:01:00.0: buf[360] : dfdfdfdfdfdfdfdf
[ 87.438247] pci-endpoint-test 0000:01:00.0: buf[368] : dfdfdfdfdfdfdfdf
[ 87.444901] pci-endpoint-test 0000:01:00.0: buf[376] : dfdfdfdfdfdfdfdf
[ 87.451571] pci-endpoint-test 0000:01:00.0: buf[384] : dfdfdfdfdfdfdfdf
[ 87.458229] pci-endpoint-test 0000:01:00.0: buf[392] : dfdfdfdfdfdfdfdf
[ 87.464881] pci-endpoint-test 0000:01:00.0: buf[400] : dfdfdfdfdfdfdfdf
[ 87.471529] pci-endpoint-test 0000:01:00.0: buf[408] : dfdfdfdfdfdfdfdf
[ 87.478179] pci-endpoint-test 0000:01:00.0: buf[416] : dfdfdfdfdfdfdfdf
[ 87.484829] pci-endpoint-test 0000:01:00.0: buf[424] : dfdfdfdfdfdfdfdf
[ 87.491479] pci-endpoint-test 0000:01:00.0: buf[432] : dfdfdfdfdfdfdfdf
[ 87.498124] pci-endpoint-test 0000:01:00.0: buf[440] : dfdfdfdfdfdfdfdf
[ 87.504779] pci-endpoint-test 0000:01:00.0: buf[448] : dfdfdfdfdfdfdfdf
[ 87.511428] pci-endpoint-test 0000:01:00.0: buf[456] : dfdfdfdfdfdfdfdf
[ 87.518087] pci-endpoint-test 0000:01:00.0: buf[464] : dfdfdfdfdfdfdfdf
[ 87.524736] pci-endpoint-test 0000:01:00.0: buf[472] : dfdfdfdfdfdfdfdf
[ 87.531385] pci-endpoint-test 0000:01:00.0: buf[480] : dfdfdfdfdfdfdfdf
[ 87.538037] pci-endpoint-test 0000:01:00.0: buf[488] : dfdfdfdfdfdfdfdf
[ 87.544688] pci-endpoint-test 0000:01:00.0: buf[496] : dfdfdfdfdfdfdfdf
[ 87.551337] pci-endpoint-test 0000:01:00.0: buf[504] : dfdfdfdfdfdfdfdf
[ 87.557987] pci-endpoint-test 0000:01:00.0: buf[512] : dfdfdfdfdfdfdfdf
[ 87.564637] pci-endpoint-test 0000:01:00.0: buf[520] : dfdfdfdfdfdfdfdf
[ 87.571301] pci-endpoint-test 0000:01:00.0: buf[528] : dfdfdfdfdfdfdfdf
[ 87.577960] pci-endpoint-test 0000:01:00.0: buf[536] : dfdfdfdfdfdfdfdf
[ 87.584612] pci-endpoint-test 0000:01:00.0: buf[544] : dfdfdfdfdfdfdfdf
[ 87.591259] pci-endpoint-test 0000:01:00.0: buf[552] : dfdfdfdfdfdfdfdf
[ 87.597911] pci-endpoint-test 0000:01:00.0: buf[560] : dfdfdfdfdfdfdfdf
[ 87.604559] pci-endpoint-test 0000:01:00.0: buf[568] : dfdfdfdfdfdfdfdf
[ 87.611209] pci-endpoint-test 0000:01:00.0: buf[576] : dfdfdfdfdfdfdfdf
[ 87.617863] pci-endpoint-test 0000:01:00.0: buf[584] : dfdfdfdfdfdfdfdf
[ 87.624514] pci-endpoint-test 0000:01:00.0: buf[592] : dfdfdfdfdfdfdfdf
[ 87.631164] pci-endpoint-test 0000:01:00.0: buf[600] : dfdfdfdfdfdfdfdf
[ 87.637812] pci-endpoint-test 0000:01:00.0: buf[608] : dfdfdfdfdfdfdfdf
[ 87.644461] pci-endpoint-test 0000:01:00.0: buf[616] : dfdfdfdfdfdfdfdf
[ 87.651109] pci-endpoint-test 0000:01:00.0: buf[624] : dfdfdfdfdfdfdfdf
[ 87.657757] pci-endpoint-test 0000:01:00.0: buf[632] : dfdfdfdfdfdfdfdf
[ 87.664404] pci-endpoint-test 0000:01:00.0: buf[640] : dfdfdfdfdfdfdfdf
[ 87.671051] pci-endpoint-test 0000:01:00.0: buf[648] : dfdfdfdfdfdfdfdf
[ 87.677701] pci-endpoint-test 0000:01:00.0: buf[656] : dfdfdfdfdfdfdfdf
[ 87.684351] pci-endpoint-test 0000:01:00.0: buf[664] : dfdfdfdfdfdfdfdf
[ 87.691014] pci-endpoint-test 0000:01:00.0: buf[672] : dfdfdfdfdfdfdfdf
[ 87.697674] pci-endpoint-test 0000:01:00.0: buf[680] : dfdfdfdfdfdfdfdf
[ 87.704327] pci-endpoint-test 0000:01:00.0: buf[688] : dfdfdfdfdfdfdfdf
[ 87.710973] pci-endpoint-test 0000:01:00.0: buf[696] : dfdfdfdfdfdfdfdf
[ 87.717630] pci-endpoint-test 0000:01:00.0: buf[704] : dfdfdfdfdfdfdfdf
[ 87.724280] pci-endpoint-test 0000:01:00.0: buf[712] : dfdfdfdfdfdfdfdf
[ 87.730932] pci-endpoint-test 0000:01:00.0: buf[720] : dfdfdfdfdfdfdfdf
[ 87.737578] pci-endpoint-test 0000:01:00.0: buf[728] : dfdfdfdfdfdfdfdf
[ 87.744230] pci-endpoint-test 0000:01:00.0: buf[736] : dfdfdfdfdfdfdfdf
[ 87.750878] pci-endpoint-test 0000:01:00.0: buf[744] : dfdfdfdfdfdfdfdf
[ 87.757527] pci-endpoint-test 0000:01:00.0: buf[752] : dfdfdfdfdfdfdfdf
[ 87.764174] pci-endpoint-test 0000:01:00.0: buf[760] : dfdfdfdfdfdfdfdf
[ 87.770823] pci-endpoint-test 0000:01:00.0: buf[768] : dfdfdfdfdfdfdfdf
[ 87.777481] pci-endpoint-test 0000:01:00.0: buf[776] : dfdfdfdfdfdfdfdf
[ 87.784133] pci-endpoint-test 0000:01:00.0: buf[784] : dfdfdfdfdfdfdfdf
[ 87.790781] pci-endpoint-test 0000:01:00.0: buf[792] : dfdfdfdfdfdfdfdf
[ 87.797433] pci-endpoint-test 0000:01:00.0: buf[800] : dfdfdfdfdfdfdfdf
[ 87.804081] pci-endpoint-test 0000:01:00.0: buf[808] : dfdfdfdfdfdfdfdf
[ 87.810737] pci-endpoint-test 0000:01:00.0: buf[816] : dfdfdfdfdfdfdfdf
[ 87.817402] pci-endpoint-test 0000:01:00.0: buf[824] : dfdfdfdfdfdfdfdf
[ 87.824073] pci-endpoint-test 0000:01:00.0: buf[832] : dfdfdfdfdfdfdfdf
[ 87.830724] pci-endpoint-test 0000:01:00.0: buf[840] : dfdfdfdfdfdfdfdf
[ 87.837374] pci-endpoint-test 0000:01:00.0: buf[848] : dfdfdfdfdfdfdfdf
[ 87.844073] pci-endpoint-test 0000:01:00.0: buf[856] : dfdfdfdfdfdfdfdf
[ 87.850740] pci-endpoint-test 0000:01:00.0: buf[864] : dfdfdfdfdfdfdfdf
[ 87.857401] pci-endpoint-test 0000:01:00.0: buf[872] : dfdfdfdfdfdfdfdf
[ 87.864056] pci-endpoint-test 0000:01:00.0: buf[880] : dfdfdfdfdfdfdfdf
[ 87.870707] pci-endpoint-test 0000:01:00.0: buf[888] : dfdfdfdfdfdfdfdf
[ 87.877360] pci-endpoint-test 0000:01:00.0: buf[896] : dfdfdfdfdfdfdfdf
[ 87.884008] pci-endpoint-test 0000:01:00.0: buf[904] : dfdfdfdfdfdfdfdf
[ 87.890660] pci-endpoint-test 0000:01:00.0: buf[912] : dfdfdfdfdfdfdfdf
[ 87.897306] pci-endpoint-test 0000:01:00.0: buf[920] : dfdfdfdfdfdfdfdf
[ 87.903960] pci-endpoint-test 0000:01:00.0: buf[928] : dfdfdfdfdfdfdfdf
[ 87.910609] pci-endpoint-test 0000:01:00.0: buf[936] : dfdfdfdfdfdfdfdf
[ 87.917257] pci-endpoint-test 0000:01:00.0: buf[944] : dfdfdfdfdfdfdfdf
[ 87.923911] pci-endpoint-test 0000:01:00.0: buf[952] : dfdfdfdfdfdfdfdf
[ 87.930563] pci-endpoint-test 0000:01:00.0: buf[960] : dfdfdfdfdfdfdfdf
[ 87.937210] pci-endpoint-test 0000:01:00.0: buf[968] : dfdfdfdfdfdfdfdf
[ 87.943864] pci-endpoint-test 0000:01:00.0: buf[976] : dfdfdfdfdfdfdfdf
[ 87.950511] pci-endpoint-test 0000:01:00.0: buf[984] : dfdfdfdfdfdfdfdf
[ 87.957160] pci-endpoint-test 0000:01:00.0: buf[992] : dfdfdfdfdfdfdfdf
[ 87.963810] pci-endpoint-test 0000:01:00.0: buf[1000] : dfdfdfdfdfdfdfdf
[ 87.970549] pci-endpoint-test 0000:01:00.0: buf[1008] : dfdfdfdfdfdfdfdf
[ 87.977300] pci-endpoint-test 0000:01:00.0: buf[1016] : dfdfdfdfdfdfdfdf
[ 87.984047] pci-endpoint-test 0000:01:00.0: buf[1024] : dfdfdfdfdfdfdfdf
[ 87.990782] pci-endpoint-test 0000:01:00.0: buf[1032] : dfdfdfdfdfdfdfdf
[ 87.997521] pci-endpoint-test 0000:01:00.0: buf[1040] : dfdfdfdfdfdfdfdf
[ 88.004256] pci-endpoint-test 0000:01:00.0: buf[1048] : dfdfdfdfdfdfdfdf
[ 88.010994] pci-endpoint-test 0000:01:00.0: buf[1056] : dfdfdfdfdfdfdfdf
[ 88.017729] pci-endpoint-test 0000:01:00.0: buf[1064] : dfdfdfdfdfdfdfdf
[ 88.024471] pci-endpoint-test 0000:01:00.0: buf[1072] : dfdfdfdfdfdfdfdf
[ 88.031208] pci-endpoint-test 0000:01:00.0: buf[1080] : dfdfdfdfdfdfdfdf
[ 88.037947] pci-endpoint-test 0000:01:00.0: buf[1088] : dfdfdfdfdfdfdfdf
[ 88.044680] pci-endpoint-test 0000:01:00.0: buf[1096] : dfdfdfdfdfdfdfdf
[ 88.051423] pci-endpoint-test 0000:01:00.0: buf[1104] : dfdfdfdfdfdfdfdf
[ 88.058160] pci-endpoint-test 0000:01:00.0: buf[1112] : dfdfdfdfdfdfdfdf
[ 88.064898] pci-endpoint-test 0000:01:00.0: buf[1120] : dfdfdfdfdfdfdfdf
[ 88.071640] pci-endpoint-test 0000:01:00.0: buf[1128] : dfdfdfdfdfdfdfdf
[ 88.078380] pci-endpoint-test 0000:01:00.0: buf[1136] : dfdfdfdfdfdfdfdf
[ 88.085118] pci-endpoint-test 0000:01:00.0: buf[1144] : dfdfdfdfdfdfdfdf
[ 88.091870] pci-endpoint-test 0000:01:00.0: buf[1152] : dfdfdfdfdfdfdfdf
[ 88.098618] pci-endpoint-test 0000:01:00.0: buf[1160] : dfdfdfdfdfdfdfdf
[ 88.105362] pci-endpoint-test 0000:01:00.0: buf[1168] : dfdfdfdfdfdfdfdf
[ 88.112096] pci-endpoint-test 0000:01:00.0: buf[1176] : dfdfdfdfdfdfdfdf
[ 88.118835] pci-endpoint-test 0000:01:00.0: buf[1184] : dfdfdfdfdfdfdfdf
[ 88.125576] pci-endpoint-test 0000:01:00.0: buf[1192] : dfdfdfdfdfdfdfdf
[ 88.132314] pci-endpoint-test 0000:01:00.0: buf[1200] : dfdfdfdfdfdfdfdf
[ 88.139052] pci-endpoint-test 0000:01:00.0: buf[1208] : dfdfdfdfdfdfdfdf
[ 88.145789] pci-endpoint-test 0000:01:00.0: buf[1216] : dfdfdfdfdfdfdfdf
[ 88.152524] pci-endpoint-test 0000:01:00.0: buf[1224] : dfdfdfdfdfdfdfdf
[ 88.159261] pci-endpoint-test 0000:01:00.0: buf[1232] : dfdfdfdfdfdfdfdf
[ 88.165995] pci-endpoint-test 0000:01:00.0: buf[1240] : dfdfdfdfdfdfdfdf
[ 88.172732] pci-endpoint-test 0000:01:00.0: buf[1248] : dfdfdfdfdfdfdfdf
[ 88.179467] pci-endpoint-test 0000:01:00.0: buf[1256] : dfdfdfdfdfdfdfdf
[ 88.186202] pci-endpoint-test 0000:01:00.0: buf[1264] : dfdfdfdfdfdfdfdf
[ 88.192940] pci-endpoint-test 0000:01:00.0: buf[1272] : dfdfdfdfdfdfdfdf
[ 88.199679] pci-endpoint-test 0000:01:00.0: buf[1280] : dfdfdfdfdfdfdfdf
[ 88.206417] pci-endpoint-test 0000:01:00.0: buf[1288] : dfdfdfdfdfdfdfdf
[ 88.213167] pci-endpoint-test 0000:01:00.0: buf[1296] : dfdfdfdfdfdfdfdf
[ 88.219911] pci-endpoint-test 0000:01:00.0: buf[1304] : dfdfdfdfdfdfdfdf
[ 88.226659] pci-endpoint-test 0000:01:00.0: buf[1312] : dfdfdfdfdfdfdfdf
[ 88.233394] pci-endpoint-test 0000:01:00.0: buf[1320] : dfdfdfdfdfdfdfdf
[ 88.240131] pci-endpoint-test 0000:01:00.0: buf[1328] : dfdfdfdfdfdfdfdf
[ 88.246864] pci-endpoint-test 0000:01:00.0: buf[1336] : dfdfdfdfdfdfdfdf
[ 88.253603] pci-endpoint-test 0000:01:00.0: buf[1344] : dfdfdfdfdfdfdfdf
[ 88.260339] pci-endpoint-test 0000:01:00.0: buf[1352] : dfdfdfdfdfdfdfdf
[ 88.267077] pci-endpoint-test 0000:01:00.0: buf[1360] : dfdfdfdfdfdfdfdf
[ 88.273812] pci-endpoint-test 0000:01:00.0: buf[1368] : dfdfdfdfdfdfdfdf
[ 88.280547] pci-endpoint-test 0000:01:00.0: buf[1376] : dfdfdfdfdfdfdfdf
[ 88.287283] pci-endpoint-test 0000:01:00.0: buf[1384] : dfdfdfdfdfdfdfdf
[ 88.294018] pci-endpoint-test 0000:01:00.0: buf[1392] : dfdfdfdfdfdfdfdf
[ 88.300752] pci-endpoint-test 0000:01:00.0: buf[1400] : dfdfdfdfdfdfdfdf
[ 88.307491] pci-endpoint-test 0000:01:00.0: buf[1408] : dfdfdfdfdfdfdfdf
[ 88.314226] pci-endpoint-test 0000:01:00.0: buf[1416] : dfdfdfdfdfdfdfdf
[ 88.320967] pci-endpoint-test 0000:01:00.0: buf[1424] : dfdfdfdfdfdfdfdf
[ 88.327721] pci-endpoint-test 0000:01:00.0: buf[1432] : dfdfdfdfdfdfdfdf
[ 88.334468] pci-endpoint-test 0000:01:00.0: buf[1440] : dfdfdfdfdfdfdfdf
[ 88.341204] pci-endpoint-test 0000:01:00.0: buf[1448] : dfdfdfdfdfdfdfdf
[ 88.347945] pci-endpoint-test 0000:01:00.0: buf[1456] : dfdfdfdfdfdfdfdf
[ 88.354680] pci-endpoint-test 0000:01:00.0: buf[1464] : dfdfdfdfdfdfdfdf
[ 88.361416] pci-endpoint-test 0000:01:00.0: buf[1472] : dfdfdfdfdfdfdfdf
[ 88.368153] pci-endpoint-test 0000:01:00.0: buf[1480] : dfdfdfdfdfdfdfdf
[ 88.374890] pci-endpoint-test 0000:01:00.0: buf[1488] : dfdfdfdfdfdfdfdf
[ 88.381626] pci-endpoint-test 0000:01:00.0: buf[1496] : dfdfdfdfdfdfdfdf
[ 88.388361] pci-endpoint-test 0000:01:00.0: buf[1504] : dfdfdfdfdfdfdfdf
[ 88.395095] pci-endpoint-test 0000:01:00.0: buf[1512] : dfdfdfdfdfdfdfdf
[ 88.401833] pci-endpoint-test 0000:01:00.0: buf[1520] : dfdfdfdfdfdfdfdf
[ 88.408569] pci-endpoint-test 0000:01:00.0: buf[1528] : dfdfdfdfdfdfdfdf
[ 88.415306] pci-endpoint-test 0000:01:00.0: buf[1536] : dfdfdfdfdfdfdfdf
[ 88.422041] pci-endpoint-test 0000:01:00.0: buf[1544] : dfdfdfdfdfdfdfdf
[ 88.428783] pci-endpoint-test 0000:01:00.0: buf[1552] : dfdfdfdfdfdfdfdf
[ 88.435521] pci-endpoint-test 0000:01:00.0: buf[1560] : dfdfdfdfdfdfdfdf
[ 88.442263] pci-endpoint-test 0000:01:00.0: buf[1568] : dfdfdfdfdfdfdfdf
[ 88.449012] pci-endpoint-test 0000:01:00.0: buf[1576] : dfdfdfdfdfdfdfdf
[ 88.455756] pci-endpoint-test 0000:01:00.0: buf[1584] : dfdfdfdfdfdfdfdf
[ 88.462494] pci-endpoint-test 0000:01:00.0: buf[1592] : dfdfdfdfdfdfdfdf
[ 88.469232] pci-endpoint-test 0000:01:00.0: buf[1600] : dfdfdfdfdfdfdfdf
[ 88.475967] pci-endpoint-test 0000:01:00.0: buf[1608] : dfdfdfdfdfdfdfdf
[ 88.482705] pci-endpoint-test 0000:01:00.0: buf[1616] : dfdfdfdfdfdfdfdf
[ 88.489442] pci-endpoint-test 0000:01:00.0: buf[1624] : dfdfdfdfdfdfdfdf
[ 88.496205] pci-endpoint-test 0000:01:00.0: buf[1632] : dfdfdfdfdfdfdfdf
[ 88.502942] pci-endpoint-test 0000:01:00.0: buf[1640] : dfdfdfdfdfdfdfdf
[ 88.509681] pci-endpoint-test 0000:01:00.0: buf[1648] : dfdfdfdfdfdfdfdf
[ 88.516416] pci-endpoint-test 0000:01:00.0: buf[1656] : dfdfdfdfdfdfdfdf
[ 88.523154] pci-endpoint-test 0000:01:00.0: buf[1664] : dfdfdfdfdfdfdfdf
[ 88.529895] pci-endpoint-test 0000:01:00.0: buf[1672] : dfdfdfdfdfdfdfdf
[ 88.536633] pci-endpoint-test 0000:01:00.0: buf[1680] : dfdfdfdfdfdfdfdf
[ 88.543371] pci-endpoint-test 0000:01:00.0: buf[1688] : dfdfdfdfdfdfdfdf
[ 88.550108] pci-endpoint-test 0000:01:00.0: buf[1696] : dfdfdfdfdfdfdfdf
[ 88.556844] pci-endpoint-test 0000:01:00.0: buf[1704] : dfdfdfdfdfdfdfdf
[ 88.563583] pci-endpoint-test 0000:01:00.0: buf[1712] : dfdfdfdfdfdfdfdf
[ 88.570318] pci-endpoint-test 0000:01:00.0: buf[1720] : dfdfdfdfdfdfdfdf
[ 88.577055] pci-endpoint-test 0000:01:00.0: buf[1728] : dfdfdfdfdfdfdfdf
[ 88.583791] pci-endpoint-test 0000:01:00.0: buf[1736] : dfdfdfdfdfdfdfdf
[ 88.590530] pci-endpoint-test 0000:01:00.0: buf[1744] : dfdfdfdfdfdfdfdf
[ 88.597263] pci-endpoint-test 0000:01:00.0: buf[1752] : dfdfdfdfdfdfdfdf
[ 88.604003] pci-endpoint-test 0000:01:00.0: buf[1760] : dfdfdfdfdfdfdfdf
[ 88.610740] pci-endpoint-test 0000:01:00.0: buf[1768] : dfdfdfdfdfdfdfdf
[ 88.617505] pci-endpoint-test 0000:01:00.0: buf[1776] : dfdfdfdfdfdfdfdf
[ 88.624251] pci-endpoint-test 0000:01:00.0: buf[1784] : dfdfdfdfdfdfdfdf
[ 88.630996] pci-endpoint-test 0000:01:00.0: buf[1792] : dfdfdfdfdfdfdfdf
[ 88.637731] pci-endpoint-test 0000:01:00.0: buf[1800] : dfdfdfdfdfdfdfdf
[ 88.644470] pci-endpoint-test 0000:01:00.0: buf[1808] : dfdfdfdfdfdfdfdf
[ 88.651206] pci-endpoint-test 0000:01:00.0: buf[1816] : dfdfdfdfdfdfdfdf
[ 88.657945] pci-endpoint-test 0000:01:00.0: buf[1824] : dfdfdfdfdfdfdfdf
[ 88.664682] pci-endpoint-test 0000:01:00.0: buf[1832] : dfdfdfdfdfdfdfdf
[ 88.671420] pci-endpoint-test 0000:01:00.0: buf[1840] : dfdfdfdfdfdfdfdf
[ 88.678155] pci-endpoint-test 0000:01:00.0: buf[1848] : dfdfdfdfdfdfdfdf
[ 88.684892] pci-endpoint-test 0000:01:00.0: buf[1856] : dfdfdfdfdfdfdfdf
[ 88.691626] pci-endpoint-test 0000:01:00.0: buf[1864] : dfdfdfdfdfdfdfdf
[ 88.698365] pci-endpoint-test 0000:01:00.0: buf[1872] : dfdfdfdfdfdfdfdf
[ 88.705103] pci-endpoint-test 0000:01:00.0: buf[1880] : dfdfdfdfdfdfdfdf
[ 88.711839] pci-endpoint-test 0000:01:00.0: buf[1888] : dfdfdfdfdfdfdfdf
[ 88.718574] pci-endpoint-test 0000:01:00.0: buf[1896] : dfdfdfdfdfdfdfdf
[ 88.725310] pci-endpoint-test 0000:01:00.0: buf[1904] : dfdfdfdfdfdfdfdf
[ 88.732051] pci-endpoint-test 0000:01:00.0: buf[1912] : dfdfdfdfdfdfdfdf
[ 88.738795] pci-endpoint-test 0000:01:00.0: buf[1920] : dfdfdfdfdfdfdfdf
[ 88.745543] pci-endpoint-test 0000:01:00.0: buf[1928] : dfdfdfdfdfdfdfdf
[ 88.752289] pci-endpoint-test 0000:01:00.0: buf[1936] : dfdfdfdfdfdfdfdf
[ 88.759029] pci-endpoint-test 0000:01:00.0: buf[1944] : dfdfdfdfdfdfdfdf
[ 88.765769] pci-endpoint-test 0000:01:00.0: buf[1952] : dfdfdfdfdfdfdfdf
[ 88.772505] pci-endpoint-test 0000:01:00.0: buf[1960] : dfdfdfdfdfdfdfdf
[ 88.779245] pci-endpoint-test 0000:01:00.0: buf[1968] : dfdfdfdfdfdfdfdf
[ 88.785978] pci-endpoint-test 0000:01:00.0: buf[1976] : dfdfdfdfdfdfdfdf
[ 88.792720] pci-endpoint-test 0000:01:00.0: buf[1984] : dfdfdfdfdfdfdfdf
[ 88.799453] pci-endpoint-test 0000:01:00.0: buf[1992] : dfdfdfdfdfdfdfdf
[ 88.806194] pci-endpoint-test 0000:01:00.0: buf[2000] : dfdfdfdfdfdfdfdf
[ 88.812930] pci-endpoint-test 0000:01:00.0: buf[2008] : dfdfdfdfdfdfdfdf
[ 88.819668] pci-endpoint-test 0000:01:00.0: buf[2016] : dfdfdfdfdfdfdfdf
[ 88.826403] pci-endpoint-test 0000:01:00.0: buf[2024] : dfdfdfdfdfdfdfdf
[ 88.833207] pci-endpoint-test 0000:01:00.0: buf[2032] : dfdfdfdfdfdfdfdf
[ 88.839961] pci-endpoint-test 0000:01:00.0: buf[2040] : dfdfdfdfdfdfdfdf
[ 88.846710] pci-endpoint-test 0000:01:00.0: buf[2048] : dfdfdfdfdfdfdfdf
[ 88.853463] pci-endpoint-test 0000:01:00.0: buf[2056] : dfdfdfdfdfdfdfdf
[ 88.860202] pci-endpoint-test 0000:01:00.0: buf[2064] : dfdfdfdfdfdfdfdf
[ 88.866939] pci-endpoint-test 0000:01:00.0: buf[2072] : dfdfdfdfdfdfdfdf
[ 88.873675] pci-endpoint-test 0000:01:00.0: buf[2080] : dfdfdfdfdfdfdfdf
[ 88.880410] pci-endpoint-test 0000:01:00.0: buf[2088] : dfdfdfdfdfdfdfdf
[ 88.887145] pci-endpoint-test 0000:01:00.0: buf[2096] : dfdfdfdfdfdfdfdf
[ 88.893879] pci-endpoint-test 0000:01:00.0: buf[2104] : dfdfdfdfdfdfdfdf
[ 88.900619] pci-endpoint-test 0000:01:00.0: buf[2112] : dfdfdfdfdfdfdfdf
[ 88.907353] pci-endpoint-test 0000:01:00.0: buf[2120] : dfdfdfdfdfdfdfdf
[ 88.914091] pci-endpoint-test 0000:01:00.0: buf[2128] : dfdfdfdfdfdfdfdf
[ 88.920825] pci-endpoint-test 0000:01:00.0: buf[2136] : dfdfdfdfdfdfdfdf
[ 88.927562] pci-endpoint-test 0000:01:00.0: buf[2144] : dfdfdfdfdfdfdfdf
[ 88.934303] pci-endpoint-test 0000:01:00.0: buf[2152] : dfdfdfdfdfdfdfdf
[ 88.941043] pci-endpoint-test 0000:01:00.0: buf[2160] : dfdfdfdfdfdfdfdf
[ 88.947781] pci-endpoint-test 0000:01:00.0: buf[2168] : dfdfdfdfdfdfdfdf
[ 88.955146] pci-endpoint-test 0000:01:00.0:
[ 88.955146]
[ 88.955146]
[ 88.955146]
[ 88.963900] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
kzalloc:ffff0004b6393000 dma handle:7e93c800 buffer CRC:ce535039 BAR
CRC:910c690d tmp buffer crc: ce535039
READ ( 2176 bytes): NOT OKAY
And starangely even tmp_buffer has the value of 0xDF, but the CRC is screwed.
Cheers,
--Prabhakar
> Best regards,
> Yoshihiro Shimoda
>
> > Cheers,
> > --Prabhakar
^ permalink raw reply related [flat|nested] 20+ messages in thread
* RE: PCIe EPF
2020-04-01 9:24 ` Lad, Prabhakar
@ 2020-04-02 1:22 ` Yoshihiro Shimoda
2020-04-02 4:58 ` Yoshihiro Shimoda
0 siblings, 1 reply; 20+ messages in thread
From: Yoshihiro Shimoda @ 2020-04-02 1:22 UTC (permalink / raw)
To: Lad, Prabhakar
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Prahakar-san,
> From: Lad, Prabhakar, Sent: Wednesday, April 1, 2020 6:24 PM
<snip>
> > The crc32 value was calculated before print the buffer.
> > This means that timing is difference between crc32_le(addr) and print_buffer(addr).
> > So, I'd like to copy the addr buffer to tmp and use the tmp for crc32_le() and print_buffer().
> >
> Following is the diff of suggested changes:
Thanks!
> diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> index 3c49514b4813..b782134838d1 100644
> --- a/drivers/misc/pci_endpoint_test.c
> +++ b/drivers/misc/pci_endpoint_test.c
> @@ -561,6 +561,23 @@ static bool pci_endpoint_test_write(struct
> pci_endpoint_test *test,
> return ret;
> }
>
> +static void print_buffer(struct device *dev, void *buff_addr, size_t size)
> +{
> + size_t i;
> + u64 *addr = buff_addr;
> +
> + for(i = 0; i < size; i += sizeof(addr))
> + dev_err(dev, "buf[%zu] : %llx\n", i, *addr);
> +
> + for(i = 0; i < size; i +=1) {
> + if (0x910c690d == crc32_le(~0, buff_addr, i))
> + dev_err(dev, "%x\n",
> + crc32_le(~0, buff_addr, i));
> + }
> +
> + dev_err(dev, "\n\n\n\n");
> +}
> +
> static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
> unsigned long arg)
> {
> @@ -574,6 +591,7 @@ static bool pci_endpoint_test_read(struct
> pci_endpoint_test *test,
> struct pci_dev *pdev = test->pdev;
> struct device *dev = &pdev->dev;
> void *orig_addr;
> + void *tmp_buffer;
> dma_addr_t orig_phys_addr;
> size_t offset;
> size_t alignment = test->alignment;
> @@ -601,14 +619,19 @@ static bool pci_endpoint_test_read(struct
> pci_endpoint_test *test,
> }
>
> orig_addr = kzalloc(size + alignment, GFP_KERNEL);
> - if (!orig_addr) {
> + tmp_buffer = kzalloc(size + alignment, GFP_KERNEL);
> + if (!orig_addr || !tmp_buffer) {
> dev_err(dev, "Failed to allocate destination address\n");
> ret = false;
> + if (orig_addr)
> + kfree(orig_addr);
> + if (tmp_buffer)
> + kfree(tmp_buffer);
> goto err;
> }
>
> orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
> - DMA_FROM_DEVICE);
> + DMA_BIDIRECTIONAL);
> if (dma_mapping_error(dev, orig_phys_addr)) {
> dev_err(dev, "failed to map source buffer address\n");
> ret = false;
> @@ -640,14 +663,24 @@ static bool pci_endpoint_test_read(struct
> pci_endpoint_test *test,
> wait_for_completion(&test->irq_raised);
>
> dma_unmap_single(dev, orig_phys_addr, size + alignment,
> - DMA_FROM_DEVICE);
> + DMA_BIDIRECTIONAL);
> +
> + memcpy(tmp_buffer, addr, size);
>
> crc32 = crc32_le(~0, addr, size);
I intended to use tmp_buffer for this crc32_le().
But, the result seems my guess (timing issue) is incorrect anyway.
> if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
> ret = true;
>
> + print_buffer(dev, addr, size);
> + print_buffer(dev, tmp_buffer, size);
> + dev_err(dev, "%s kzalloc:%px dma handle:%llx buffer CRC:%x BAR
> CRC:%x tmp buffer crc: %x\n",
> + __func__, orig_addr, orig_phys_addr, crc32,
> + pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM),
> + crc32_le(~0, tmp_buffer, size));
Calling crc32_le(~0, tmp_buffer, size) here is useful.
> +
> err_phys_addr:
> kfree(orig_addr);
> + kfree(tmp_buffer);
> err:
> return ret;
> }
<snip>
> > By the way, your PCIe host environment (RZ/G2N) output the following log?
> >
> > [ 0.000000] software IO TLB: mapped [mem 0x7bfff000-0x7ffff000] (64MB)
> >
> > If so, I guess this issue is related to this software IO TLB behavior.
> > IIUC, if we use coherent buffer or GPF_DMA buffer, the kernel will not
> > use software IO TLB for these buffers.
> >
>
> root@hihope-rzg2m:~# dmesg | grep TLB
> [ 0.000000] software IO TLB: mapped [mem 0x7bfff000-0x7ffff000] (64MB)
I got it.
<snip>
> [ 88.963900] pci-endpoint-test 0000:01:00.0: pci_endpoint_test_read
> kzalloc:ffff0004b6393000 dma handle:7e93c800 buffer CRC:ce535039 BAR
> CRC:910c690d tmp buffer crc: ce535039
> READ ( 2176 bytes): NOT OKAY
>
> And starangely even tmp_buffer has the value of 0xDF, but the CRC is screwed.
I agree. this result is strange.
So, I have no idea why this issue happened..
By the way, according to previous your report, you are using pci/next branch
and the branch is based on v5.6-rc1. There is no evidence though,
I'd like to use next-20200401 tag from linux-next repo to use v5.6 based kernel
whether the strange issue happens on the latest v5.6 kernel code or not.
Note that I confirmed the next-20200401 tag has a commit d293237739414d
("misc: pci_endpoint_test: Use streaming DMA APIs for buffer allocation").
Best regards,
Yoshihiro Shimoda
^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: PCIe EPF
2020-04-02 1:22 ` Yoshihiro Shimoda
@ 2020-04-02 4:58 ` Yoshihiro Shimoda
2020-04-02 17:44 ` Lad, Prabhakar
0 siblings, 1 reply; 20+ messages in thread
From: Yoshihiro Shimoda @ 2020-04-02 4:58 UTC (permalink / raw)
To: Lad, Prabhakar
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Prabhakar-san,
> From: Yoshihiro Shimoda, Sent: Thursday, April 2, 2020 10:23 AM
<snip>
> By the way, according to previous your report, you are using pci/next branch
> and the branch is based on v5.6-rc1. There is no evidence though,
> I'd like to use next-20200401 tag from linux-next repo to use v5.6 based kernel
> whether the strange issue happens on the latest v5.6 kernel code or not.
> Note that I confirmed the next-20200401 tag has a commit d293237739414d
> ("misc: pci_endpoint_test: Use streaming DMA APIs for buffer allocation").
I'm afraid but I'd like to recall this because next-20200401 seems unstable on
R-Car Gen3 environment (a lot of WARING happens). So, perhaps using linux mainline
v5.6 + merging pci/next branch is better.
Best regards,
Yoshihiro Shimoda
> Best regards,
> Yoshihiro Shimoda
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: PCIe EPF
2020-04-02 4:58 ` Yoshihiro Shimoda
@ 2020-04-02 17:44 ` Lad, Prabhakar
0 siblings, 0 replies; 20+ messages in thread
From: Lad, Prabhakar @ 2020-04-02 17:44 UTC (permalink / raw)
To: Yoshihiro Shimoda
Cc: Kishon Vijay Abraham I, Chris Paterson, Lorenzo Pieralisi, linux-pci
Hi Shimoda-san,
On Thu, Apr 2, 2020 at 5:58 AM Yoshihiro Shimoda
<yoshihiro.shimoda.uh@renesas.com> wrote:
>
> Hi Prabhakar-san,
>
> > From: Yoshihiro Shimoda, Sent: Thursday, April 2, 2020 10:23 AM
> <snip>
> > By the way, according to previous your report, you are using pci/next branch
> > and the branch is based on v5.6-rc1. There is no evidence though,
> > I'd like to use next-20200401 tag from linux-next repo to use v5.6 based kernel
> > whether the strange issue happens on the latest v5.6 kernel code or not.
> > Note that I confirmed the next-20200401 tag has a commit d293237739414d
> > ("misc: pci_endpoint_test: Use streaming DMA APIs for buffer allocation").
>
> I'm afraid but I'd like to recall this because next-20200401 seems unstable on
> R-Car Gen3 environment (a lot of WARING happens). So, perhaps using linux mainline
> v5.6 + merging pci/next branch is better.
>
Taking your advice I rebased my patches on Geert's tree [1] (master
branch) and choose
M3N as my root device, and with rigorous testing I can confirm
read/write/copy work
without any issues. Using the same branch with G2N as root device still failed.
Since this is a Root device issue and not ep I am posting v6 and
swiotlb can be handled
separately for G2x platforms.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git/
Cheers,
--Prabhakar
> Best regards,
> Yoshihiro Shimoda
>
> > Best regards,
> > Yoshihiro Shimoda
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-04-02 17:45 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-18 11:37 PCIe EPF Lad, Prabhakar
2020-03-20 5:28 ` Kishon Vijay Abraham I
2020-03-21 22:49 ` Lad, Prabhakar
2020-03-24 1:58 ` Kishon Vijay Abraham I
2020-03-24 14:41 ` Lad, Prabhakar
2020-03-28 18:44 ` Lad, Prabhakar
2020-03-29 14:04 ` Lad, Prabhakar
2020-03-30 11:59 ` Kishon Vijay Abraham I
2020-03-30 13:09 ` Lad, Prabhakar
2020-03-30 13:20 ` Kishon Vijay Abraham I
2020-03-30 13:46 ` Lad, Prabhakar
2020-03-31 7:10 ` Yoshihiro Shimoda
2020-03-31 17:25 ` Lad, Prabhakar
2020-04-01 1:25 ` Yoshihiro Shimoda
2020-04-01 8:00 ` Lad, Prabhakar
2020-04-01 8:50 ` Yoshihiro Shimoda
2020-04-01 9:24 ` Lad, Prabhakar
2020-04-02 1:22 ` Yoshihiro Shimoda
2020-04-02 4:58 ` Yoshihiro Shimoda
2020-04-02 17:44 ` Lad, Prabhakar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).