PCI DMA to user mem on mpc83xx

PCI DMA to user mem on mpc83xx

[Andre Schwarz]

Ira,

we have a pretty old PCI device driver here that needs some basic rework 
running on 2.6.27 on several MPC83xx.
It's a simple char-device with "give me some data" implemented using 
read() resulting in zero-copy DMA to user mem.

There's get_user_pages() working under the hood along with 
SetPageDirty() and page_cache_release().

Main goal is to prepare a sg-list that gets fed into a DMA controller.

I wonder if there's a more up-to-date/efficient and future proof scheme 
of creating the mapping.


Could you provide some pointers or would you stick to the current scheme ?


Regards,
André

MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

Re: PCI DMA to user mem on mpc83xx

[Ira W. Snyder]

On Mon, May 23, 2011 at 11:12:41AM +0200, Andre Schwarz wrote:
> Ira,
> 
> we have a pretty old PCI device driver here that needs some basic rework 
> running on 2.6.27 on several MPC83xx.
> It's a simple char-device with "give me some data" implemented using 
> read() resulting in zero-copy DMA to user mem.
> 
> There's get_user_pages() working under the hood along with 
> SetPageDirty() and page_cache_release().
> 
> Main goal is to prepare a sg-list that gets fed into a DMA controller.
> 
> I wonder if there's a more up-to-date/efficient and future proof scheme 
> of creating the mapping.
> 
> 
> Could you provide some pointers or would you stick to the current scheme ?
> 

This scheme is the best you'll come up with for zero-copy IO. I used
get_user_pages_fast(), but otherwise my implementation was the same.
These interfaces should be fairly future proof.

In the end, I realized that most of my transfers were 4 bytes in length,
and zero copy IO was a waste of effort. I decided to use mmap instead.

Ira

RE: PCI DMA to user mem on mpc83xx

[David Laight]

=20
> we have a pretty old PCI device driver here that needs some
> basic rework running on 2.6.27 on several MPC83xx.
> It's a simple char-device with "give me some data" implemented
> using read() resulting in zero-copy DMA to user mem.
>=20
> There's get_user_pages() working under the hood along with=20
> SetPageDirty() and page_cache_release().

Does that dma use the userspace virtual address, or the
physical address - or are you remapping the user memory into
kernel address space.

If the memory is remapped into the kernel address space, the
cost of the mmu and tlb operations (especially on MP systems)
is such that a dma to kernel memory followed by copyout/copytouser
may well be faster!

That may even be the case even if the dma is writing to the
user virtual (or physical) addresses when it is only
necessary to ensure the memory page is resident and that
the caches are coherent.

In any case the second copy is probably far faster than the
PCI one!

I've recently written driver that supports a pread/pwrite interface
to the memory windows on a PCIe card. It was important to use
dma for the PCIe transfers (to get a sensible transfer size).
I overlapped the copyin/copyout with the next dma transfer.
The dma's are fast enough that it is worth spinning waiting
for completion - but slow enough to make the overlapped
operation worthwhile (same speed as a single word pio transfer).

	David

Re: PCI DMA to user mem on mpc83xx

[Andre Schwarz]

Ira,

> On Mon, May 23, 2011 at 11:12:41AM +0200, Andre Schwarz wrote:
>> Ira,
>>
>> we have a pretty old PCI device driver here that needs some basic rework
>> running on 2.6.27 on several MPC83xx.
>> It's a simple char-device with "give me some data" implemented using
>> read() resulting in zero-copy DMA to user mem.
>>
>> There's get_user_pages() working under the hood along with
>> SetPageDirty() and page_cache_release().
>>
>> Main goal is to prepare a sg-list that gets fed into a DMA controller.
>>
>> I wonder if there's a more up-to-date/efficient and future proof scheme
>> of creating the mapping.
>>
>>
>> Could you provide some pointers or would you stick to the current scheme ?
>>
> This scheme is the best you'll come up with for zero-copy IO. I used
> get_user_pages_fast(), but otherwise my implementation was the same.
> These interfaces should be fairly future proof.
excellent - thanks.
Will stick to it then ...
> In the end, I realized that most of my transfers were 4 bytes in length,
> and zero copy IO was a waste of effort. I decided to use mmap instead.
>

I'm using 98% page sized (4KiB) scatter gather transfers summing up to 
~80MiB/sec sustained throughput.




Cheers,
André

MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

Re: PCI DMA to user mem on mpc83xx

[Andre Schwarz]

David,

>> we have a pretty old PCI device driver here that needs some
>> basic rework running on 2.6.27 on several MPC83xx.
>> It's a simple char-device with "give me some data" implemented
>> using read() resulting in zero-copy DMA to user mem.
>>
>> There's get_user_pages() working under the hood along with
>> SetPageDirty() and page_cache_release().
> Does that dma use the userspace virtual address, or the
> physical address - or are you remapping the user memory into
> kernel address space.

no mapping at all AFAIK.

I'm using get_user_pages() followed by allocation of a struct 
scatterlist being filled with sg_set_page().
After the transfer the pages are marked dirty using SetPageDirty().

> If the memory is remapped into the kernel address space, the
> cost of the mmu and tlb operations (especially on MP systems)
> is such that a dma to kernel memory followed by copyout/copytouser
> may well be faster!
no mapping.

> That may even be the case even if the dma is writing to the
> user virtual (or physical) addresses when it is only
> necessary to ensure the memory page is resident and that
> the caches are coherent.
All I need is physical addresses of user mem.
Since the allocating user driver is using mlock() and there's no swap I 
expect to be safe ... is this a stupid assumption ?
> In any case the second copy is probably far faster than the
> PCI one!
huh - I observed memcpy() to be very expensive (at least on 83xx PowerPC).
> I've recently written driver that supports a pread/pwrite interface
> to the memory windows on a PCIe card. It was important to use
> dma for the PCIe transfers (to get a sensible transfer size).
> I overlapped the copyin/copyout with the next dma transfer.
> The dma's are fast enough that it is worth spinning waiting
> for completion - but slow enough to make the overlapped
> operation worthwhile (same speed as a single word pio transfer).


Thanks for your feedback.


Cheers,
André


MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner