How to DMA data from a pci device to a user buffer directly

How to DMA data from a pci device to a user buffer directly

[Lee, Shuyu]

Hello, All.

In my device driver, I want to DMA image data from a PCI device (a frame
grabber) to a user buffer directly. Right now I am stuck at how to convert a
user virtual address to a physical address that can be used by the DMA
controller. 

Here is my sequence:
1) User allocates an image buffer, 
2) User passes the pointer (pUserVirtualAddr) of that buffer to the driver
through an ioctl call,
3) Driver calls alloc_kiovec(1, &iobuf);
4) Driver calls map_user_kiobuf(WRITE, iobuf, (ULONG)pUserVirtualAddr,
imageBufferSize);
I believe map_user_kiobuf() is executed successfully, for the following
reasons:
1) map_user_kiobuf() returns 0,
2) For imageBufferSize = 640*160, iobuf->nr_pages = 26. Because
pUserVirtualAddr is not page-aligned, 26 is correct. 
3) iobuf->offset is equal to (ULONG)pUserVirtualAddr & 0xFFF, which is also
correct.
What is strange to me is that iobuf->maplist[idx]->virtual = 0 for idx =
0,1,2,...,25.

One driver developer suggested that I do the following to get the physical
address for the first page:
physAddr = virt_to_bus(page_address(iobuf->maplist[0])) + iobuf->offset;
Because page_address() is "#define page_address(page) ((page)->virtual)" and
"virtual" is equal to 0, this approach will not work for me.

Another suggested that I add this to my code:
#define page_to_bus(page) \
        (ULONG)(((page) - mem_map) << PAGE_SHIFT),
and get the physical address for the first page this way:
physAddr = page_to_bus(iobuf->maplist[0])+iobuf->offset;
Because it is not clear to me what exactly is "mem_map", I am not sure how
"maplist" of type "struct page*", is subtracted by "mem_map" of type
"mem_map_t*".
In my test, I have:
1) maplist[0] = 0xC114ECEC;
2) mem_map = 0xC1000010.
Ignore the offset, what the physical address should be?

My development environment:
1) OS: RedHat 7.2 (2.4.7-10)
2) GCC: 3.2.1
3) PC:  P-III single processor w/ 128Mbytes of memory,
4) BUS: PCI,
5) My DMA controller has unlimited scatter-gather capability.

By the way, I have tested the rest of my code by DMA the image data to a
kernel buffer allocated using kmalloc() first, then do a memcpy() to copy
the image data to a user buffer. This alternative seems to work fine.

All suggestions and comments are greatly appreciated.
Shuyu

Re: How to DMA data from a pci device to a user buffer directly

[Pete Zaitcev]

> 5) My DMA controller has unlimited scatter-gather capability.

> By the way, I have tested the rest of my code by DMA the image data to a
> kernel buffer allocated using kmalloc() first, then do a memcpy() to copy
> the image data to a user buffer. This alternative seems to work fine.

Use mmap to make the kmalloc-ed buffer available to user
application without the overhead of memcpy().

It is very wonderful that you can do s/g, so on the next stage
you can kmalloc a bunch of blocks with small order (1) and use
those instead of relying on bootmem allocation and Pauline's
bigphysarea patch. Most older controllers cannot do it.

-- Pete