Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

[Leon Woestenberg]

Hello,

On Wed, May 18, 2011 at 4:41 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, May 18, 2011 at 03:02:30PM +0200, Leon Woestenberg wrote:
>>
>> memory allocated with pci_alloc_consistent() returns the (kernel)
>> virtual address and the bus address (which may be different from the
>> physical memory address).
>>
>> What is the correct implementation of the driver mmap (file operation
>> method) for such memory?
>
> You are going to use the physical address from the CPU side. So not
> the bus address. Instead use the virtual address and find the
> physical address from that. page_to_pfn() does a good job.
>
pci_alloc_consistent() returns a kernel virtual address. To find the
page I think virt_to_page() suits me better, right?

#define virt_to_page(kaddr)     pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)

> Then you can call 'vm_insert_page(vma...)'
>
> Or 'vm_insert_mixed'

Thanks, that opens a whole new learning curve experience.

Can I call vmalloc_to_page() on memory allocated with
pci_alloc_consistent()? If so, then remap_vmalloc_range() looks
promising.

I could not find PCI driver examples calling vm_insert_page() and I am
know I can trip into the different memory type pointers easily.

How does your suggestion relate to using the vma ops fault() (formerly
known as nopage() to mmap memory allocated by pci_alloc_consistent()?
i.e. Such as suggested in
http://www.gossamer-threads.com/lists/linux/kernel/702127#702127

> Use 'cscope' on the Linux kernel.

Thanks for the suggestion. How would cscope help me find
vm_insert_page() given my question?

On hind-sight all questions seem to be easy once finding the correct
Documentation / source-code in the first place. I usually use
http://lxr.linux.no/ and friends.


Regards,
-- 
Leon

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

[Konrad Rzeszutek Wilk]

On Wed, May 18, 2011 at 05:03:41PM +0200, Leon Woestenberg wrote:
> Hello,
> 
> On Wed, May 18, 2011 at 4:41 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, May 18, 2011 at 03:02:30PM +0200, Leon Woestenberg wrote:
> >>
> >> memory allocated with pci_alloc_consistent() returns the (kernel)
> >> virtual address and the bus address (which may be different from the
> >> physical memory address).
> >>
> >> What is the correct implementation of the driver mmap (file operation
> >> method) for such memory?
> >
> > You are going to use the physical address from the CPU side. So not
> > the bus address. Instead use the virtual address and find the
> > physical address from that. page_to_pfn() does a good job.
> >
> pci_alloc_consistent() returns a kernel virtual address. To find the
> page I think virt_to_page() suits me better, right?
> 
> #define virt_to_page(kaddr)     pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
> 
> > Then you can call 'vm_insert_page(vma...)'
> >
> > Or 'vm_insert_mixed'
> 
> Thanks, that opens a whole new learning curve experience.
> 
> Can I call vmalloc_to_page() on memory allocated with
> pci_alloc_consistent()? If so, then remap_vmalloc_range() looks
> promising.

No. That is b/c pci_alloc_consistent allocates pages from ..
well, this is a bit complex and varies on the platform. But _mostly_
if your device is 32-bit, it allocates it from ZONE_DMA32. Otherwise
it is from other zones. The 'vmalloc' pages are quite different and
are usually not exposed to the PCI devices, unless you do some extra
jumps (you need to kmap them).
> 
> I could not find PCI driver examples calling vm_insert_page() and I am
> know I can trip into the different memory type pointers easily.

ttm_bo_vm.c ?
fb_defio.c ?

> 
> How does your suggestion relate to using the vma ops fault() (formerly
> known as nopage() to mmap memory allocated by pci_alloc_consistent()?

You can use the pages that you had allocated via pci_alloc_consistent
and stitch them in the userspace vma.

> i.e. Such as suggested in
> http://www.gossamer-threads.com/lists/linux/kernel/702127#702127
> 
> > Use 'cscope' on the Linux kernel.
> 
> Thanks for the suggestion. How would cscope help me find
> vm_insert_page() given my question?

You can find examples of who uses it.
> 
> On hind-sight all questions seem to be easy once finding the correct
> Documentation / source-code in the first place. I usually use
> http://lxr.linux.no/ and friends.
> 
> 
> Regards,
> -- 
> Leon

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

[Leon Woestenberg]

Hello Konrad,

On Wed, May 18, 2011 at 5:40 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, May 18, 2011 at 05:03:41PM +0200, Leon Woestenberg wrote:
>> On Wed, May 18, 2011 at 4:41 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Wed, May 18, 2011 at 03:02:30PM +0200, Leon Woestenberg wrote:
>> >>
>> >> memory allocated with pci_alloc_consistent() returns the (kernel)
>> >> virtual address and the bus address (which may be different from the
>> >> physical memory address).
>> >>
>> >> What is the correct implementation of the driver mmap (file operation
>> >> method) for such memory?
>> >
>>
>> I could not find PCI driver examples calling vm_insert_page() and I am
>> know I can trip into the different memory type pointers easily.
>
> ttm_bo_vm.c ?
> fb_defio.c ?
>
None of which use pci/dma_alloc_consistent().

Obviously, I have no complete understanding of the Linux memory
management subsystem, and the info on vm_insert_page() is rather
shallow in the case of pci_alloc_consistent().

http://lxr.linux.no/#linux+v2.6.38/mm/memory.c#L1789

1789        update_mmu_cache(vma, addr, pte); /* XXX: why not for
insert_page? */


I tried this:

static int buffer_mmap(struct file *file, struct vm_area_struct *vma)
{
        ...

	/* pages must not be cached as this would result in cache line sized
	accesses to the end point */
	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
	/* VM_RESERVED: prevent the pages from being swapped out */
	vma->vm_flags |= VM_RESERVED;
	vma->vm_private_data = file->private_data;

	/* vaddr is the (virtual) address returned by pci_alloc_consistent();
	 * vsize is the corresponding size */

	start = vma->vm_start;
	/* size is page-aligned */
	while (vsize > 0) {
		printk(KERN_DEBUG "vaddr = %p\n", lro_char->engine->ringbuffer_virt);
		printk(KERN_DEBUG "start = %p\n", start);
		struct page *page = virt_to_page(vaddr);
		printk(KERN_DEBUG "page = %p\n", page);
		printk(KERN_DEBUG "vm_insert_page(...0x%08lx)\n", (unsigned long)vaddr);
		/* insert the given page into vma, mapped at the given start address */
		err = vm_insert_page(vma, start, page);
		if (err) {
			printk(KERN_DEBUG "vm_insert_page()\n");
			return err;
		}
		start += PAGE_SIZE;
		vaddr += PAGE_SIZE;
		vsize -= PAGE_SIZE;
	}
	return 0;
}

which hard crashes my system.

Any ideas on a generic function that mmap() pci_alloc_consistent()
memory to user space?

Thanks,
-- 
Leon

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

[Leon Woestenberg]

Hello,

On Wed, May 18, 2011 at 9:35 PM, Leon Woestenberg
<leon.woestenberg@gmail.com> wrote:
> On Wed, May 18, 2011 at 5:40 PM, Konrad Rzeszutek Wilk
>>> > On Wed, May 18, 2011 at 03:02:30PM +0200, Leon Woestenberg wrote:
>>> >>
>>> >> What is the correct implementation of the driver mmap (file operation
>>> >> method) for such memory?
>>> >

I have written an implementation based on vm_insert_pfn() and friends,
and posted the code in a new thread.

It doesn't work yet but I hope some of you kernel experts can look along.

Regards,
-- 
Leon

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: driver mmap implementation for memory allocated with pci_alloc_consistent()?

[Leon Woestenberg]

Hello,

On Thu, May 19, 2011 at 3:19 AM, Shaohua Li <shaohua.li@intel.com> wrote:
> On Wed, 2011-05-18 at 21:02 +0800, Leon Woestenberg wrote:
> why use pci_alloc_consistent? you can allocate pages and mmap it to
> userspace. when you want to do dma, you can use pci_map_page to get dma
> address for the pages and do whatever.
>
Thanks for thinking along.

I need contiguous memory in this case.  But yes, I have just found out
that __get_free_pages() with pci_map_single()  does work with my
mmap() fault() handler.
See my other thread with the code posted.

I just want to understand how this would work with
pci_alloc_consistent(), as that is the generic interface for PCI
drivers.

Note that the latter provides consistent / coherent mapping, whereas
pci_map_single() does not in general.   On x86 it probably is the same
due to bus-snooping (right?).

Regards,
--
Leon

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>