DMA using data buffer vmapped in kernel space

DMA using data buffer vmapped in kernel space

[Thomas Koeller]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Hi,

I am writing a driver that loads a firmware blob into its device. The driver
calls request_firmware() and gets a struct firmware * that contains a
data buffer pointer in its 'data' member. The buffer is then passed to a
SPI driver to send it to the device.

The SPI driver uses DMA and, in preparation for that, eventually calls
dma_cache_maint() (contained in arch/arm/mm/dma-mapping.c). At this
point, the whole thing goes bad because there is a check:

	BUG_ON(!virt_addr_valid(start) || !virt_addr_valid(start + size - 1));

and the address obtained via request_firmware() fails this check.

The request_firmware() function creates the data buffer by allocating a
number of single pages in a loop, places them in an array, and finally
creates a kernel mapping by calling vmap(). The vmap() result is the
buffer address passed to the DMA.

I do not understand the purpose of the failing check. The virt_addr_valid()
macro checks whether its argument is below high_memory. But why would the
virtual address of a DMA data buffer matter at all? After all, the pages
are resident, so I cannot see any problem here. Can anybody explain?

Thomas

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

DMA using data buffer vmapped in kernel space

[Thomas Koeller]

Hi,

I am writing a driver that loads a firmware blob into its device. The driver
calls request_firmware() and gets a struct firmware * that contains a
data buffer pointer in its 'data' member. The buffer is then passed to a
SPI driver to send it to the device.

The SPI driver uses DMA and, in preparation for that, eventually calls
dma_cache_maint() (contained in arch/arm/mm/dma-mapping.c). At this
point, the whole thing goes bad because there is a check:

	BUG_ON(!virt_addr_valid(start) || !virt_addr_valid(start + size - 1));

and the address obtained via request_firmware() fails this check.

The request_firmware() function creates the data buffer by allocating a
number of single pages in a loop, places them in an array, and finally
creates a kernel mapping by calling vmap(). The vmap() result is the
buffer address passed to the DMA.

I do not understand the purpose of the failing check. The virt_addr_valid()
macro checks whether its argument is below high_memory. But why would the
virtual address of a DMA data buffer matter at all? After all, the pages
are resident, so I cannot see any problem here. Can anybody explain?

Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20100304/11d54161/attachment.sig>

Re: DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Thu, Mar 04, 2010 at 10:11:08PM +0100, Thomas Koeller wrote:
> I do not understand the purpose of the failing check. The virt_addr_valid()
> macro checks whether its argument is below high_memory. But why would the
> virtual address of a DMA data buffer matter at all? After all, the pages
> are resident, so I cannot see any problem here. Can anybody explain?

Cache maintainence is done using virtual addresses for L1, and
physical addresses for L2.  There's the need for virtual addresses
to be translatable to physical addresses, which is only true for
the kernel direct mapped region (pages between PAGE_OFFSET and
high_memory).

DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Thu, Mar 04, 2010 at 10:11:08PM +0100, Thomas Koeller wrote:
> I do not understand the purpose of the failing check. The virt_addr_valid()
> macro checks whether its argument is below high_memory. But why would the
> virtual address of a DMA data buffer matter at all? After all, the pages
> are resident, so I cannot see any problem here. Can anybody explain?

Cache maintainence is done using virtual addresses for L1, and
physical addresses for L2.  There's the need for virtual addresses
to be translatable to physical addresses, which is only true for
the kernel direct mapped region (pages between PAGE_OFFSET and
high_memory).

Re: DMA using data buffer vmapped in kernel space

[Thomas Koeller]

Am Donnerstag, 4. März 2010 22:36:34 schrieb Russell King - ARM Linux:
> Cache maintainence is done using virtual addresses for L1, and
> physical addresses for L2.  There's the need for virtual addresses
> to be translatable to physical addresses, which is only true for
> the kernel direct mapped region (pages between PAGE_OFFSET and
> high_memory).

Isn't the mapping created by vmap() sufficient for the virt/phys
translation? In which way is this case different from a buffer
passed in from user space, where the constituent pages are not
in the directly mapped kernel region either?

tk

DMA using data buffer vmapped in kernel space

[Thomas Koeller]

Am Donnerstag, 4. M?rz 2010 22:36:34 schrieb Russell King - ARM Linux:
> Cache maintainence is done using virtual addresses for L1, and
> physical addresses for L2.  There's the need for virtual addresses
> to be translatable to physical addresses, which is only true for
> the kernel direct mapped region (pages between PAGE_OFFSET and
> high_memory).

Isn't the mapping created by vmap() sufficient for the virt/phys
translation? In which way is this case different from a buffer
passed in from user space, where the constituent pages are not
in the directly mapped kernel region either?

tk

Re: DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Sat, Mar 06, 2010 at 02:07:12PM +0100, Thomas Koeller wrote:
> Am Donnerstag, 4. März 2010 22:36:34 schrieb Russell King - ARM Linux:
> > Cache maintainence is done using virtual addresses for L1, and
> > physical addresses for L2.  There's the need for virtual addresses
> > to be translatable to physical addresses, which is only true for
> > the kernel direct mapped region (pages between PAGE_OFFSET and
> > high_memory).
> 
> Isn't the mapping created by vmap() sufficient for the virt/phys
> translation? In which way is this case different from a buffer
> passed in from user space, where the constituent pages are not
> in the directly mapped kernel region either?

No different.

The requirement is that dma_map_single() is passed a virtual address
in the kernel direct-mapped memory region, which is translatable using
virt_to_phys() and friends.

Anything which requires a page table lookup to obtain the physical
address is just not acceptable - that requires taking locks and other
messy things, plus is grossly inefficient.

DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Sat, Mar 06, 2010 at 02:07:12PM +0100, Thomas Koeller wrote:
> Am Donnerstag, 4. M?rz 2010 22:36:34 schrieb Russell King - ARM Linux:
> > Cache maintainence is done using virtual addresses for L1, and
> > physical addresses for L2.  There's the need for virtual addresses
> > to be translatable to physical addresses, which is only true for
> > the kernel direct mapped region (pages between PAGE_OFFSET and
> > high_memory).
> 
> Isn't the mapping created by vmap() sufficient for the virt/phys
> translation? In which way is this case different from a buffer
> passed in from user space, where the constituent pages are not
> in the directly mapped kernel region either?

No different.

The requirement is that dma_map_single() is passed a virtual address
in the kernel direct-mapped memory region, which is translatable using
virt_to_phys() and friends.

Anything which requires a page table lookup to obtain the physical
address is just not acceptable - that requires taking locks and other
messy things, plus is grossly inefficient.

Re: DMA using data buffer vmapped in kernel space

[Lin Mac]

2010/3/7 Russell King - ARM Linux <linux@arm.linux.org.uk>:
> On Sat, Mar 06, 2010 at 02:07:12PM +0100, Thomas Koeller wrote:
>> Am Donnerstag, 4. März 2010 22:36:34 schrieb Russell King - ARM Linux:
>> > Cache maintainence is done using virtual addresses for L1, and
>> > physical addresses for L2.  There's the need for virtual addresses
>> > to be translatable to physical addresses, which is only true for
>> > the kernel direct mapped region (pages between PAGE_OFFSET and
>> > high_memory).
>>
>> Isn't the mapping created by vmap() sufficient for the virt/phys
>> translation? In which way is this case different from a buffer
>> passed in from user space, where the constituent pages are not
>> in the directly mapped kernel region either?
>
> No different.
>
> The requirement is that dma_map_single() is passed a virtual address
> in the kernel direct-mapped memory region, which is translatable using
> virt_to_phys() and friends.
I had encounter a similiar problem and I simply allocated a new
buffer, copy the data, then DMA. It seems slow and stupid.

I'm wondering wether could I translate the vmap virt to phys(don't
know how to yet), then use phys_to_virt to get the virt in
direct-mapped memory region?

Is there other possible ways?

> Anything which requires a page table lookup to obtain the physical
> address is just not acceptable - that requires taking locks and other
> messy things, plus is grossly inefficient.

Best Regards,
Mac Lin

DMA using data buffer vmapped in kernel space

[Lin Mac]

2010/3/7 Russell King - ARM Linux <linux@arm.linux.org.uk>:
> On Sat, Mar 06, 2010 at 02:07:12PM +0100, Thomas Koeller wrote:
>> Am Donnerstag, 4. M?rz 2010 22:36:34 schrieb Russell King - ARM Linux:
>> > Cache maintainence is done using virtual addresses for L1, and
>> > physical addresses for L2. ?There's the need for virtual addresses
>> > to be translatable to physical addresses, which is only true for
>> > the kernel direct mapped region (pages between PAGE_OFFSET and
>> > high_memory).
>>
>> Isn't the mapping created by vmap() sufficient for the virt/phys
>> translation? In which way is this case different from a buffer
>> passed in from user space, where the constituent pages are not
>> in the directly mapped kernel region either?
>
> No different.
>
> The requirement is that dma_map_single() is passed a virtual address
> in the kernel direct-mapped memory region, which is translatable using
> virt_to_phys() and friends.
I had encounter a similiar problem and I simply allocated a new
buffer, copy the data, then DMA. It seems slow and stupid.

I'm wondering wether could I translate the vmap virt to phys(don't
know how to yet), then use phys_to_virt to get the virt in
direct-mapped memory region?

Is there other possible ways?

> Anything which requires a page table lookup to obtain the physical
> address is just not acceptable - that requires taking locks and other
> messy things, plus is grossly inefficient.

Best Regards,
Mac Lin

Re: DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Tue, Mar 09, 2010 at 03:45:54PM +0800, Lin Mac wrote:
> I had encounter a similiar problem and I simply allocated a new
> buffer, copy the data, then DMA. It seems slow and stupid.
> 
> I'm wondering wether could I translate the vmap virt to phys(don't
> know how to yet), then use phys_to_virt to get the virt in
> direct-mapped memory region?
> 
> Is there other possible ways?

I think you need to talk to the firmware people... which is unfortunately
orphaned.

What you could do is prepare a patch to allow the firmware support to
export a scatterlist via the struct firmware, and then use that with
dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.

DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Tue, Mar 09, 2010 at 03:45:54PM +0800, Lin Mac wrote:
> I had encounter a similiar problem and I simply allocated a new
> buffer, copy the data, then DMA. It seems slow and stupid.
> 
> I'm wondering wether could I translate the vmap virt to phys(don't
> know how to yet), then use phys_to_virt to get the virt in
> direct-mapped memory region?
> 
> Is there other possible ways?

I think you need to talk to the firmware people... which is unfortunately
orphaned.

What you could do is prepare a patch to allow the firmware support to
export a scatterlist via the struct firmware, and then use that with
dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.

Re: DMA using data buffer vmapped in kernel space

[David Miller]

From: Russell King - ARM Linux <linux@arm.linux.org.uk>
Date: Tue, 9 Mar 2010 09:14:03 +0000

> What you could do is prepare a patch to allow the firmware support to
> export a scatterlist via the struct firmware, and then use that with
> dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.

It's so common to want a scatterlist from a vmalloc/vmap area
that a generic common helper is probably the best.

Just iterate over the vmalloc/vmap area using vmalloc_to_page()
to fill in the SG entries.

DMA using data buffer vmapped in kernel space

[David Miller]

From: Russell King - ARM Linux <linux@arm.linux.org.uk>
Date: Tue, 9 Mar 2010 09:14:03 +0000

> What you could do is prepare a patch to allow the firmware support to
> export a scatterlist via the struct firmware, and then use that with
> dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.

It's so common to want a scatterlist from a vmalloc/vmap area
that a generic common helper is probably the best.

Just iterate over the vmalloc/vmap area using vmalloc_to_page()
to fill in the SG entries.

Re: DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Tue, Mar 09, 2010 at 01:29:35AM -0800, David Miller wrote:
> From: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Date: Tue, 9 Mar 2010 09:14:03 +0000
> 
> > What you could do is prepare a patch to allow the firmware support to
> > export a scatterlist via the struct firmware, and then use that with
> > dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.
> 
> It's so common to want a scatterlist from a vmalloc/vmap area
> that a generic common helper is probably the best.
> 
> Just iterate over the vmalloc/vmap area using vmalloc_to_page()
> to fill in the SG entries.

It looks like the firmware interface may return a pointer to either the
kernel binary itself for built-in firmware, or to a set of pages which
have been vmap'd.  In the former case, vmalloc_to_page() will fail.

DMA using data buffer vmapped in kernel space

[Russell King - ARM Linux]

On Tue, Mar 09, 2010 at 01:29:35AM -0800, David Miller wrote:
> From: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Date: Tue, 9 Mar 2010 09:14:03 +0000
> 
> > What you could do is prepare a patch to allow the firmware support to
> > export a scatterlist via the struct firmware, and then use that with
> > dma_map_sg().  Let's hope that the SPI interfaces support scatterlists.
> 
> It's so common to want a scatterlist from a vmalloc/vmap area
> that a generic common helper is probably the best.
> 
> Just iterate over the vmalloc/vmap area using vmalloc_to_page()
> to fill in the SG entries.

It looks like the firmware interface may return a pointer to either the
kernel binary itself for built-in firmware, or to a set of pages which
have been vmap'd.  In the former case, vmalloc_to_page() will fail.

Re: DMA using data buffer vmapped in kernel space

[David Miller]

From: Russell King - ARM Linux <linux@arm.linux.org.uk>
Date: Tue, 9 Mar 2010 10:16:11 +0000

> It looks like the firmware interface may return a pointer to either the
> kernel binary itself for built-in firmware, or to a set of pages which
> have been vmap'd.  In the former case, vmalloc_to_page() will fail.

Indeed, then you're right, the firmware layer would need to provide
the interface since only it knows where the memory is.

However, there's also no real portable way to compute physical
addresses from kernel image virtual addresses.

BTW, it's been my experience that drivers copy the firmware into a
locally allocated buffer exactly to avoid this problem.

DMA using data buffer vmapped in kernel space

[David Miller]

From: Russell King - ARM Linux <linux@arm.linux.org.uk>
Date: Tue, 9 Mar 2010 10:16:11 +0000

> It looks like the firmware interface may return a pointer to either the
> kernel binary itself for built-in firmware, or to a set of pages which
> have been vmap'd.  In the former case, vmalloc_to_page() will fail.

Indeed, then you're right, the firmware layer would need to provide
the interface since only it knows where the memory is.

However, there's also no real portable way to compute physical
addresses from kernel image virtual addresses.

BTW, it's been my experience that drivers copy the firmware into a
locally allocated buffer exactly to avoid this problem.