scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Michal Simek]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Hi Rusty and Jens,

I am getting problem with your patch which you have added to the kernel.
The problem is with my arm zynq remoteproc driver where
I use dma_declare_coherent_memory() to specify memory for remoteproc
which is ioremap to the vmalloc area.

Based on that buf addr is not inside ram even this coherent memory is in ram.
That's why virt_addr_valid(buf) is failing.

Thanks for you input,
Michal
-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Rusty Russell]

Michal Simek <monstr@monstr.eu> writes:
> Hi Rusty and Jens,
>
> I am getting problem with your patch which you have added to the kernel.
> The problem is with my arm zynq remoteproc driver where
> I use dma_declare_coherent_memory() to specify memory for remoteproc
> which is ioremap to the vmalloc area.
>
> Based on that buf addr is not inside ram even this coherent memory is in ram.
> That's why virt_addr_valid(buf) is failing.

But sg_set_buf() calls virt_to_page(buf).  So does that work?

I would think calling sg_set_page() directly would be what you want
here...

Cheers,
Rusty.

Re: scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Michal Simek]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

+ Ohad

On 07/22/2013 03:47 AM, Rusty Russell wrote:
> Michal Simek <monstr@monstr.eu> writes:
>> Hi Rusty and Jens,
>>
>> I am getting problem with your patch which you have added to the kernel.
>> The problem is with my arm zynq remoteproc driver where
>> I use dma_declare_coherent_memory() to specify memory for remoteproc
>> which is ioremap to the vmalloc area.
>>
>> Based on that buf addr is not inside ram even this coherent memory is in ram.
>> That's why virt_addr_valid(buf) is failing.
> 
> But sg_set_buf() calls virt_to_page(buf).  So does that work?
> 
> I would think calling sg_set_page() directly would be what you want
> here...

Let me take some code from virtio_rpmsg_bus.c to show that problematic part.

bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
				RPMSG_TOTAL_BUF_SPACE,
				&vrp->bufs_dma, GFP_KERNEL);
vrp->rbufs = bufs_va;
for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
	struct scatterlist sg;
	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;

	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
}


dma_alloc_coherent returns ioremaped ram address and sg_init_one()
is checking if addr is valid.

It is no problem to call instead of sg_init_one()
	sg_init_table(sg, 1);
	sg_set_page(sg, virt_to_page(cpu_addr), RPMSG_BUF_SIZE, offset_in_page(cpu_addr));


But my question is if SG lists have to be used just for memory which is not remapped.

Ohad: Can you see similar problem with your remoteproc drivers when DEBUG_SG is enabled?

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Rusty Russell]

Michal Simek <monstr@monstr.eu> writes:
> + Ohad
>
> On 07/22/2013 03:47 AM, Rusty Russell wrote:
>> Michal Simek <monstr@monstr.eu> writes:
>>> Hi Rusty and Jens,
>>>
>>> I am getting problem with your patch which you have added to the kernel.
>>> The problem is with my arm zynq remoteproc driver where
>>> I use dma_declare_coherent_memory() to specify memory for remoteproc
>>> which is ioremap to the vmalloc area.
>>>
>>> Based on that buf addr is not inside ram even this coherent memory is in ram.
>>> That's why virt_addr_valid(buf) is failing.
>> 
>> But sg_set_buf() calls virt_to_page(buf).  So does that work?
>> 
>> I would think calling sg_set_page() directly would be what you want
>> here...
>
> Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
>
> bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
> 				RPMSG_TOTAL_BUF_SPACE,
> 				&vrp->bufs_dma, GFP_KERNEL);
> vrp->rbufs = bufs_va;
> for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
> 	struct scatterlist sg;
> 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
>
> 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
> }

Hmm.  Looking at arch/arm/include/asm/memory.h:

#define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)

pfn_to_page in ARM seems to be the asm-generic one, which depends on
CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.

Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
that you shouldn't call virt_to_page() on something for which
!virt_addr_valid().

Cheers,
Rusty.

scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Rusty Russell]

Michal Simek <monstr@monstr.eu> writes:
> + Ohad
>
> On 07/22/2013 03:47 AM, Rusty Russell wrote:
>> Michal Simek <monstr@monstr.eu> writes:
>>> Hi Rusty and Jens,
>>>
>>> I am getting problem with your patch which you have added to the kernel.
>>> The problem is with my arm zynq remoteproc driver where
>>> I use dma_declare_coherent_memory() to specify memory for remoteproc
>>> which is ioremap to the vmalloc area.
>>>
>>> Based on that buf addr is not inside ram even this coherent memory is in ram.
>>> That's why virt_addr_valid(buf) is failing.
>> 
>> But sg_set_buf() calls virt_to_page(buf).  So does that work?
>> 
>> I would think calling sg_set_page() directly would be what you want
>> here...
>
> Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
>
> bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
> 				RPMSG_TOTAL_BUF_SPACE,
> 				&vrp->bufs_dma, GFP_KERNEL);
> vrp->rbufs = bufs_va;
> for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
> 	struct scatterlist sg;
> 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
>
> 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
> }

Hmm.  Looking at arch/arm/include/asm/memory.h:

#define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)

pfn_to_page in ARM seems to be the asm-generic one, which depends on
CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.

Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
that you shouldn't call virt_to_page() on something for which
!virt_addr_valid().

Cheers,
Rusty.

Re: scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Russell King - ARM Linux]

On Tue, Jul 23, 2013 at 12:00:30PM +0930, Rusty Russell wrote:
> Michal Simek <monstr@monstr.eu> writes:
> > Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
> >
> > bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
> > 				RPMSG_TOTAL_BUF_SPACE,
> > 				&vrp->bufs_dma, GFP_KERNEL);
> > vrp->rbufs = bufs_va;
> > for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
> > 	struct scatterlist sg;
> > 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
> >
> > 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
> > }
> 
> Hmm.  Looking at arch/arm/include/asm/memory.h:
> 
> #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
> #define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
> 
> pfn_to_page in ARM seems to be the asm-generic one, which depends on
> CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.
> 
> Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
> that you shouldn't call virt_to_page() on something for which
> !virt_addr_valid().

The above code fragment is just wrong.  You can't make any assumptions about
the memory returned from dma_alloc_coherent(), because:

- On x86, it is kernel direct mapped memory, where things like
  virt_to_phys(cpuaddr) will work fine with it.

- On any architecture which needs to remap memory to make it coherent
  with the DMA device, the key word there is "remap" - it's not kernel
  direct mapped memory, and virt_to_phys(cpuaddr) on it is illegal.

The only valid operation with scatterlists and such memory returned from
dma_alloc_coherent() is:

	sg_dma_address(sg) = dma_address;
	sg_dma_len(sg) = dma_length;

and not to use the virtual address(es) at all in the scatterlist.  And
it is very very important to realise that you must not mix the streaming
and coherent DMA APIs - you must never pass coherent memory into the
dma_map_* functions.

scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Russell King - ARM Linux]

On Tue, Jul 23, 2013 at 12:00:30PM +0930, Rusty Russell wrote:
> Michal Simek <monstr@monstr.eu> writes:
> > Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
> >
> > bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
> > 				RPMSG_TOTAL_BUF_SPACE,
> > 				&vrp->bufs_dma, GFP_KERNEL);
> > vrp->rbufs = bufs_va;
> > for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
> > 	struct scatterlist sg;
> > 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
> >
> > 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
> > }
> 
> Hmm.  Looking at arch/arm/include/asm/memory.h:
> 
> #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
> #define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
> 
> pfn_to_page in ARM seems to be the asm-generic one, which depends on
> CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.
> 
> Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
> that you shouldn't call virt_to_page() on something for which
> !virt_addr_valid().

The above code fragment is just wrong.  You can't make any assumptions about
the memory returned from dma_alloc_coherent(), because:

- On x86, it is kernel direct mapped memory, where things like
  virt_to_phys(cpuaddr) will work fine with it.

- On any architecture which needs to remap memory to make it coherent
  with the DMA device, the key word there is "remap" - it's not kernel
  direct mapped memory, and virt_to_phys(cpuaddr) on it is illegal.

The only valid operation with scatterlists and such memory returned from
dma_alloc_coherent() is:

	sg_dma_address(sg) = dma_address;
	sg_dma_len(sg) = dma_length;

and not to use the virtual address(es) at all in the scatterlist.  And
it is very very important to realise that you must not mix the streaming
and coherent DMA APIs - you must never pass coherent memory into the
dma_map_* functions.

Re: scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Michal Simek]

[-- Attachment #1: Type: text/plain, Size: 2525 bytes --]

On 07/23/2013 12:37 PM, Russell King - ARM Linux wrote:
> On Tue, Jul 23, 2013 at 12:00:30PM +0930, Rusty Russell wrote:
>> Michal Simek <monstr@monstr.eu> writes:
>>> Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
>>>
>>> bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
>>> 				RPMSG_TOTAL_BUF_SPACE,
>>> 				&vrp->bufs_dma, GFP_KERNEL);
>>> vrp->rbufs = bufs_va;
>>> for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
>>> 	struct scatterlist sg;
>>> 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
>>>
>>> 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
>>> }
>>
>> Hmm.  Looking at arch/arm/include/asm/memory.h:
>>
>> #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
>> #define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
>>
>> pfn_to_page in ARM seems to be the asm-generic one, which depends on
>> CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.
>>
>> Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
>> that you shouldn't call virt_to_page() on something for which
>> !virt_addr_valid().
> 
> The above code fragment is just wrong.  You can't make any assumptions about
> the memory returned from dma_alloc_coherent(), because:
> 
> - On x86, it is kernel direct mapped memory, where things like
>   virt_to_phys(cpuaddr) will work fine with it.
> 
> - On any architecture which needs to remap memory to make it coherent
>   with the DMA device, the key word there is "remap" - it's not kernel
>   direct mapped memory, and virt_to_phys(cpuaddr) on it is illegal.
> 
> The only valid operation with scatterlists and such memory returned from
> dma_alloc_coherent() is:
> 
> 	sg_dma_address(sg) = dma_address;
> 	sg_dma_len(sg) = dma_length;
> 
> and not to use the virtual address(es) at all in the scatterlist.  And
> it is very very important to realise that you must not mix the streaming
> and coherent DMA APIs - you must never pass coherent memory into the
> dma_map_* functions.

Ohad: This code is probably yours and you know the whole history of it.
Can you please comment it?

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

scatterlist: sg_set_buf() argument must be in linear mapping (sha1: ac4e97abce9b80c020e7113325f49e58b7b15e3f)

[Michal Simek]

On 07/23/2013 12:37 PM, Russell King - ARM Linux wrote:
> On Tue, Jul 23, 2013 at 12:00:30PM +0930, Rusty Russell wrote:
>> Michal Simek <monstr@monstr.eu> writes:
>>> Let me take some code from virtio_rpmsg_bus.c to show that problematic part.
>>>
>>> bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
>>> 				RPMSG_TOTAL_BUF_SPACE,
>>> 				&vrp->bufs_dma, GFP_KERNEL);
>>> vrp->rbufs = bufs_va;
>>> for (i = 0; i < RPMSG_NUM_BUFS / 2; i++) {
>>> 	struct scatterlist sg;
>>> 	void *cpu_addr = vrp->rbufs + i * RPMSG_BUF_SIZE;
>>>
>>> 	sg_init_one(&sg, cpu_addr, RPMSG_BUF_SIZE);
>>> }
>>
>> Hmm.  Looking at arch/arm/include/asm/memory.h:
>>
>> #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
>> #define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
>>
>> pfn_to_page in ARM seems to be the asm-generic one, which depends on
>> CONFIG_SPARSEMEM/CONFIG_DISCONTIGMEM/CONFIG_FLATMEM etc.
>>
>> Perhaps virt_addr_valid() is wrong for your config?  It's pretty clear
>> that you shouldn't call virt_to_page() on something for which
>> !virt_addr_valid().
> 
> The above code fragment is just wrong.  You can't make any assumptions about
> the memory returned from dma_alloc_coherent(), because:
> 
> - On x86, it is kernel direct mapped memory, where things like
>   virt_to_phys(cpuaddr) will work fine with it.
> 
> - On any architecture which needs to remap memory to make it coherent
>   with the DMA device, the key word there is "remap" - it's not kernel
>   direct mapped memory, and virt_to_phys(cpuaddr) on it is illegal.
> 
> The only valid operation with scatterlists and such memory returned from
> dma_alloc_coherent() is:
> 
> 	sg_dma_address(sg) = dma_address;
> 	sg_dma_len(sg) = dma_length;
> 
> and not to use the virtual address(es) at all in the scatterlist.  And
> it is very very important to realise that you must not mix the streaming
> and coherent DMA APIs - you must never pass coherent memory into the
> dma_map_* functions.

Ohad: This code is probably yours and you know the whole history of it.
Can you please comment it?

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130723/e40ca146/attachment.sig>