dma_alloc_coherent fragmentation

dma_alloc_coherent fragmentation

[Kamat, Nishant]

Forwarding on behalf of Nishanth Menon:

-----Original Message-----
From: Menon, Nishanth 
Sent: Tuesday, March 17, 2009 8:17 AM
To: 'linux-arm-kernel@lists.arm.linux.org.uk'
Cc: Gupta, Ramesh; Kevin Hilman; linux-omap@vger.kernel.org; Ramirez Luna, Omar; Kanigeri, Hari; Ameya Palande; Guzman Lugo, Fernando; Ameya Palande
Subject: dma_alloc_coherent fragmentation

Looping in linux-arm ML:

Discussion Ref: [1](linux-omap mailing list)
While working with Linux OMAP kernel[2] we found that on allocating 4 meg chunks using dma_alloc_coherent and de-allocating with dma_free_coherent in a loop using a test driver[7], the memory is getting fragmented as shown by Ameya's observation [3] finally resulting in dump_stack due to lack of pages.

Searching through previous archives, found [4] which seem to imply that allocating chunks greater than 1 page could cause fragmentation.

I wonder why we see this even if we allocate and free the memory in sequence as in [5]. This seems to happen on 2.6.28 and 2.6.29 but not on the 2.6.27 based kernel[6].

Regards,
Nishanth Menon

Ref:
[1] http://marc.info/?t=123719779100002&r=1&w=2
[2] http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git
[3] http://marc.info/?l=linux-omap&m=123722460022336&w=2
[4] http://lkml.indiana.edu/hypermail/linux/kernel/0706.0/0370.html 
[5] http://marc.info/?l=linux-omap&m=123719772811746&w=2
[6] http://git.omapzoom.com/?p=repo/omapkernel.git;
[7] http://marc.info/?l=linux-omap&m=123719772811746&q=p3 

Re: dma_alloc_coherent fragmentation

[Russell King - ARM Linux]

On Wed, Mar 18, 2009 at 08:03:45AM -0500, Kamat, Nishant wrote:
> Forwarding on behalf of Nishanth Menon:

Make sure you're up to date with fixes to the ioremap code, specifically
24f11ec001920f1cfaeeed8e8b55725d900bbb56.  This is not in 2.6.28,
2.6.29-rc1, 2.6.29-rc2, but is in -rc3 and later.  2.6.29 itself hasn't
been released yet so I don't know what "seems to happen on 2.6.28 and
2.6.29" actually means because its nonsense.

So, please test with 2.6.29-rc3 or later.

RE: dma_alloc_coherent fragmentation

[Menon, Nishanth]

> -----Original Message-----
> From: Russell King - ARM Linux [mailto:linux@arm.linux.org.uk]
> Sent: Wednesday, March 18, 2009 9:25 PM
> To: Kamat, Nishant
> Cc: linux-arm-kernel@lists.arm.linux.org.uk; Menon, Nishanth; linux-
> omap@vger.kernel.org; Ramirez Luna, Omar; Kanigeri, Hari; Guzman Lugo,
> Fernando; Kevin Hilman; 2ameya@gmail.com
> Subject: Re: dma_alloc_coherent fragmentation

> Make sure you're up to date with fixes to the ioremap code, specifically
> 24f11ec001920f1cfaeeed8e8b55725d900bbb56.  This is not in 2.6.28,
> 2.6.29-rc1, 2.6.29-rc2, but is in -rc3 and later.  2.6.29 itself hasn't
> been released yet so I don't know what "seems to happen on 2.6.28 and
> 2.6.29" actually means because its nonsense.

Apologies on the confusion.

For this issue, the pertinent tree would be the master branch of [1] linux-omap tree which is synced to 2.6.29-rc8. And yes, the mentioned commit ID is part of the git log.

Relevant test code and report of the crash is in [2]. Additional observation in [3]

Regards,
Nishanth Menon

Ref:
[1] http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git
[2] http://marc.info/?l=linux-omap&m=123719772811746&w=2
[3] http://marc.info/?l=linux-omap&m=123722460022336&w=2

Re: dma_alloc_coherent fragmentation

[Hiroshi DOYU]

Hi Nishant,

From: "ext Kamat, Nishant" <nskamat@ti.com>
Subject: dma_alloc_coherent fragmentation
Date: Wed, 18 Mar 2009 14:03:45 +0100

> Forwarding on behalf of Nishanth Menon:
> 
> -----Original Message-----
> From: Menon, Nishanth 
> Sent: Tuesday, March 17, 2009 8:17 AM
> To: 'linux-arm-kernel@lists.arm.linux.org.uk'
> Cc: Gupta, Ramesh; Kevin Hilman; linux-omap@vger.kernel.org; Ramirez Luna, Omar; Kanigeri, Hari; Ameya Palande; Guzman Lugo, Fernando; Ameya Palande
> Subject: dma_alloc_coherent fragmentation
> 
> Looping in linux-arm ML:
> 
> Discussion Ref: [1](linux-omap mailing list)
> While working with Linux OMAP kernel[2] we found that on allocating
> 4 meg chunks using dma_alloc_coherent and de-allocating with
> dma_free_coherent in a loop using a test driver[7], the memory is
> getting fragmented as shown by Ameya's observation [3] finally
> resulting in dump_stack due to lack of pages.

What would happen with __GFP_RECLAIMABLE for dma_alloc_coherent()?

RE: dma_alloc_coherent fragmentation

[Menon, Nishanth]

Hi Hiroshi,
> -----Original Message-----
> From: Hiroshi DOYU [mailto:Hiroshi.DOYU@nokia.com]
> Sent: Thursday, March 19, 2009 8:43 AM

> 
> 
> What would happen with __GFP_RECLAIMABLE for dma_alloc_coherent()?
As per [1], GFP_TEMPORARY might be the right approach.. but yes,
Changing dummy to do:
dma_alloc_coherent(NULL, b[i].size, &b[i].phy, GFP_TEMPORARY);

Seems to resolve the issue with the dummy driver.

The question however still remains: For a driver to allocate a memory which is not temporary in nature we'd recommend GFP_KERNEL. But as part of the exit sequence the driver will deallocate that memory and as part of init sequence, reallocate. Given that the memory is not reclaimed immediately, it looks like the only way to ensure that would be to do GFP_KERNEL | __GFP_RECLAIMABLE (which essentially is the definition of GFP_TEMPORARY).

Regards,
Nishanth Menon
Ref:
[1] http://lkml.indiana.edu/hypermail/linux/kernel/0705.2/0638.html

Re: dma_alloc_coherent fragmentation

[pramod gurav]

Hi Nishant,
I am facing the same problem to allocate a chunk of 4MB on omap3evm
using dma_alloc_coherent. I used the test driver mentioned here
http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html

I applied the patch for
ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
Russell, but stiill seeing the error saying

coherent allocation too big (requested 0x400000 mask 0xffffffff)
alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
Allocation failed idx=0
insmod: cannot insert `dummy.ko': Cannot allocate memory (-1): Cannot
allocate memory

I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
__GFP_RECLAIMABLE), but failed to insmod the module.
I am working on linux-omap 2.6.28 kernel. Even I could not allocate
memory size greater than 1MB.
Has this been fixed? Are there any patches to fix this issue?

On Thu, Mar 19, 2009 at 2:36 PM, Menon, Nishanth <nm@ti.com> wrote:
> Hi Hiroshi,
>> -----Original Message-----
>> From: Hiroshi DOYU [mailto:Hiroshi.DOYU@nokia.com]
>> Sent: Thursday, March 19, 2009 8:43 AM
>
>>
>>
>> What would happen with __GFP_RECLAIMABLE for dma_alloc_coherent()?
> As per [1], GFP_TEMPORARY might be the right approach.. but yes,
> Changing dummy to do:
> dma_alloc_coherent(NULL, b[i].size, &b[i].phy, GFP_TEMPORARY);
>
> Seems to resolve the issue with the dummy driver.
>
> The question however still remains: For a driver to allocate a memory which is not temporary in nature we'd recommend GFP_KERNEL. But as part of the exit sequence the driver will deallocate that memory and as part of init sequence, reallocate. Given that the memory is not reclaimed immediately, it looks like the only way to ensure that would be to do GFP_KERNEL | __GFP_RECLAIMABLE (which essentially is the definition of GFP_TEMPORARY).
>
> Regards,
> Nishanth Menon
> Ref:
> [1] http://lkml.indiana.edu/hypermail/linux/kernel/0705.2/0638.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Best Regards
Pramod
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: dma_alloc_coherent fragmentation

[Shilimkar, Santosh]

DMA pool is better solution for such usecases.
> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
> Sent: Wednesday, April 22, 2009 1:17 PM
> To: Menon, Nishanth
> Cc: linux-omap@vger.kernel.org
> Subject: Re: dma_alloc_coherent fragmentation
> 
> Hi Nishant,
> I am facing the same problem to allocate a chunk of 4MB on omap3evm
> using dma_alloc_coherent. I used the test driver mentioned here
> http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html
> 
> I applied the patch for
> ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
> Russell, but stiill seeing the error saying
> 
> coherent allocation too big (requested 0x400000 mask 0xffffffff)
> alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
> Allocation failed idx=0
> insmod: cannot insert `dummy.ko': Cannot allocate memory (-1): Cannot
> allocate memory
> 
> I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
> __GFP_RECLAIMABLE), but failed to insmod the module.
> I am working on linux-omap 2.6.28 kernel. Even I could not allocate
> memory size greater than 1MB.
> Has this been fixed? Are there any patches to fix this issue?
> 
> On Thu, Mar 19, 2009 at 2:36 PM, Menon, Nishanth <nm@ti.com> wrote:

Re: dma_alloc_coherent fragmentation

[pramod gurav]

Thanks for the quick reply.
Can you please point me to some links on DMA pools. I am just blank on this. :(
I will still search for this in net.
-- 
Best Regards
Pramod

On Wed, Apr 22, 2009 at 1:20 PM, Shilimkar, Santosh
<santosh.shilimkar@ti.com> wrote:
> DMA pool is better solution for such usecases.
>> -----Original Message-----
>> From: linux-omap-owner@vger.kernel.org
>> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
>> Sent: Wednesday, April 22, 2009 1:17 PM
>> To: Menon, Nishanth
>> Cc: linux-omap@vger.kernel.org
>> Subject: Re: dma_alloc_coherent fragmentation
>>
>> Hi Nishant,
>> I am facing the same problem to allocate a chunk of 4MB on omap3evm
>> using dma_alloc_coherent. I used the test driver mentioned here
>> http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html
>>
>> I applied the patch for
>> ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
>> Russell, but stiill seeing the error saying
>>
>> coherent allocation too big (requested 0x400000 mask 0xffffffff)
>> alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
>> Allocation failed idx=0
>> insmod: cannot insert `dummy.ko': Cannot allocate memory (-1): Cannot
>> allocate memory
>>
>> I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
>> __GFP_RECLAIMABLE), but failed to insmod the module.
>> I am working on linux-omap 2.6.28 kernel. Even I could not allocate
>> memory size greater than 1MB.
>> Has this been fixed? Are there any patches to fix this issue?
>>
>> On Thu, Mar 19, 2009 at 2:36 PM, Menon, Nishanth <nm@ti.com> wrote:
>

RE: dma_alloc_coherent fragmentation

[Shilimkar, Santosh]

The actual apis are in mm/dmapool.c. There are quite a few users in the usb stack.

Here are few examples.

Amd5536udc.c (drivers\usb\gadget):	
Buffer.c (drivers\usb\core):
Ci13xxx_udc.c (drivers\usb\gadget)
Ci13xxx_udc.c (drivers\usb\gadget)
Cppi_dma.c (drivers\usb\musb):	
Ehci-mem.c (drivers\usb\host):
Ehci-mem.c (drivers\usb\host):
Ehci-mem.c (drivers\usb\host):
Ehci-mem.c (drivers\usb\host):
Ixp4xx_crypto.c (drivers\crypto):	
Ixp4xx_crypto.c (drivers\crypto):	
Ohci-mem.c (drivers\usb\host):
Ohci-mem.c (drivers\usb\host):
Uhci-hcd.c (drivers\usb\host):
Uhci-hcd.c (drivers\usb\host):

Regards,
Santosh
 

> -----Original Message-----
> From: pramod gurav [mailto:pramodforum@gmail.com] 
> Sent: Wednesday, April 22, 2009 1:54 PM
> To: Shilimkar, Santosh
> Cc: Menon, Nishanth; linux-omap@vger.kernel.org
> Subject: Re: dma_alloc_coherent fragmentation
> 
> Thanks for the quick reply.
> Can you please point me to some links on DMA pools. I am just 
> blank on this. :(
> I will still search for this in net.
> -- 

RE: dma_alloc_coherent fragmentation

[Gupta, Ramesh]

Hi Pramod,
 

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
> Sent: Wednesday, April 22, 2009 1:17 PM
> To: Menon, Nishanth
> Cc: linux-omap@vger.kernel.org
> Subject: Re: dma_alloc_coherent fragmentation
> 
> Hi Nishant,
> I am facing the same problem to allocate a chunk of 4MB on omap3evm
> using dma_alloc_coherent. I used the test driver mentioned here
> http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html
> 
> I applied the patch for
> ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
> Russell, but stiill seeing the error saying
> 
> coherent allocation too big (requested 0x400000 mask 0xffffffff)
> alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
> Allocation failed idx=0
> insmod: cannot insert `dummy.ko': Cannot allocate memory (-1): Cannot
> allocate memory
> 
> I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
> __GFP_RECLAIMABLE), but failed to insmod the module.
> I am working on linux-omap 2.6.28 kernel. Even I could not allocate
> memory size greater than 1MB.
> Has this been fixed? Are there any patches to fix this issue?
> 

What is the value for 'CONFIG_FB_OMAP_CONSISTENT_DMA_SIZE' (default is 2 in omap_3430sdp_defconfig), Can you please
try increasing to 8?

Regards
Ramesh Gupta G

Re: dma_alloc_coherent fragmentation

[pramod gurav]

Hi Ramesh,
This worked. Thanks a lot.


-- 
Best Regards
Pramod


On Wed, Apr 22, 2009 at 2:44 PM, Gupta, Ramesh <grgupta@ti.com> wrote:
> Hi Pramod,
>
>
>> -----Original Message-----
>> From: linux-omap-owner@vger.kernel.org
>> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
>> Sent: Wednesday, April 22, 2009 1:17 PM
>> To: Menon, Nishanth
>> Cc: linux-omap@vger.kernel.org
>> Subject: Re: dma_alloc_coherent fragmentation
>>
>> Hi Nishant,
>> I am facing the same problem to allocate a chunk of 4MB on omap3evm
>> using dma_alloc_coherent. I used the test driver mentioned here
>> http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html
>>
>> I applied the patch for
>> ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
>> Russell, but stiill seeing the error saying
>>
>> coherent allocation too big (requested 0x400000 mask 0xffffffff)
>> alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
>> Allocation failed idx=0
>> insmod: cannot insert `dummy.ko': Cannot allocate memory (-1): Cannot
>> allocate memory
>>
>> I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
>> __GFP_RECLAIMABLE), but failed to insmod the module.
>> I am working on linux-omap 2.6.28 kernel. Even I could not allocate
>> memory size greater than 1MB.
>> Has this been fixed? Are there any patches to fix this issue?
>>
>
> What is the value for 'CONFIG_FB_OMAP_CONSISTENT_DMA_SIZE' (default is 2 in omap_3430sdp_defconfig), Can you please
> try increasing to 8?
>
> Regards
> Ramesh Gupta G
>

RE: dma_alloc_coherent fragmentation

[Shilimkar, Santosh]

This would just temporarily delay the problem. People have even tried with values of 14 MB and all. Nishant M knows one such recent issue.

Bye the way use of this macro for general DMAable memory is discouraged. You can find many threads on ARM mailing list. 

Regards,
Santosh
 

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
> Sent: Wednesday, April 22, 2009 3:14 PM
> To: Gupta, Ramesh
> Cc: Menon, Nishanth; linux-omap@vger.kernel.org
> Subject: Re: dma_alloc_coherent fragmentation
> 
> Hi Ramesh,
> This worked. Thanks a lot.
> 
> 
> -- 
> Best Regards
> Pramod
> 
> 
> On Wed, Apr 22, 2009 at 2:44 PM, Gupta, Ramesh <grgupta@ti.com> wrote:
> > Hi Pramod,
> >
> >
> >> -----Original Message-----
> >> From: linux-omap-owner@vger.kernel.org
> >> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of pramod gurav
> >> Sent: Wednesday, April 22, 2009 1:17 PM
> >> To: Menon, Nishanth
> >> Cc: linux-omap@vger.kernel.org
> >> Subject: Re: dma_alloc_coherent fragmentation
> >>
> >> Hi Nishant,
> >> I am facing the same problem to allocate a chunk of 4MB on omap3evm
> >> using dma_alloc_coherent. I used the test driver mentioned here
> >> 
> http://www.mail-archive.com/linux-omap@vger.kernel.org/msg11074.html
> >>
> >> I applied the patch for
> >> ioremap(24f11ec001920f1cfaeeed8e8b55725d900bbb56) suggested by
> >> Russell, but stiill seeing the error saying
> >>
> >> coherent allocation too big (requested 0x400000 mask 0xffffffff)
> >> alloc[0] - virt=0x00000000 phy=0xFFFFFFFF size=0x00400000
> >> Allocation failed idx=0
> >> insmod: cannot insert `dummy.ko': Cannot allocate memory 
> (-1): Cannot
> >> allocate memory
> >>
> >> I changed the GFP_KERNEL to GFP_TEMPORARY and also to (GFP_KERNEL |
> >> __GFP_RECLAIMABLE), but failed to insmod the module.
> >> I am working on linux-omap 2.6.28 kernel. Even I could not allocate
> >> memory size greater than 1MB.
> >> Has this been fixed? Are there any patches to fix this issue?
> >>
> >
> > What is the value for 'CONFIG_FB_OMAP_CONSISTENT_DMA_SIZE' 
> (default is 2 in omap_3430sdp_defconfig), Can you please
> > try increasing to 8?
> >
> > Regards
> > Ramesh Gupta G
> >
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

dma_alloc_coherent fragmentation

[Menon, Nishanth]

Looping in linux-arm ML:

Discussion Ref: [1](linux-omap mailing list)
While working with Linux OMAP kernel[2] we found that on allocating 4 meg chunks using dma_alloc_coherent and de-allocating with dma_free_coherent in a loop using a test driver[7], the memory is getting fragmented as shown by Ameya's observation [3] finally resulting in dump_stack due to lack of pages.

Searching through previous archives, found [4] which seem to imply that allocating chunks greater than 1 page could cause fragmentation.

I wonder why we see this even if we allocate and free the memory in sequence as in [5]. This seems to happen on 2.6.28 and 2.6.29 but not on the 2.6.27 based kernel[6].

Regards,
Nishanth Menon

Ref:
[1] http://marc.info/?t=123719779100002&r=1&w=2
[2] http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git
[3] http://marc.info/?l=linux-omap&m=123722460022336&w=2
[4] http://lkml.indiana.edu/hypermail/linux/kernel/0706.0/0370.html 
[5] http://marc.info/?l=linux-omap&m=123719772811746&w=2
[6] http://git.omapzoom.com/?p=repo/omapkernel.git;
[7] http://marc.info/?l=linux-omap&m=123719772811746&q=p3