allocating memory at boot

allocating memory at boot

[Philip Downer]

Hi folks,

I'm currently trying to port a (currently for unreleased hardware and 
under nda) driver from x86 to arm and in doing so I've run into a 
problem with allocating memory using pci_alloc_consistent. Looking into 
it I can't allocate more than 1MB whereas I need to allocate at least 
2MB of contiguous memory. Looking further into it, the conventional 
method to get around this at the moment seems to be to statically link 
the driver into the kernel and use alloc_bootmem or alloc_bootmem_low 
(mentioned in ldd3 and all over google).

The problem I've run into however is that I get a warning at boot which is:

WARNING: at mm/bootmem.c:672 alloc_arch_preferred_bootmem+0x34/0x64()

Looking at that code in bootmem.c this is a warning which I think is 
there to tell me that SLAB is available and warn against trying to 
allocate at boot. However it reportedly allocates the memory but when 
the driver tries to use this memory later, then the system hangs.

Is alloc_bootmem currently the correct way to get memory allocated at 
boot or is there a different call I should be using. Alternatively have 
I done something wrong by using alloc_bootmem as a drop-in replacement 
for pci_alloc_consistent? I've tried to grep the kernel sources for 
other drivers that use alloc_bootmem but there aren't many which appear 
in the listing and I find this strange for a technique which is referred 
to a lot.

Thanks,

Phil

allocating memory at boot

[sk.syed2]

> I'm currently trying to port a (currently for unreleased hardware and
> under nda) driver from x86 to arm and in doing so I've run into a
> problem with allocating memory using pci_alloc_consistent. Looking into
> it I can't allocate more than 1MB whereas I need to allocate at least
> 2MB of contiguous memory.
   I guess pci_alloc_consistent calls dma_alloc_coherent, so why does
it fail to allocate 2MB of memory?
What is the error do you see?

Looking further into it, the conventional
> method to get around this at the moment seems to be to statically link
> the driver into the kernel and use alloc_bootmem or alloc_bootmem_low
> (mentioned in ldd3 and all over google).
>
> The problem I've run into however is that I get a warning at boot which is:
>
> WARNING: at mm/bootmem.c:672 alloc_arch_preferred_bootmem+0x34/0x64()

You need to call alloc_bootmem at an early stage before slab is initialized.

>
> Looking at that code in bootmem.c this is a warning which I think is
> there to tell me that SLAB is available and warn against trying to
> allocate at boot. However it reportedly allocates the memory but when
> the driver tries to use this memory later, then the system hangs.

Is the driver using the memory for DMA? Are you using the memory
returned by alloc_bootmem directly?
Remember DMA works on physical addresses not virtual. You might have
to use dma_map_single/dma_unmap_single.
Also does the h/w has any limitations of the physical memory range it
can access? Like only first 64MB of RAM?


regards
-syed

allocating memory at boot

[Philip Downer]

sk.syed2 wrote:
>> I'm currently trying to port a (currently for unreleased hardware and
>> under nda) driver from x86 to arm and in doing so I've run into a
>> problem with allocating memory using pci_alloc_consistent. Looking into
>> it I can't allocate more than 1MB whereas I need to allocate at least
>> 2MB of contiguous memory.
>    I guess pci_alloc_consistent calls dma_alloc_coherent, so why does
> it fail to allocate 2MB of memory?
> What is the error do you see?

Well there is no warning or panic, pci_alloc_consistent returns NULL if 
I request 2MB. If I change my request to 1MB then it succeeds, 
everything is allocated correctly and I can use the memory without 
issues, but of course the device requires 2MB (well 6 buffers all 2MB in 
size) so it's of no use. If I attempt to request a second 1MB then 
pci_alloc_consistent also returns NULL for that second allocation.

I have looked at the code for dma_alloc_coherent etc and it seems to me 
that it's returning null because there isn't the space available to 
allocate the requested memory.

 > > The problem I've run into however is that I get a warning at boot
 > > which is:
 > >
 > > WARNING: at mm/bootmem.c:672
 > > alloc_arch_preferred_bootmem+0x34/0x64()
 >
 > You need to call alloc_bootmem at an early stage before slab is
 > initialized.

I was thinking that this might be the problem, but in order to do that 
I'd have to put the alloc_bootmem in core kernel code wouldn't I? This 
doesn't seem very appealing.

> Is the driver using the memory for DMA? Are you using the memory
> returned by alloc_bootmem directly?

Yes, it is used for DMA and I am using the memory returned by 
alloc_bootmem directly.

> Remember DMA works on physical addresses not virtual. You might have
> to use dma_map_single/dma_unmap_single.

Yes, I was aware that dma uses physical addresses, isn't that what 
alloc_bootmem returns? I'm not aware of dma_map_single etc I will look 
at that soon.

> Also does the h/w has any limitations of the physical memory range it
> can access? Like only first 64MB of RAM?

It's limited to 32bit addressing, but that should be fine for the 512mb 
of ram on the arm board which I have.

Thanks,

Phil

allocating memory at boot

[sk.syed2]

> I have looked at the code for dma_alloc_coherent etc and it seems to me that
> it's returning null because there isn't the space available to allocate the
> requested memory.
You might have to increase the consistent(or dma) memory size. In
recent kernels its defined by  CONSISTENT_DMA_SIZE
(ach/arm/include/asm/memory.h). Which kernel version are you using?
You might have to override this value to match your needs. Are there
any other drivers that allocate using dma_alloc_coherent?

> I was thinking that this might be the problem, but in order to do that I'd
> have to put the alloc_bootmem in core kernel code wouldn't I? This doesn't
> seem very appealing.
Yes. Nobody wants to do that. Best option is to use
dma_alloc_coherent. Alternatively you can use kmalloc in your driver
itself and then use dma_map_single/dma_unmap_single.

> Yes, I was aware that dma uses physical addresses, isn't that what
> alloc_bootmem returns? I'm not aware of dma_map_single etc I will look at
> that soon.
No alloc_bootmem doesnt return physical addresses. It returns virtual
address. That is the reason you need to use dma_map_single and
dma_unmap_single. These apis return physical address(dma_addr_t) for
the corresponding virtual address. Also they flush/invalidate the
cache so there are no side effects.
  But dma_alloc_coherent gives you a memory whose cache lines are
disabled, so you wouldnt have to worry about caching side effects.

-syed

allocating memory at boot

[Philip Downer]

sk.syed2 wrote:
>> I have looked at the code for dma_alloc_coherent etc and it seems to me that
>> it's returning null because there isn't the space available to allocate the
>> requested memory.
> You might have to increase the consistent(or dma) memory size. In
> recent kernels its defined by  CONSISTENT_DMA_SIZE
> (ach/arm/include/asm/memory.h). Which kernel version are you using?
> You might have to override this value to match your needs. Are there
> any other drivers that allocate using dma_alloc_coherent?

Oh, awesome! This was it, I'm using kernel version 2.6.37 and changing 
CONSISTENT_DMA_SIZE for arm has allowed me to allocate enough memory 
using dma_alloc_coherent so I can use the original code which I had 
working on x86.

Is there any reason why this isn't a kconfig option?

Thanks for the help.

Phil

allocating memory at boot

[sk.syed2]

>
> Is there any reason why this isn't a kconfig option?
The memory/RAM  related defaults are present in
arch/arm/include/asm/memory.h. But you can override them by defining
them in
/arch/arm/mach-YOURMACHINE/include/mach/memory.h. I think the
developers intended to keep it this way so machine specific defines go
in machine specific directories, and it will not bloat the kernel
config.

-syed