Reserved pages in PowerPC

Reserved pages in PowerPC

[Ankita Garg]

Hi,

I am trying to hotplug/offline sections of memory on a Power machine.
I boot the kernel with kernelcore=1G commandline parameter. I see that except
for 512MB, the rest of the memory is movable. When trying to do hot-remove, I
notice that I am unable to remove the very last section of memory, one with
the highest physical address. It is always marked as non-movable.

With some debugging I found that that section has reserved pages. On
instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
that many of the memory areas are reserved for kernel and initrd by the
memblock reserve() itself. reserve_bootmem then looks at the pages already
reserved and marks them reserved. However, for the very last section, I see
that bootmem reserves it but I am unable to find a corresponding reservation
by the memblock code.


memblock_reserve: start 0 size 3519 
reserve_bootmem 0 dbf000 nid=0
memblock_reserve: start 12096 size 15372
reserve_bootmem 2f40000 ccc000 nid=0
memblock_reserve: start 15628 size 15650
reserve_bootmem 3d0c000 16000 nid=0
...
...
memblock_reserve: start 1982455 size 1982464
reserve_bootmem 1e3ff7c00 8400 nid=0
reserve_bootmem 3d7f64000 3f000 nid=1
reserve_bootmem 3d7fa3c00 48400 nid=1
reserve_bootmem 3d7feeda8 11258 nid=1
..
 
Is it a known behavior on Power ? If yes, for what purpose is the memory
in the higher address reserved for ? I have seen that even if the system
has multiple nodes, only the very last section of the last node is not
removable.

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Reserved pages in PowerPC

[Ankita Garg]

Hi,

I am trying to hotplug/offline sections of memory on a Power machine.
I boot the kernel with kernelcore=1G commandline parameter. I see that except
for 512MB, the rest of the memory is movable. When trying to do hot-remove, I
notice that I am unable to remove the very last section of memory, one with
the highest physical address. It is always marked as non-movable.

With some debugging I found that that section has reserved pages. On
instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
that many of the memory areas are reserved for kernel and initrd by the
memblock reserve() itself. reserve_bootmem then looks at the pages already
reserved and marks them reserved. However, for the very last section, I see
that bootmem reserves it but I am unable to find a corresponding reservation
by the memblock code.


memblock_reserve: start 0 size 3519 
reserve_bootmem 0 dbf000 nid=0
memblock_reserve: start 12096 size 15372
reserve_bootmem 2f40000 ccc000 nid=0
memblock_reserve: start 15628 size 15650
reserve_bootmem 3d0c000 16000 nid=0
...
...
memblock_reserve: start 1982455 size 1982464
reserve_bootmem 1e3ff7c00 8400 nid=0
reserve_bootmem 3d7f64000 3f000 nid=1
reserve_bootmem 3d7fa3c00 48400 nid=1
reserve_bootmem 3d7feeda8 11258 nid=1
..
 
Is it a known behavior on Power ? If yes, for what purpose is the memory
in the higher address reserved for ? I have seen that even if the system
has multiple nodes, only the very last section of the last node is not
removable.

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India

Re: Reserved pages in PowerPC

[Benjamin Herrenschmidt]

On Thu, 2010-09-16 at 10:53 +0530, Ankita Garg wrote:
> 
> With some debugging I found that that section has reserved pages. On
> instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
> that many of the memory areas are reserved for kernel and initrd by the
> memblock reserve() itself. reserve_bootmem then looks at the pages already
> reserved and marks them reserved. However, for the very last section, I see
> that bootmem reserves it but I am unable to find a corresponding reservation
> by the memblock code.

It's probably RTAS (firmware runtime services). I'ts instanciated at
boot from prom_init and we do favor high addresses for it below 1G iirc.

Cheers,
Ben.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Reserved pages in PowerPC

[Benjamin Herrenschmidt]

On Thu, 2010-09-16 at 10:53 +0530, Ankita Garg wrote:
> 
> With some debugging I found that that section has reserved pages. On
> instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
> that many of the memory areas are reserved for kernel and initrd by the
> memblock reserve() itself. reserve_bootmem then looks at the pages already
> reserved and marks them reserved. However, for the very last section, I see
> that bootmem reserves it but I am unable to find a corresponding reservation
> by the memblock code.

It's probably RTAS (firmware runtime services). I'ts instanciated at
boot from prom_init and we do favor high addresses for it below 1G iirc.

Cheers,
Ben.

Re: Reserved pages in PowerPC

[Ankita Garg]

Hi Ben,

On Thu, Sep 16, 2010 at 08:04:24PM +1000, Benjamin Herrenschmidt wrote:
> On Thu, 2010-09-16 at 10:53 +0530, Ankita Garg wrote:
> > 
> > With some debugging I found that that section has reserved pages. On
> > instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
> > that many of the memory areas are reserved for kernel and initrd by the
> > memblock reserve() itself. reserve_bootmem then looks at the pages already
> > reserved and marks them reserved. However, for the very last section, I see
> > that bootmem reserves it but I am unable to find a corresponding reservation
> > by the memblock code.
> 
> It's probably RTAS (firmware runtime services). I'ts instanciated at
> boot from prom_init and we do favor high addresses for it below 1G iirc.
>

Thanks Ben for taking a look at this. So I checked the rtas messages on
the serial console and see the following:

instantiating rtas at 0x000000000f632000... done

Which does not correspond to the higher addresses that I see as reserved
(observation on a 16G machine).

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Reserved pages in PowerPC

[Ankita Garg]

Hi Ben,

On Thu, Sep 16, 2010 at 08:04:24PM +1000, Benjamin Herrenschmidt wrote:
> On Thu, 2010-09-16 at 10:53 +0530, Ankita Garg wrote:
> > 
> > With some debugging I found that that section has reserved pages. On
> > instrumenting the memblock_reserve() and reserve_bootmem() routines, I can see
> > that many of the memory areas are reserved for kernel and initrd by the
> > memblock reserve() itself. reserve_bootmem then looks at the pages already
> > reserved and marks them reserved. However, for the very last section, I see
> > that bootmem reserves it but I am unable to find a corresponding reservation
> > by the memblock code.
> 
> It's probably RTAS (firmware runtime services). I'ts instanciated at
> boot from prom_init and we do favor high addresses for it below 1G iirc.
>

Thanks Ben for taking a look at this. So I checked the rtas messages on
the serial console and see the following:

instantiating rtas at 0x000000000f632000... done

Which does not correspond to the higher addresses that I see as reserved
(observation on a 16G machine).

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India

Re: Reserved pages in PowerPC

[Benjamin Herrenschmidt]

On Thu, 2010-09-16 at 17:38 +0530, Ankita Garg wrote:
> Thanks Ben for taking a look at this. So I checked the rtas messages
> on
> the serial console and see the following:
> 
> instantiating rtas at 0x000000000f632000... done
> 
> Which does not correspond to the higher addresses that I see as
> reserved
> (observation on a 16G machine). 

Well, I'd suggest you audit prom_init.c which builds the reserve map,
and the various memblock_reserve() calls in prom.c

Cheers,
Ben.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Reserved pages in PowerPC

[Benjamin Herrenschmidt]

On Thu, 2010-09-16 at 17:38 +0530, Ankita Garg wrote:
> Thanks Ben for taking a look at this. So I checked the rtas messages
> on
> the serial console and see the following:
> 
> instantiating rtas at 0x000000000f632000... done
> 
> Which does not correspond to the higher addresses that I see as
> reserved
> (observation on a 16G machine). 

Well, I'd suggest you audit prom_init.c which builds the reserve map,
and the various memblock_reserve() calls in prom.c

Cheers,
Ben.

Re: Reserved pages in PowerPC

[Ankita Garg]

Hi Ben,

On Fri, Sep 17, 2010 at 07:52:31AM +1000, Benjamin Herrenschmidt wrote:
> On Thu, 2010-09-16 at 17:38 +0530, Ankita Garg wrote:
> > Thanks Ben for taking a look at this. So I checked the rtas messages
> > on
> > the serial console and see the following:
> > 
> > instantiating rtas at 0x000000000f632000... done
> > 
> > Which does not correspond to the higher addresses that I see as
> > reserved
> > (observation on a 16G machine). 
> 
> Well, I'd suggest you audit prom_init.c which builds the reserve map,
> and the various memblock_reserve() calls in prom.c
>

I studied and instrumented memblock_reserve() and also reserve_mem().
However, all the reserved addresses seem to correspond to lower memory.
I also observed that these reserved addresses are accessed quite rapidly
when a workload is being run.. 

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Reserved pages in PowerPC

[Ankita Garg]

Hi Ben,

On Fri, Sep 17, 2010 at 07:52:31AM +1000, Benjamin Herrenschmidt wrote:
> On Thu, 2010-09-16 at 17:38 +0530, Ankita Garg wrote:
> > Thanks Ben for taking a look at this. So I checked the rtas messages
> > on
> > the serial console and see the following:
> > 
> > instantiating rtas at 0x000000000f632000... done
> > 
> > Which does not correspond to the higher addresses that I see as
> > reserved
> > (observation on a 16G machine). 
> 
> Well, I'd suggest you audit prom_init.c which builds the reserve map,
> and the various memblock_reserve() calls in prom.c
>

I studied and instrumented memblock_reserve() and also reserve_mem().
However, all the reserved addresses seem to correspond to lower memory.
I also observed that these reserved addresses are accessed quite rapidly
when a workload is being run.. 

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India