* [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
@ 2017-05-23 15:15 Michael Bringmann
2017-05-23 15:52 ` Reza Arbab
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-05-23 15:15 UTC (permalink / raw)
To: linuxppc-dev, linux-kernel
Cc: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Reza Arbab, Thomas Gleixner, Bharata B Rao, Balbir Singh,
Michael Bringmann, Shailendra Singh, Aneesh Kumar K.V,
Sebastian Andrzej Siewior
Removing or adding memory via the PowerPC hotplug interface shows
anomalies in the association between memory and nodes. The code
was updated to initialize more possible nodes to make them available
to subsequent DLPAR hotplug-memory operations, even if they are not
needed at boot time.
Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com>
---
arch/powerpc/mm/numa.c | 44 ++++++++++++++++++++++++++++++++------------
1 file changed, 32 insertions(+), 12 deletions(-)
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 15c2dd5..3d58c1f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -870,7 +870,7 @@ void __init dump_numa_cpu_topology(void)
}
/* Initialize NODE_DATA for a node on the local memory */
-static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
+static void setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
{
u64 spanned_pages = end_pfn - start_pfn;
const size_t nd_size = roundup(sizeof(pg_data_t), SMP_CACHE_BYTES);
@@ -878,23 +878,41 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
void *nd;
int tnid;
- nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
- nd = __va(nd_pa);
+ if (!node_data[nid]) {
+ nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
+ nd = __va(nd_pa);
- /* report and initialize */
- pr_info(" NODE_DATA [mem %#010Lx-%#010Lx]\n",
- nd_pa, nd_pa + nd_size - 1);
- tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
- if (tnid != nid)
- pr_info(" NODE_DATA(%d) on node %d\n", nid, tnid);
+ node_data[nid] = nd;
+ memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
+ NODE_DATA(nid)->node_id = nid;
+
+ /* report and initialize */
+ pr_info(" NODE_DATA [mem %#010Lx-%#010Lx]\n",
+ nd_pa, nd_pa + nd_size - 1);
+ tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
+ if (tnid != nid)
+ pr_info(" NODE_DATA(%d) on node %d\n", nid, tnid);
+ } else {
+ nd_pa = (u64) node_data[nid];
+ nd = __va(nd_pa);
+ }
- node_data[nid] = nd;
- memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
- NODE_DATA(nid)->node_id = nid;
NODE_DATA(nid)->node_start_pfn = start_pfn;
NODE_DATA(nid)->node_spanned_pages = spanned_pages;
}
+static void setup_nodes(void)
+{
+ int i, l = 32 /* MAX_NUMNODES */;
+
+ for (i = 0; i < l; i++) {
+ if (!node_possible(i)) {
+ setup_node_data(i, 0, 0);
+ node_set(i, node_possible_map);
+ }
+ }
+}
+
void __init initmem_init(void)
{
int nid, cpu;
@@ -914,6 +932,8 @@ void __init initmem_init(void)
*/
nodes_and(node_possible_map, node_possible_map, node_online_map);
+ setup_nodes();
+
for_each_online_node(nid) {
unsigned long start_pfn, end_pfn;
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 15:15 [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann
@ 2017-05-23 15:52 ` Reza Arbab
2017-05-23 20:05 ` Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Reza Arbab @ 2017-05-23 15:52 UTC (permalink / raw)
To: Michael Bringmann, Balbir Singh
Cc: linuxppc-dev, linux-kernel, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, Thomas Gleixner, Bharata B Rao,
Shailendra Singh, Aneesh Kumar K.V, Sebastian Andrzej Siewior
On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>+static void setup_nodes(void)
>+{
>+ int i, l = 32 /* MAX_NUMNODES */;
>+
>+ for (i = 0; i < l; i++) {
>+ if (!node_possible(i)) {
>+ setup_node_data(i, 0, 0);
>+ node_set(i, node_possible_map);
>+ }
>+ }
>+}
This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset
node_possible_map to only node_online_map").
Balbir, you have a patchset which reverts it. Do you think that will be
getting merged?
http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
(see patch 3/3)
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 15:52 ` Reza Arbab
@ 2017-05-23 20:05 ` Michael Bringmann
2017-05-23 21:49 ` Reza Arbab
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-05-23 20:05 UTC (permalink / raw)
To: Reza Arbab, Balbir Singh
Cc: Sebastian Andrzej Siewior, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev
On 05/23/2017 10:52 AM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>> +static void setup_nodes(void)
>> +{
>> + int i, l = 32 /* MAX_NUMNODES */;
>> +
>> + for (i = 0; i < l; i++) {
>> + if (!node_possible(i)) {
>> + setup_node_data(i, 0, 0);
>> + node_set(i, node_possible_map);
>> + }
>> + }
>> +}
>
> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
They may be related, but that commit is not a replacement. The above patch ensures that
there are enough of the nodes initialized at startup to allow for memory hot-add into a
node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
recording that the node was initialized.
I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by
'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
>
> Balbir, you have a patchset which reverts it. Do you think that will be getting merged?
>
> http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
> (see patch 3/3)
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 20:05 ` Michael Bringmann
@ 2017-05-23 21:49 ` Reza Arbab
2017-05-23 22:41 ` Michael Bringmann
2017-05-23 22:44 ` Michael Bringmann
0 siblings, 2 replies; 33+ messages in thread
From: Reza Arbab @ 2017-05-23 21:49 UTC (permalink / raw)
To: Michael Bringmann
Cc: Balbir Singh, Sebastian Andrzej Siewior, linux-kernel,
Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev
On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>On 05/23/2017 10:52 AM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>> +static void setup_nodes(void)
>>> +{
>>> + int i, l = 32 /* MAX_NUMNODES */;
>>> +
>>> + for (i = 0; i < l; i++) {
>>> + if (!node_possible(i)) {
>>> + setup_node_data(i, 0, 0);
>>> + node_set(i, node_possible_map);
>>> + }
>>> + }
>>> +}
>>
>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>
>They may be related, but that commit is not a replacement. The above patch ensures that
>there are enough of the nodes initialized at startup to allow for memory hot-add into a
>node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>recording that the node was initialized.
Is it really necessary to preinitialize these empty nodes using
setup_node_data()? When you do memory hotadd into a node that was not
used at boot, the node data already gets set up by
add_memory
add_memory_resource
hotadd_new_pgdat
arch_alloc_nodedata <-- allocs the pg_data_t
...
free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
Removing setup_node_data() from that loop leaves only the call to
node_set(). If 3af229f2071f (which reduces node_possible_map) was
reverted, you wouldn't need to do that either.
>I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by
>'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
Are you sure you're looking at 3af229f2071f? It only adds one line of
code; the reduction of node_possible_map.
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 21:49 ` Reza Arbab
@ 2017-05-23 22:41 ` Michael Bringmann
2017-05-23 22:44 ` Michael Bringmann
1 sibling, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-23 22:41 UTC (permalink / raw)
To: Reza Arbab
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> + for (i = 0; i < l; i++) {
>>>> + if (!node_possible(i)) {
>>>> + setup_node_data(i, 0, 0);
>>>> + node_set(i, node_possible_map);
>>>> + }
>>>> + }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement. The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>> recording that the node was initialized.
>
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>
> add_memory
> add_memory_resource
> hotadd_new_pgdat
> arch_alloc_nodedata <-- allocs the pg_data_t
> ...
> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
I see that code now, but for some reason it did not work when I hot-added
memory.
>
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
>
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
>
The 3rd file in the patch set removes,
- nodes_and(node_possible_map, node_possible_map, node_online_map);
I need to add bits to 'node_possible_map' -- bits which may not be used
for the memory at boot, but which would be used when memory is hot-added
later. I haven't found anything outside of the boot code that adds bits
to the 'possible' mask.
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 21:49 ` Reza Arbab
2017-05-23 22:41 ` Michael Bringmann
@ 2017-05-23 22:44 ` Michael Bringmann
2017-05-24 11:19 ` Michael Ellerman
2017-05-24 14:36 ` Reza Arbab
1 sibling, 2 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-23 22:44 UTC (permalink / raw)
To: Reza Arbab
Cc: Balbir Singh, Sebastian Andrzej Siewior, linux-kernel,
Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev
On 05/23/2017 04:49 PM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>> +static void setup_nodes(void)
>>>> +{
>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>> +
>>>> + for (i = 0; i < l; i++) {
>>>> + if (!node_possible(i)) {
>>>> + setup_node_data(i, 0, 0);
>>>> + node_set(i, node_possible_map);
>>>> + }
>>>> + }
>>>> +}
>>>
>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>
>> They may be related, but that commit is not a replacement. The above patch ensures that
>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>> recording that the node was initialized.
>
> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>
> add_memory
> add_memory_resource
> hotadd_new_pgdat
> arch_alloc_nodedata <-- allocs the pg_data_t
> ...
> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>
> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
With or without 3af229f2071f, we would still need to add something, somewhere to add new
bits to the 'node_possible_map'. That is not being done.
>
>> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map'
>> which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by
>> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask.
>
> Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 22:44 ` Michael Bringmann
@ 2017-05-24 11:19 ` Michael Ellerman
2017-05-24 23:55 ` Michael Bringmann
2017-05-24 14:36 ` Reza Arbab
1 sibling, 1 reply; 33+ messages in thread
From: Michael Ellerman @ 2017-05-24 11:19 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> + for (i = 0; i < l; i++) {
>>>>> + if (!node_possible(i)) {
>>>>> + setup_node_data(i, 0, 0);
>>>>> + node_set(i, node_possible_map);
>>>>> + }
>>>>> + }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement. The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>>> recording that the node was initialized.
>>
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>
>> add_memory
>> add_memory_resource
>> hotadd_new_pgdat
>> arch_alloc_nodedata <-- allocs the pg_data_t
>> ...
>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
> With or without 3af229f2071f, we would still need to add something, somewhere to add new
> bits to the 'node_possible_map'. That is not being done.
You mustn't add bits to the possible map after boot.
That's its purpose, to tell you what nodes could ever *possibly* exist.
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-23 22:44 ` Michael Bringmann
2017-05-24 11:19 ` Michael Ellerman
@ 2017-05-24 14:36 ` Reza Arbab
2017-05-24 23:41 ` Michael Bringmann
1 sibling, 1 reply; 33+ messages in thread
From: Reza Arbab @ 2017-05-24 14:36 UTC (permalink / raw)
To: Michael Bringmann
Cc: Balbir Singh, Sebastian Andrzej Siewior, linux-kernel,
Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev
On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote:
>On 05/23/2017 04:49 PM, Reza Arbab wrote:
>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>> +static void setup_nodes(void)
>>>>> +{
>>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>>> +
>>>>> + for (i = 0; i < l; i++) {
>>>>> + if (!node_possible(i)) {
>>>>> + setup_node_data(i, 0, 0);
>>>>> + node_set(i, node_possible_map);
>>>>> + }
>>>>> + }
>>>>> +}
>>>>
>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>
>>> They may be related, but that commit is not a replacement. The above patch ensures that
>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>>> recording that the node was initialized.
>>
>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>
>> add_memory
>> add_memory_resource
>> hotadd_new_pgdat
>> arch_alloc_nodedata <-- allocs the pg_data_t
>> ...
>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>
>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>
>With or without 3af229f2071f, we would still need to add something, somewhere to add new
>bits to the 'node_possible_map'. That is not being done.
Without 3af229f2071f, those bits would already BE set in
node_possible_map. You wouldn't have to do anything.
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-24 14:36 ` Reza Arbab
@ 2017-05-24 23:41 ` Michael Bringmann
0 siblings, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-24 23:41 UTC (permalink / raw)
To: Reza Arbab
Cc: Balbir Singh, Sebastian Andrzej Siewior, linux-kernel,
Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev
I will get a log based on the latest 4.12 kernel to show what happens
one way or the other, with this patch removed.
On 05/24/2017 09:36 AM, Reza Arbab wrote:
> On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote:
>> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>>> +static void setup_nodes(void)
>>>>>> +{
>>>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>>>> +
>>>>>> + for (i = 0; i < l; i++) {
>>>>>> + if (!node_possible(i)) {
>>>>>> + setup_node_data(i, 0, 0);
>>>>>> + node_set(i, node_possible_map);
>>>>>> + }
>>>>>> + }
>>>>>> +}
>>>>>
>>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>>
>>>> They may be related, but that commit is not a replacement. The above patch ensures that
>>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>>>> recording that the node was initialized.
>>>
>>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>>
>>> add_memory
>>> add_memory_resource
>>> hotadd_new_pgdat
>>> arch_alloc_nodedata <-- allocs the pg_data_t
>>> ...
>>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>>
>>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>>
>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>> bits to the 'node_possible_map'. That is not being done.
>
> Without 3af229f2071f, those bits would already BE set in node_possible_map. You wouldn't have to do anything.
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-24 11:19 ` Michael Ellerman
@ 2017-05-24 23:55 ` Michael Bringmann
2017-05-25 6:19 ` Michael Ellerman
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-05-24 23:55 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On 05/24/2017 06:19 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>
>> On 05/23/2017 04:49 PM, Reza Arbab wrote:
>>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote:
>>>> On 05/23/2017 10:52 AM, Reza Arbab wrote:
>>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote:
>>>>>> +static void setup_nodes(void)
>>>>>> +{
>>>>>> + int i, l = 32 /* MAX_NUMNODES */;
>>>>>> +
>>>>>> + for (i = 0; i < l; i++) {
>>>>>> + if (!node_possible(i)) {
>>>>>> + setup_node_data(i, 0, 0);
>>>>>> + node_set(i, node_possible_map);
>>>>>> + }
>>>>>> + }
>>>>>> +}
>>>>>
>>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map").
>>>>
>>>> They may be related, but that commit is not a replacement. The above patch ensures that
>>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a
>>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and
>>>> recording that the node was initialized.
>>>
>>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by
>>>
>>> add_memory
>>> add_memory_resource
>>> hotadd_new_pgdat
>>> arch_alloc_nodedata <-- allocs the pg_data_t
>>> ...
>>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc.
>>>
>>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either.
>>
>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>> bits to the 'node_possible_map'. That is not being done.
>
> You mustn't add bits to the possible map after boot.
>
> That's its purpose, to tell you what nodes could ever *possibly* exist.
The problem that I have been encountering is that the 'possible map' did *not*
show all of the possible nodes. Rather, it showed only the nodes that were
assigned memory at boot-up. If more memory were hot-added to the kernel, it
could be assigned into one of the nodes that were skipped at boot. However,
nothing was updating the 'node_possible_map' correctly in the kernel memory
code.
Reza pointed out a code change in commit 3af229f2071f that has not made it into
the 4.12 checkout i.e. removing the instruction that reduces the node_possible_map.
This may well be a suitable replacement for the code that I have here, and I
will test it here next.
>
> cheers
>
>
Later.
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-24 23:55 ` Michael Bringmann
@ 2017-05-25 6:19 ` Michael Ellerman
2017-05-25 13:40 ` Michael Bringmann
2017-05-25 15:10 ` Reza Arbab
0 siblings, 2 replies; 33+ messages in thread
From: Michael Ellerman @ 2017-05-25 6:19 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 05/24/2017 06:19 AM, Michael Ellerman wrote:
>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>
>>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>>> bits to the 'node_possible_map'. That is not being done.
>>
>> You mustn't add bits to the possible map after boot.
>>
>> That's its purpose, to tell you what nodes could ever *possibly* exist.
>
> The problem that I have been encountering is that the 'possible map' did *not*
> show all of the possible nodes.
OK so how did that happen?
The commit message for 3af229f2071f says:
In practice, we never see a system with 256 NUMA nodes, and in fact, we
do not support node hotplug on power in the first place, so the nodes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
that are online when we come up are the nodes that will be present for
the lifetime of this kernel.
Is that no longer true?
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-25 6:19 ` Michael Ellerman
@ 2017-05-25 13:40 ` Michael Bringmann
2017-05-25 15:10 ` Reza Arbab
1 sibling, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-25 13:40 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On 05/25/2017 01:19 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>
>> On 05/24/2017 06:19 AM, Michael Ellerman wrote:
>>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>>
>>>> With or without 3af229f2071f, we would still need to add something, somewhere to add new
>>>> bits to the 'node_possible_map'. That is not being done.
>>>
>>> You mustn't add bits to the possible map after boot.
>>>
>>> That's its purpose, to tell you what nodes could ever *possibly* exist.
>>
>> The problem that I have been encountering is that the 'possible map' did *not*
>> show all of the possible nodes.
>
> OK so how did that happen?
>
> The commit message for 3af229f2071f says:
>
> In practice, we never see a system with 256 NUMA nodes, and in fact, we
> do not support node hotplug on power in the first place, so the nodes
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> that are online when we come up are the nodes that will be present for
> the lifetime of this kernel.
>
> Is that no longer true?
Take a look at the last part of commit 3af229f2071f for file numa.c. It undoes
a piece of code that restricts the 'node possible map', created earlier, to the
set of online nodes. That piece of code has not made it into the mainline, at
least not into 4.12. I am testing to verify whether it is sufficient for my
configuration now.
>
> cheers
>
Regards.
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-25 6:19 ` Michael Ellerman
2017-05-25 13:40 ` Michael Bringmann
@ 2017-05-25 15:10 ` Reza Arbab
2017-05-25 15:26 ` Michael Bringmann
` (2 more replies)
1 sibling, 3 replies; 33+ messages in thread
From: Reza Arbab @ 2017-05-25 15:10 UTC (permalink / raw)
To: Michael Ellerman, Balbir Singh
Cc: Michael Bringmann, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>The commit message for 3af229f2071f says:
>
> In practice, we never see a system with 256 NUMA nodes, and in fact, we
> do not support node hotplug on power in the first place, so the nodes
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> that are online when we come up are the nodes that will be present for
> the lifetime of this kernel.
>
>Is that no longer true?
I don't know what the reasoning behind that statement was at the time,
but as far as I can tell, the only thing missing for node hotplug now is
Balbir's patchset [1]. He fixes the resource issue which motivated
3af229f2071f and reverts it.
With that set, I can instantiate a new numa node just by doing
add_memory(nid, ...) where nid doesn't currently exist.
[1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-25 15:10 ` Reza Arbab
@ 2017-05-25 15:26 ` Michael Bringmann
2017-05-26 3:46 ` Balbir Singh
2017-05-26 3:46 ` Michael Ellerman
2 siblings, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-25 15:26 UTC (permalink / raw)
To: Reza Arbab, Michael Ellerman, Balbir Singh
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On 05/25/2017 10:10 AM, Reza Arbab wrote:
> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>> The commit message for 3af229f2071f says:
>>
>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>> do not support node hotplug on power in the first place, so the nodes
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> that are online when we come up are the nodes that will be present for
>> the lifetime of this kernel.
>>
>> Is that no longer true?
>
> I don't know what the reasoning behind that statement was at the time, but as far as I can tell, the only thing missing for node hotplug now is Balbir's patchset [1]. He fixes the resource issue which motivated 3af229f2071f and reverts it.
>
> With that set, I can instantiate a new numa node just by doing add_memory(nid, ...) where nid doesn't currently exist.
>
> [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
>
Yes, the change to 'numa.c' looks to be sufficient for my needs as well.
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-25 15:10 ` Reza Arbab
2017-05-25 15:26 ` Michael Bringmann
@ 2017-05-26 3:46 ` Balbir Singh
2017-05-26 3:46 ` Michael Ellerman
2 siblings, 0 replies; 33+ messages in thread
From: Balbir Singh @ 2017-05-26 3:46 UTC (permalink / raw)
To: Reza Arbab
Cc: Michael Ellerman, Michael Bringmann, linux-kernel,
Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On Thu, 25 May 2017 10:10:11 -0500
Reza Arbab <arbab@linux.vnet.ibm.com> wrote:
> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
> >The commit message for 3af229f2071f says:
> >
> > In practice, we never see a system with 256 NUMA nodes, and in fact, we
> > do not support node hotplug on power in the first place, so the nodes
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > that are online when we come up are the nodes that will be present for
> > the lifetime of this kernel.
> >
> >Is that no longer true?
>
> I don't know what the reasoning behind that statement was at the time,
> but as far as I can tell, the only thing missing for node hotplug now is
> Balbir's patchset [1]. He fixes the resource issue which motivated
> 3af229f2071f and reverts it.
>
> With that set, I can instantiate a new numa node just by doing
> add_memory(nid, ...) where nid doesn't currently exist.
>
> [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
>
I guess I should try and revive that patchset. One of the suggestions of
then was to limit maximum possible nodes in firmware, but I'm double checking
to see if we can do that in a well defined manner.
Balbir Singh
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-25 15:10 ` Reza Arbab
2017-05-25 15:26 ` Michael Bringmann
2017-05-26 3:46 ` Balbir Singh
@ 2017-05-26 3:46 ` Michael Ellerman
2017-05-26 12:31 ` Michael Bringmann
2017-05-26 14:31 ` Reza Arbab
2 siblings, 2 replies; 33+ messages in thread
From: Michael Ellerman @ 2017-05-26 3:46 UTC (permalink / raw)
To: Reza Arbab, Balbir Singh
Cc: Michael Bringmann, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
Reza Arbab <arbab@linux.vnet.ibm.com> writes:
> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>The commit message for 3af229f2071f says:
>>
>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>> do not support node hotplug on power in the first place, so the nodes
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> that are online when we come up are the nodes that will be present for
>> the lifetime of this kernel.
>>
>>Is that no longer true?
>
> I don't know what the reasoning behind that statement was at the time,
> but as far as I can tell, the only thing missing for node hotplug now is
> Balbir's patchset [1]. He fixes the resource issue which motivated
> 3af229f2071f and reverts it.
>
> With that set, I can instantiate a new numa node just by doing
> add_memory(nid, ...) where nid doesn't currently exist.
But does that actually happen on any real system?
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-26 3:46 ` Michael Ellerman
@ 2017-05-26 12:31 ` Michael Bringmann
2017-05-26 14:31 ` Reza Arbab
1 sibling, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-05-26 12:31 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab, Balbir Singh
Cc: linux-kernel, Paul Mackerras, Aneesh Kumar K.V, Bharata B Rao,
Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
I am running into this problem on PowerPC systems where Balbir's patch set
was targeted. So, yes, I do need to be able to add/enable a new numa node
during system execution in cases where more resources (memory, virtual
processors) are added to the system dynamically.
On 05/25/2017 10:46 PM, Michael Ellerman wrote:
> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>
>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>> The commit message for 3af229f2071f says:
>>>
>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>> do not support node hotplug on power in the first place, so the nodes
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> that are online when we come up are the nodes that will be present for
>>> the lifetime of this kernel.
>>>
>>> Is that no longer true?
>>
>> I don't know what the reasoning behind that statement was at the time,
>> but as far as I can tell, the only thing missing for node hotplug now is
>> Balbir's patchset [1]. He fixes the resource issue which motivated
>> 3af229f2071f and reverts it.
>>
>> With that set, I can instantiate a new numa node just by doing
>> add_memory(nid, ...) where nid doesn't currently exist.
>
> But does that actually happen on any real system?
>
> cheers
>
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-26 3:46 ` Michael Ellerman
2017-05-26 12:31 ` Michael Bringmann
@ 2017-05-26 14:31 ` Reza Arbab
2017-05-29 5:32 ` Michael Ellerman
1 sibling, 1 reply; 33+ messages in thread
From: Reza Arbab @ 2017-05-26 14:31 UTC (permalink / raw)
To: Michael Ellerman
Cc: Balbir Singh, Michael Bringmann, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>
>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>The commit message for 3af229f2071f says:
>>>
>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>> do not support node hotplug on power in the first place, so the nodes
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> that are online when we come up are the nodes that will be present for
>>> the lifetime of this kernel.
>>>
>>>Is that no longer true?
>>
>> I don't know what the reasoning behind that statement was at the time,
>> but as far as I can tell, the only thing missing for node hotplug now is
>> Balbir's patchset [1]. He fixes the resource issue which motivated
>> 3af229f2071f and reverts it.
>>
>> With that set, I can instantiate a new numa node just by doing
>> add_memory(nid, ...) where nid doesn't currently exist.
>
>But does that actually happen on any real system?
I don't know if anything currently tries to do this. My interest in
having this working is so that in the future, our coherent gpu memory
could be added as a distinct node by the device driver.
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-26 14:31 ` Reza Arbab
@ 2017-05-29 5:32 ` Michael Ellerman
2017-05-31 14:05 ` Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Michael Ellerman @ 2017-05-29 5:32 UTC (permalink / raw)
To: Reza Arbab
Cc: Balbir Singh, Michael Bringmann, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
Reza Arbab <arbab@linux.vnet.ibm.com> writes:
> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>
>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>The commit message for 3af229f2071f says:
>>>>
>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>> do not support node hotplug on power in the first place, so the nodes
>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> that are online when we come up are the nodes that will be present for
>>>> the lifetime of this kernel.
>>>>
>>>>Is that no longer true?
>>>
>>> I don't know what the reasoning behind that statement was at the time,
>>> but as far as I can tell, the only thing missing for node hotplug now is
>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>> 3af229f2071f and reverts it.
>>>
>>> With that set, I can instantiate a new numa node just by doing
>>> add_memory(nid, ...) where nid doesn't currently exist.
>>
>>But does that actually happen on any real system?
>
> I don't know if anything currently tries to do this. My interest in
> having this working is so that in the future, our coherent gpu memory
> could be added as a distinct node by the device driver.
Sure. If/when that happens, we would hopefully still have some way to
limit the size of the possible map.
That would ideally be a firmware property that tells us the maximum
number of GPUs that might be hot-added, or we punt and cap it at some
"sane" maximum number.
But until that happens it's silly to say we can have up to 256 nodes
when in practice most of our systems have 8 or less.
So I'm still waiting for an explanation from Michael B on how he's
seeing this bug in practice.
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-29 5:32 ` Michael Ellerman
@ 2017-05-31 14:05 ` Michael Bringmann
2017-06-01 9:36 ` Michael Ellerman
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-05-31 14:05 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
On 05/29/2017 12:32 AM, Michael Ellerman wrote:
> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>
>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>
>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>> The commit message for 3af229f2071f says:
>>>>>
>>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>> do not support node hotplug on power in the first place, so the nodes
>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>> that are online when we come up are the nodes that will be present for
>>>>> the lifetime of this kernel.
>>>>>
>>>>> Is that no longer true?
>>>>
>>>> I don't know what the reasoning behind that statement was at the time,
>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>> 3af229f2071f and reverts it.
>>>>
>>>> With that set, I can instantiate a new numa node just by doing
>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>
>>> But does that actually happen on any real system?
>>
>> I don't know if anything currently tries to do this. My interest in
>> having this working is so that in the future, our coherent gpu memory
>> could be added as a distinct node by the device driver.
>
> Sure. If/when that happens, we would hopefully still have some way to
> limit the size of the possible map.
>
> That would ideally be a firmware property that tells us the maximum
> number of GPUs that might be hot-added, or we punt and cap it at some
> "sane" maximum number.
>
> But until that happens it's silly to say we can have up to 256 nodes
> when in practice most of our systems have 8 or less.
>
> So I'm still waiting for an explanation from Michael B on how he's
> seeing this bug in practice.
I already answered this in an earlier message. I will give an example.
* Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
and 10G of memory in a shared processor configuration.
* At boot time, 4 nodes are put into the possible map by the PowerPC boot
code.
* Subsequently, the NUMA code executes and puts the 10G memory into nodes
4 & 5. No memory goes into Node 0. So we now have 2 nodes in the
node_online_map.
* The VP and its threads get assigned to Node 4.
* Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
node_and(node_possible_map, node_possible_map, node_online_map);
the content of the node_possible_map is reduced to nodes 4-5.
* Later on we hot-add 90G of memory to the system. It tries to put the
memory into nodes 0, 4-5, 8 based on the memory association map. We
should see memory put into all 4 nodes. However, since we have reduced
the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
into 2 of the configured nodes.
# We want to be able to put memory into all 4 nodes via hot-add operations,
not only the nodes that 'survive' boot time initialization. We could
make a number of changes to ensure that all of the nodes in the initial
configuration provided by the pHyp can be used, but this one appears to
be the simplest, only using resources requested by the pHyp at boot --
even if those resource are not used immediately.
>
> cheers
>
Regards,
Michael
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-05-31 14:05 ` Michael Bringmann
@ 2017-06-01 9:36 ` Michael Ellerman
2017-06-01 21:33 ` Reza Arbab
2017-06-02 5:24 ` Michael Bringmann
0 siblings, 2 replies; 33+ messages in thread
From: Michael Ellerman @ 2017-06-01 9:36 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 05/29/2017 12:32 AM, Michael Ellerman wrote:
>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>
>>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>>
>>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>>> The commit message for 3af229f2071f says:
>>>>>>
>>>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>>> do not support node hotplug on power in the first place, so the nodes
>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>> that are online when we come up are the nodes that will be present for
>>>>>> the lifetime of this kernel.
>>>>>>
>>>>>> Is that no longer true?
>>>>>
>>>>> I don't know what the reasoning behind that statement was at the time,
>>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>>> 3af229f2071f and reverts it.
>>>>>
>>>>> With that set, I can instantiate a new numa node just by doing
>>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>>
>>>> But does that actually happen on any real system?
>>>
>>> I don't know if anything currently tries to do this. My interest in
>>> having this working is so that in the future, our coherent gpu memory
>>> could be added as a distinct node by the device driver.
>>
>> Sure. If/when that happens, we would hopefully still have some way to
>> limit the size of the possible map.
>>
>> That would ideally be a firmware property that tells us the maximum
>> number of GPUs that might be hot-added, or we punt and cap it at some
>> "sane" maximum number.
>>
>> But until that happens it's silly to say we can have up to 256 nodes
>> when in practice most of our systems have 8 or less.
>>
>> So I'm still waiting for an explanation from Michael B on how he's
>> seeing this bug in practice.
>
> I already answered this in an earlier message.
Which one? I must have missed it.
> I will give an example.
>
> * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
> and 10G of memory in a shared processor configuration.
> * At boot time, 4 nodes are put into the possible map by the PowerPC boot
> code.
I'm pretty sure we never add nodes to the possible map, it starts out
with MAX_NUMNODES possible and that's it.
Do you actually see mention of nodes 0 and 8 in the dmesg?
What does it say?
> * Subsequently, the NUMA code executes and puts the 10G memory into nodes
> 4 & 5. No memory goes into Node 0. So we now have 2 nodes in the
> node_online_map.
> * The VP and its threads get assigned to Node 4.
> * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
> node_and(node_possible_map, node_possible_map, node_online_map);
> the content of the node_possible_map is reduced to nodes 4-5.
> * Later on we hot-add 90G of memory to the system. It tries to put the
> memory into nodes 0, 4-5, 8 based on the memory association map. We
> should see memory put into all 4 nodes. However, since we have reduced
> the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
> into 2 of the configured nodes.
Right. So it's not that you're hot adding memory into a previously
unseen node as you implied in earlier mails.
> # We want to be able to put memory into all 4 nodes via hot-add operations,
> not only the nodes that 'survive' boot time initialization. We could
> make a number of changes to ensure that all of the nodes in the initial
> configuration provided by the pHyp can be used, but this one appears to
> be the simplest, only using resources requested by the pHyp at boot --
> even if those resource are not used immediately.
I don't think that's what the patch does. It just marks 32 (!?) nodes as
online. Or if you're talking about reverting 3af229f2071f that leaves
you with 256 possible nodes. Both of which are wasteful.
The right fix is to make sure any nodes which are present at boot remain
in the possible map, even if they don't have memory/CPUs assigned at
boot.
What does your device tree look like? Can you send us the output of:
$ lsprop /proc/device-tree
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-01 9:36 ` Michael Ellerman
@ 2017-06-01 21:33 ` Reza Arbab
2017-06-07 8:06 ` Balbir Singh
2017-06-07 12:07 ` Michael Ellerman
2017-06-02 5:24 ` Michael Bringmann
1 sibling, 2 replies; 33+ messages in thread
From: Reza Arbab @ 2017-06-01 21:33 UTC (permalink / raw)
To: Michael Ellerman
Cc: Michael Bringmann, Balbir Singh, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
>I don't think that's what the patch does. It just marks 32 (!?) nodes
>as online. Or if you're talking about reverting 3af229f2071f that
>leaves you with 256 possible nodes. Both of which are wasteful.
To be clear, with Balbir's set the latter is no longer wasteful.
>The right fix is to make sure any nodes which are present at boot
>remain in the possible map, even if they don't have memory/CPUs
>assigned at boot.
I'm still hoping 3af229f2071f could indeed be reverted some day, but
until then the following would follow your suggestion for our GPU nodes.
What do you think?
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
void __init initmem_init(void)
{
int nid, cpu;
+ struct device_node *dn;
max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
max_pfn = max_low_pfn;
@@ -911,6 +912,18 @@ void __init initmem_init(void)
*/
nodes_and(node_possible_map, node_possible_map, node_online_map);
+ /*
+ * Consider an ibm,coherent-device-memory node possible. Even though
+ * it is not online at boot, it may be hotplugged later.
+ */
+ for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
+ nid = of_node_to_nid_single(dn);
+ if (nid < 0)
+ continue;
+
+ node_set(nid, node_possible_map);
+ }
+
for_each_online_node(nid) {
unsigned long start_pfn, end_pfn;
--
Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-01 9:36 ` Michael Ellerman
2017-06-01 21:33 ` Reza Arbab
@ 2017-06-02 5:24 ` Michael Bringmann
2017-06-06 9:48 ` Michael Ellerman
1 sibling, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-06-02 5:24 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
[-- Attachment #1: Type: text/plain, Size: 5747 bytes --]
On 06/01/2017 04:36 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>
>> On 05/29/2017 12:32 AM, Michael Ellerman wrote:
>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>
>>>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote:
>>>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes:
>>>>>
>>>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote:
>>>>>>> The commit message for 3af229f2071f says:
>>>>>>>
>>>>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we
>>>>>>> do not support node hotplug on power in the first place, so the nodes
>>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>>> that are online when we come up are the nodes that will be present for
>>>>>>> the lifetime of this kernel.
>>>>>>>
>>>>>>> Is that no longer true?
>>>>>>
>>>>>> I don't know what the reasoning behind that statement was at the time,
>>>>>> but as far as I can tell, the only thing missing for node hotplug now is
>>>>>> Balbir's patchset [1]. He fixes the resource issue which motivated
>>>>>> 3af229f2071f and reverts it.
>>>>>>
>>>>>> With that set, I can instantiate a new numa node just by doing
>>>>>> add_memory(nid, ...) where nid doesn't currently exist.
>>>>>
>>>>> But does that actually happen on any real system?
>>>>
>>>> I don't know if anything currently tries to do this. My interest in
>>>> having this working is so that in the future, our coherent gpu memory
>>>> could be added as a distinct node by the device driver.
>>>
>>> Sure. If/when that happens, we would hopefully still have some way to
>>> limit the size of the possible map.
>>>
>>> That would ideally be a firmware property that tells us the maximum
>>> number of GPUs that might be hot-added, or we punt and cap it at some
>>> "sane" maximum number.
>>>
>>> But until that happens it's silly to say we can have up to 256 nodes
>>> when in practice most of our systems have 8 or less.
>>>
>>> So I'm still waiting for an explanation from Michael B on how he's
>>> seeing this bug in practice.
>>
>> I already answered this in an earlier message.
>
> Which one? I must have missed it.
>
>> I will give an example.
>>
>> * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP
>> and 10G of memory in a shared processor configuration.
>> * At boot time, 4 nodes are put into the possible map by the PowerPC boot
>> code.
>
> I'm pretty sure we never add nodes to the possible map, it starts out
> with MAX_NUMNODES possible and that's it.
Let me reword that. It enables the nodes in the possible map.
>
> Do you actually see mention of nodes 0 and 8 in the dmesg?
When the 'numa.c' code is built with debug messages, and the system was
given that configuration by pHyp, yes, I did.
>
> What does it say?
The debug message for each core thread would be something like,
removing cpu 64 from node 0
adding cpu 64 to node 8
repeated for all 8 threads of the CPU, and usually with the messages
for all of the CPUs coming out intermixed on the console/dmesg log.
>
>> * Subsequently, the NUMA code executes and puts the 10G memory into nodes
>> 4 & 5. No memory goes into Node 0. So we now have 2 nodes in the
>> node_online_map.
>> * The VP and its threads get assigned to Node 4.
>> * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction,
>> node_and(node_possible_map, node_possible_map, node_online_map);
>> the content of the node_possible_map is reduced to nodes 4-5.
>> * Later on we hot-add 90G of memory to the system. It tries to put the
>> memory into nodes 0, 4-5, 8 based on the memory association map. We
>> should see memory put into all 4 nodes. However, since we have reduced
>> the 'node_possible_map' to only nodes 4 & 5, we can now only put memory
>> into 2 of the configured nodes.
>
> Right. So it's not that you're hot adding memory into a previously
> unseen node as you implied in earlier mails.
In the sense that the nodes were defined in the device tree, that is correct.
In the sense that those nodes are currently deleted from node_possible_map in
'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
node_online_map);', the nodes are no longer available to place memory or CPU.
>> # We want to be able to put memory into all 4 nodes via hot-add operations,
>> not only the nodes that 'survive' boot time initialization. We could
>> make a number of changes to ensure that all of the nodes in the initial
>> configuration provided by the pHyp can be used, but this one appears to
>> be the simplest, only using resources requested by the pHyp at boot --
>> even if those resource are not used immediately.
>
> I don't think that's what the patch does. It just marks 32 (!?) nodes as
> online. Or if you're talking about reverting 3af229f2071f that leaves
> you with 256 possible nodes. Both of which are wasteful> > The right fix is to make sure any nodes which are present at boot remain
> in the possible map, even if they don't have memory/CPUs assigned at
> boot.
Okay, I can try to insert code that extracts all of the nodes from the
ibm,associativity-lookup-arrays property and merge them with the nodes
put into the online map from the CPUs that were found previously during
boot of the powerpc code.
> What does your device tree look like? Can you send us the output of:
>
> $ lsprop /proc/device-tree
See attachment 'device-tree.log'. Note though that this boot of my test
system only has 2 nodes, 0 and 2.
>
> cheers
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: device-tree.log --]
[-- Type: text/x-log; name="device-tree.log", Size: 52468 bytes --]
ibm,partition-name
"ltcalpine2-lp20"
ibm,aix-diagnostics
ibm,drc-names 0000219e 4c4d4233 004c4d42 34004c4d
4235004c 4d423600 4c4d4237 004c4d42
38004c4d 4239004c 4d423130 004c4d42
3131004c 4d423132 004c4d42 3133004c
4d423134 004c4d42 3135004c 4d423136
004c4d42 3137004c 4d423138 004c4d42
3139004c 4d423230 004c4d42 3231004c
4d423232 004c4d42 3233004c 4d423234
[73812 bytes total]
hmc-managed? ffffffff ffffffff
ibm,drc-types 0000219e 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
4d454d00 4d454d00 4d454d00 4d454d00
[34428 bytes total]
compatible "IBM,8408-E8E"
ibm,extended-clock-frequency
00000000 00000000
ibm,max-boot-devices
00000005
ibm,drc-indexes 0000219e 80000002 80000003 80000004
80000005 80000006 80000007 80000008
80000009 8000000a 8000000b 8000000c
8000000d 8000000e 8000000f 80000010
80000011 80000012 80000013 80000014
80000015 80000016 80000017 80000018
80000019 8000001a 8000001b 8000001c
8000001d 8000001e 8000001f 80000020
[34428 bytes total]
ibm,max-vios-function-level
00000000
device_type "chrp"
ibm,fw-bytes-per-boot-device
00000100 (256)
ibm,fw-net-compatibility
"1"
ibm,service-indicator-mode
00000001
ibm,model-class "E5"
ibm,partition-performance-parameters-level
00000001
model "IBM,8408-E8E"
ibm,partition-no 00000014 (20)
ibm,migratable-partition
#address-cells 00000002
ibm,fru-9006-deactivate
00000001
#size-cells 00000002
ibm,drc-power-domains
0000219e ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
[34428 bytes total]
ibm,fw-net-version
"bPF160504 00:04:36 /gsa/ausgsa-p1/01/pfw/pfw860/PF160504/obj/rios_aix_4/of"
ibm,partition-uuid
"205f9615-9df1-4423-afb0-60d21df83633"
ibm,plat-res-int-priorities
0000000c 000000e4
clock-frequency 00000000
ibm,converged-loc-codes
ibm,phandle ffffffff (-1)
ibm,fault-behavior
00000001
ibm,platform-hardware-notification
00000001 504f5745 52385f30 30344230 32303100
ibm,eeh-default 00000001
linux,phandle 00c91548 (13178184)
ibm,ignore-hp-po-fails-for-dlpar
ibm,enable-ci64-capable
system-id "IBM,0210A7ACV"
ibm,extended-address
name ""
ibm,pci-full-cfg 00000001
ibm,lpar-capable
ibm,linux-le-capable
/proc/device-tree/ibm,serial:
compatible "hvterm1"
device_type "serial"
linux,phandle 00c9c140 (13222208)
used-by-rtas
name "ibm,serial"
/proc/device-tree/vdevice:
interrupt-ranges 000a0000 000001f3
ibm,drc-names 000001f4 55383430 382e4538 452e3130
41374143 562d5632 302d4330 00553834
30382e45 38452e31 30413741 43562d56
32302d43 31005538 3430382e 4538452e
31304137 4143562d 5632302d 43320055
38343038 2e453845 2e313041 37414356
2d563230 2d433300 55383430 382e4538
452e3130 41374143 562d5632 302d4334
[13394 bytes total]
ibm,drc-types 000001f4 534c4f54 00534c4f 5400534c
4f540053 4c4f5400 534c4f54 00534c4f
5400534c 4f540053 4c4f5400 534c4f54
00534c4f 5400534c 4f540053 4c4f5400
534c4f54 00534c4f 5400534c 4f540053
4c4f5400 534c4f54 00534c4f 5400534c
4f540053 4c4f5400 534c4f54 00534c4f
5400534c 4f540053 4c4f5400 534c4f54
[2504 bytes total]
compatible "IBM,vdevice"
ibm,max-virtual-dma-size
00050000 (327680)
ibm,drc-indexes 000001f4 30000000 30000001 30000002
30000003 30000004 30000005 30000006
30000007 30000008 30000009 3000000a
3000000b 3000000c 3000000d 3000000e
3000000f 30000010 30000011 30000012
30000013 30000014 30000015 30000016
30000017 30000018 30000019 3000001a
3000001b 3000001c 3000001d 3000001e
[2004 bytes total]
device_type "vdevice"
#interrupt-cells 00000002
#address-cells 00000001
#size-cells 00000000
ibm,drc-power-domains
000001f4 ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
[2004 bytes total]
ibm,reserved-virtual-addresses
00000fff ffd1fff0 00010000 00000fff
ffd2fff0 00010000 00000fff ffd3fff0
00010000 00000fff ffd4fff0 00010000
00000fff ffd5fff0 00010000 00000fff
ffd6fff0 00010000 00000fff ffd7fff0
00010000 00000fff ffd8fff0 00010000
00000fff ffd9fff0 00010000 00000fff
ffdafff0 00010000 00000fff ffdbfff0
[540 bytes total]
ibm,phandle fffffffc (-4)
linux,phandle 00e0f338 (14742328)
name "vdevice"
interrupt-controller
/proc/device-tree/vdevice/rtc@4001:
device_type "rtc"
reg 00004001 (16385)
linux,phandle 00e3f240 (14938688)
ibm,fw-device-code
da190000
used-by-rtas
name "rtc"
/proc/device-tree/vdevice/IBM,sp@4000:
device_type "IBM,sp"
reg 00004000 (16384)
linux,phandle 00e3e738 (14935864)
used-by-rtas
name "IBM,sp"
/proc/device-tree/vdevice/gscsi@4004:
device_type "scsi-2"
#address-cells 00000001
#size-cells 00000000
reg 00004004 (16388)
linux,phandle 00e3fe40 (14941760)
used-by-rtas
name "gscsi"
/proc/device-tree/vdevice/gscsi@4004/disk:
device_type "block"
ibm,write-supported
linux,phandle 00e40738 (14944056)
name "disk"
/proc/device-tree/vdevice/v-scsi@30000078:
ibm,#vscsi-address-bits
00000040 (64)
ibm,#dma-size-cells
00000002
compatible "IBM,v-scsi"
ibm,fw-adapter-name
"Virtual SCSI"
device_type "vscsi"
ibm,my-drc-index 30000078 (805306488)
ibm,#dma-address-cells
00000002
interrupts 000a0078 00000000
ibm,write-blocks-supported
ibm,my-dma-window
10000078 00000000 00000000 00000000 20000000
ibm,loc-code "U8408.E8E.10A7ACV-V20-C120-T1"
reg 30000078 (805306488)
ibm,phandle 30000078 (805306488)
linux,phandle 00e25560 (14832992)
name "v-scsi"
/proc/device-tree/vdevice/v-scsi@30000078/disk:
device_type "block"
ibm,write-supported
ibm,16byte-cdb-supported
linux,phandle 00e2d478 (14865528)
name "disk"
/proc/device-tree/vdevice/v-scsi@30000078/tape:
device_type "byte"
linux,phandle 00e2fd90 (14876048)
name "tape"
/proc/device-tree/vdevice/nvram@4002:
device_type "nvram"
#bytes 00003c00 (15360)
reg 00004002 (16386)
linux,phandle 00e3fb50 (14941008)
used-by-rtas
name "nvram"
/proc/device-tree/vdevice/l-lan@30000002:
ibm,#dma-size-cells
00000002
compatible "IBM,l-lan"
mac-address ba 68 98 2a 1b 02 .h.*..
ibm,illan-options
local-mac-address
ba 68 98 2a 1b 02 .h.*..
ibm,fw-adapter-name
"Interpartition Logical LAN"
device_type "network"
max-frame-size 000005dc (1500)
ibm,my-drc-index 30000002 (805306370)
supported-network-types
"ethernet,auto,none,auto"
ibm,mac-address-filters
000000ff (255)
ibm,#dma-address-cells
00000002
interrupts 000a0002 00000000
chosen-network-type
"ethernet,auto,none,auto"
ibm,vserver
address-bits 00000030 (48)
ibm,my-dma-window
10000002 00000000 00000000 00000000 10000000
ibm,loc-code "U8408.E8E.10A7ACV-V20-C2-T1"
reg 30000002 (805306370)
ibm,phandle 30000002 (805306370)
linux,phandle 00e1d2b0 (14799536)
name "l-lan"
/proc/device-tree/vdevice/vty@30000000:
compatible "hvterm1"
device_type "serial"
ibm,my-drc-index "0"
""
""
interrupts 000a0000 00000000
ibm,loc-code "U8408.E8E.10A7ACV-V20-C0"
reg "0"
""
""
ibm,phandle "0"
""
""
linux,phandle 00e1b528 (14791976)
name "vty"
/proc/device-tree/memory@0:
device_type "memory"
ibm,associativity
00000004 00000000 00000001 00000002 00000002
available 00000000 001fffff 00000000 00000001
00000000 0021ffff 00000000 00000001
00000000 00bfffff 00000000 00000001
00000000 0a6bdbf3 00000000 0000240d
00000000 0cd02878 00000000 000fd788
00000000 0ef84b1a 00000000 0000b4e6
00000000 0f090000 00000000 0f9c0000
#address-cells 00000001
#size-cells 00000000
reg 00000000 00000000 00000000 20000000
ibm,phandle fffffffa (-6)
linux,phandle 00cae9d0 (13298128)
name "memory"
/proc/device-tree/event-sources:
interrupt-ranges 00090000 00000008
#interrupt-cells 00000002
linux,phandle 00e026e8 (14690024)
name "event-sources"
ibm,fw-lpevent-xirr
00090000 (589824)
interrupt-controller
/proc/device-tree/event-sources/epow-events:
interrupts 00090001 00000000
linux,phandle 00e048a0 (14698656)
name "epow-events"
/proc/device-tree/ibm,dynamic-reconfiguration-memory:
ibm,lmb-size 00000000 10000000
ibm,memory-flags-mask
000000ff (255)
ibm,dynamic-memory
0000059e 00000000 20000000 80000002
00000000 00000001 00000008 00000000
30000000 80000003 00000000 00000001
00000008 00000000 40000000 80000004
00000000 00000001 00000008 00000000
50000000 80000005 00000000 00000001
00000008 00000000 60000000 80000006
00000000 00000001 00000008 00000000
[34516 bytes total]
ibm,phandle fffffff9 (-7)
linux,phandle 00ce70c0 (13529280)
ibm,associativity-lookup-arrays
00000002 00000004 00000000 00000000 00000000
00000000 00000000 00000001 00000002 00000002
ibm,memory-preservation-time
00000000
name "ibm,dynamic-reconfiguration-memory"
/proc/device-tree/options:
oem-banner? "false"
screen-#columns "80"
ibm,shadow-boot-device
",AIX-VDASD-IBM IPR-0 58C1F90000000220"
ibm,fw-default-mac-address?
"false"
ibm,fw-force-ior "false"
real-base "12582912"
ibm,fw-dc-select "100"
use-nvramrc? "false"
ibm,fw-n-dbfp "00000000"
ibm,fw-find-tape-alias
"false"
ibm,fw-n-ru "Y"
ibm,fw-tty-language
"1"
boot-command "boot"
ibm,fw-find-cdrom-alias
"false"
menu? "false"
reboot-command ""
ibm,last-booted "AIX-VDASD-IBM IPR-0 58C1F90000000220"
ibm,fw-nbr-reboots
"0"
ibm,fw-src-os-vec5
19 00 f3 00 c0 c0 05 00 00 00 00 01 00 00 00 00................
00 e0 00 00 00 01 00 00 00 00 00 ...........
input-device "/vdevice/vty"
real-size "134217728"
ibm,fw-override-cas
"false"
ibm,fw-usedrcinfo?
"false"
ibm,fw-new-mem-def
"true"
ibm,fw-n-dafp "00000000"
oem-logo? "false"
real-mode? "true"
boot-file ""
diag-switch? "false"
nvramrc ""
boot-device 2f766465 76696365 2f6c2d6c 616e4033
30303030 3030323a 73706565 643d6175
746f2c64 75706c65 783d6175 746f2c39
2e34302e 3139322e 38322c62 6f6f746c
6f616465 722d3031 2d62612d 36382d39
382d3261 2d31622d 30322c39 2e34302e
3139352e 3230332c 392e3430 2e313935
2e312c35 2c352c32 35352e32 35352e32
[184 bytes total]
ibm,fw-forced-boot
""
virt-base "4294967295"
output-device "/vdevice/vty"
oem-banner ""
ibm,fw-show-true-pcie?
"true"
little-endian? "false"
ibm,fw-oflite-dbg
"false"
ibm,fw-n-rc "A"
ibm,fw-n-bc "255.255.255.255"
ibm,associativity-form
"1"
diag-device 2f766465 76696365 2f6c2d6c 616e4033
30303030 3030323a 73706565 643d6175
746f2c64 75706c65 783d6175 746f2c39
2e34302e 3139322e 38322c2c 392e3430
2e313935 2e323033 2c392e34 302e3139
352e312c 352c352c 3235352e 3235352e
3235352e 302c3531 32202f76 64657669
63652f76 2d736373 69403330 30303030
[153 bytes total]
fcode-debug? "true"
ibm,fw-prev-boot-vpd
oem-logo
ibm,fw-prev-neg-vec5
14 00 f3 00 00 c0 00 00 00 00 00 01 00 00 00 00................
00 e0 00 00 00 00 ......
ibm,fw-menu-ba68982a1b02
"auto,auto,9.40.195.203,9.40.192.82,9.40.195.1,255.255.255.0,Standard,Yes,,5,512,5"
ibm,fw-usedynmemv2?
"false"
auto-boot? "true"
ibm,fw-vlan-menu-ba68982a1b02
"0,0"
ibm,phandle fffffffb (-5)
selftest-#megs "0"
ibm,dasd-spin-interval
"5"
boot-last-label 73657470 6172616d 73202752 65642048
61742045 6e746572 70726973 65204c69
6e757820 53657276 65722028 342e3132
2e302d72 63332e77 69393132 37355f30
3534615f 30353031 30312e70 70633634
6c652b29 20372e33 20284d61 69706f29
270a0a09 6c6f6164 5f766964 656f0a09
696e736d 6f642067 7a696f0a 09696e73
[832 bytes total]
load-base "16384"
ibm,fw-src-os-vec4
01 00 01 ...
ibm,fw-keyboard "1"
linux,phandle 00c9e0c0 (13230272)
virt-size "4294967295"
diag-file "diag"
name "options"
ibm,os-restart-policy
"1"
screen-#rows "24"
/proc/device-tree/rtas:
power-domains-names
"root power domain"
get-sensor-state 0000000c (12)
linux,rtas-entry 1ea51548 (514135368)
ibm,configure-kernel-dump-version
00010001 (65537)
rtas-sensors 00000001 00000000 00000005 00000000
00000009 00000000 0000232d 00000000
system-reboot 00000014 (20)
stop-self 00000018 (24)
ibm,hypertas-functions
6863616c 6c2d7066 74006863 616c6c2d
74636500 6863616c 6c2d7370 72673000
6863616c 6c2d636f 70790068 63616c6c
2d646562 75670068 63616c6c 2d746572
6d006863 616c6c2d 696e7465 72727570
74006863 616c6c2d 6d696772 61746500
6863616c 6c2d7065 72666d6f 6e006863
616c6c2d 73706c70 61720053 4c422d62
[434 bytes total]
ibm,lpar-perftools
00000045 (69)
ibm,create-pe-dma-window
00000054 (84)
ibm,nmi-register-2
00000030 (48)
ibm,nmi-register 00000030 (48)
ibm,configure-pe 00000052 (82)
ibm,display-line-length
00000010 (16)
ibm,lrdr-capacity
0000005a 00000000 00000000 10000000 00000020
ibm,dma-delay-time
0003e800 (256000)
ibm,nmi-interlock
00000031 (49)
ibm,indicator-0002
""
ibm,get-indicator-indices-types
0000232e 0000232f
ibm,read-slot-reset-state2
00000048 (72)
ibm,slot-error-detail
00000036 (54)
ibm,update-nodes 0000004f (79)
platform-power-sources
00035b60 00004a38 000038a4 00000032
ibm,errinjct 00000027 (39)
ibm,set-dynamic-indicator
00000046 (70)
ibm,update-properties
00000050 (80)
ibm,sensor-0009 ""
ibm,request-partition-shutdown
00000000
ibm,power-off-ups
0000004a (74)
ibm,flash-block-version
00000001
ibm,recoverable-epow3
ibm,integrated-stop-self
set-indicator 0000000b (11)
nvram-fetch 00000001
ibm,change-msi 0000004b (75)
ibm,manage-flash-image
0000003c (60)
ibm,configure-connector
0000002a (42)
rtas-error-log-max
00000800 (2048)
power-domains-tree
00000000 00000002 00000000 00000001
00000000 000000fa 00000064 00000001
00000000 000000fa 00000000
ibm,sensor-9005 ""
ibm,change-msix-capable
ibm,get-vpd 00000042 (66)
ibm,configure-kernel-dump-sizes
00000001 00000000 00200020 00000002 00000000 00001000
ibm,set-eeh-option
0000002b (43)
ibm,form-feed 0000000c (12)
ibm,get-sensor-indices-types
0000232e 0000232f
ibm,remove-pe-dma-window
00000055 (85)
ibm,sensor-0005 ""
ibm,query-interrupt-source-number
0000004c (76)
rtas-display-device
00e3e738 (14935864)
ibm,indicator-9005
""
ibm,write-pci-config
0000002f (47)
ibm,int-on 00000024 (36)
ibm,reset-pe-dma-window
00000056 (86)
ibm,max-associativity-domains
00000005 00000001 00000008 00000020 00000020 00000100
ibm,set-system-parameter
00000034 (52)
ibm,set-slot-reset
0000002c (44)
ibm,rks-hcalls e0000000
start-cpu 00000019 (25)
ibm,get-config-addr-info
0000004d (77)
ibm,reset-capabilities
00000001
ibm,set-xive 00000023 (35)
get-power-level 0000000e (14)
ibm,display-truncation-length
00000010 00000050
ibm,activate-firmware
00000043 (67)
ibm,suspend-me 0000004e (78)
ibm,extended-os-term
event-scan 00000006
set-power-level 0000000d (13)
ibm,configure-bridge
00000038 (56)
ibm,validate-flash-image
0000003d (61)
ibm,configure-kernel-dump
00000051 (81)
ibm,get-dynamic-sensor-state
00000047 (71)
ibm,sensor-0001 ""
ibm,open-errinjct
00000026 (38)
rtas-indicators 00000001 00000000 00000002 00000000 0000232d 00000000
ibm,associativity-reference-points
00000004 00000002
ibm,close-errinjct
00000028 (40)
ibm,get-config-addr-info2
0000004d (77)
ibm,os-term 0000001a (26)
display-character
0000000a (10)
rtas-size 015b0000 (22740992)
get-time-of-day 00000003
ibm,update-flash-64-and-reboot
00000021 (33)
check-exception 00000007
nvram-store 00000002
ibm,indicator-0001
""
ibm,vpd-size 0004b000 (307200)
ibm,phandle fffffffd (-3)
power-sources-names
"power supply"
set-time-of-day 00000004
power-off 00000011 (17)
set-time-for-power-on
00000005
rtas-event-scan-rate
00000004
linux,rtas-base 1ea50000 (514129920)
ibm,read-pci-config
0000002e (46)
ibm,get-indices 00000037 (55)
linux,phandle 00e068a0 (14706848)
ibm,read-slot-reset-state-functions
00000003
power-domains-controllers
00e3e738 (14935864)
ibm,int-off 00000025 (37)
ibm,get-system-parameter
00000032 (50)
query-cpu-stopped-state
0000001e (30)
rtas-version 00000001
ibm,display-number-of-lines
00000002
ibm,errinjct-tokens
7265636f 76657265 642d7370 65636961
6c2d6576 656e7400 00000003 636f7272
75707465 642d7061 67650000 00000469
6f612d62 75732d65 72726f72 00000000
07636f72 72757074 65642d64 63616368
652d7374 61727400 00000009 636f7272
75707465 642d6463 61636865 2d656e64
00000000 0a636f72 72757074 65642d69
[236 bytes total]
ibm,get-xive 00000022 (34)
ibm,query-pe-dma-window
00000053 (83)
name "rtas"
rtas-last-error 0000001f (31)
ibm,platform-dump
0000003b (59)
power-on-max-latency
ffffffff (-1)
/proc/device-tree/interrupt-controller@0:
compatible "IBM,ppc-xicp"
device_type "PowerPC-External-Interrupt-Presentation"
model "IBM, Logical PowerPC-PIC, 00"
ibm,interrupt-buid-size
00000014 (20)
built-in
reg 00000000 00000000 00000000 00000000
ibm,interrupt-server-ranges
00000000 00000080
linux,phandle 00e04a10 (14699024)
name "interrupt-controller"
interrupt-controller
/proc/device-tree/aliases:
nvram "/vdevice/nvram@4002"
net "/vdevice/l-lan@30000002"
scsi "/vdevice/v-scsi@30000078"
ibm,sp "/vdevice/IBM,sp@4000"
network "/vdevice/l-lan@30000002"
linux,phandle 00cefa00 (13564416)
name "aliases"
rtc "/vdevice/rtc@4001"
/proc/device-tree/chosen:
linux,stdout-package
00e1b528 (14791976)
ibm,architecture-vec-5
14 00 f3 00 00 c0 00 00 00 00 00 01 00 00 00 00................
00 e0 00 00 00 00 ......
nvram 08b05100 (145772800)
bootargs 424f4f54 5f494d41 47453d2f 766d6c69
6e757a2d 342e3132 2e302d72 63332e77
69393132 37355f30 3534615f 30353031
30312e70 70633634 6c652b20 726f6f74
3d2f6465 762f6d61 70706572 2f726865
6c5f6c74 63616c70 696e6532 2d2d6c70
32302d72 6f6f7420 726f2063 72617368
6b65726e 656c3d61 75746f20 72642e6c
[196 bytes total]
linux,initrd-start
00000000 0ce00000
cpu 08bc0480 (146539648)
linux,kernel-end 00000000 02550000
ibm,client-architecture-support-reboot
00000001
linux,initrd-end 00000000 0ef84b1a
stdout 08b04f80 (145772416)
memory 08bc0700 (146540288)
ibm,phandle fffffff7 (-9)
stdin 08b05800 (145774592)
linux,stdout-path
"/vdevice/vty@30000000"
linux,phandle 00c9dd50 (13229392)
bootpath "/vdevice/v-scsi@30000078/disk@8100000000000000"
name "chosen"
mmu 08bc0480 (146539648)
linux,memory-limit
00000000 00000000
/proc/device-tree/ibm,platform-facilities:
interrupt-ranges 00090400 00000400
ibm,interrupt-pool
00000000
device_type "ibm,platform-facilities"
#address-cells 00000001
#size-cells 00000000
ibm,phandle fffffff6 (-10)
linux,phandle 00d26c88 (13790344)
name "ibm,platform-facilities"
ibm,max-async-ops-per-processor
00000004
/proc/device-tree/ibm,platform-facilities/ibm,compression-v1:
compatible "ibm,compression"
ibm,max-sg-len 00000ff0 (4080)
ibm,resource-id 00000000
ibm,max-sync-cop 00000001 00010000 000001fe 00000001 00010000 000001fe
status "okay"
ibm,phandle fffffff4 (-12)
linux,phandle 00d28550 (13796688)
name "ibm,compression-v1"
ibm,shared-interrupt-pool
00d26c88 (13790344)
/proc/device-tree/ibm,platform-facilities/ibm,random-v1:
compatible "ibm,random"
ibm,max-sync-cop 00000000
status "okay"
ibm,phandle fffffff5 (-11)
linux,phandle 00d28368 (13796200)
name "ibm,random-v1"
ibm,shared-interrupt-pool
00d26c88 (13790344)
/proc/device-tree/ibm,platform-facilities/ibm,sym-encryption-v1:
compatible "ibm,sym-encryption"
ibm,max-sg-len 00000ff0 (4080)
ibm,resource-id 00000001
ibm,max-sync-cop 00000000 00000000 00000003 00000080
00062000 000000c4 000000c0 00062000
000000c4 00000100 00062000 000000c4
00000000 00000001 00000003 00000080
00062000 000000c4 000000c0 00062000
000000c4 00000100 00062000 000000c4
00000000 00000002 00000003 00000080
00062000 000000c4 000000c0 00062000
[768 bytes total]
status "okay"
ibm,phandle fffffff3 (-13)
linux,phandle 00d287c0 (13797312)
name "ibm,sym-encryption-v1"
ibm,shared-interrupt-pool
00d26c88 (13790344)
/proc/device-tree/openprom:
relative-addressing
ibm,fw-vernum_encoded
"FW860.00 (TV860_016)"
model "IBM,FW860.00 (TV860_016)"
ibm,phandle fffffffe (-2)
linux,phandle 00d25650 (13784656)
ibm,fw-next-bank "T"
name "openprom"
/proc/device-tree/packages:
linux,phandle 00c9dfc8 (13230024)
name "packages"
/proc/device-tree/packages/obp-tftp:
network "bootp"
"iscsi"
"ipv6"
"dhcp"
"vtag"
"csarch"
iscsi "iscsi"
"ipv6"
"dhcp"
linux,phandle 00d5d120 (14012704)
name "obp-tftp"
/proc/device-tree/packages/lpevents:
linux,phandle 00cf4390 (13583248)
name "lpevents"
/proc/device-tree/packages/fat-files:
linux,phandle 00dcce70 (14470768)
name "fat-files"
/proc/device-tree/packages/deblocker:
linux,phandle 00d56a98 (13986456)
name "deblocker"
/proc/device-tree/packages/gui:
linux,phandle 00e47f28 (14974760)
name "gui"
/proc/device-tree/packages/gui/iscsi:
linux,phandle 00e668b0 (15100080)
name "iscsi"
/proc/device-tree/packages/post:
linux,phandle 00e79878 (15177848)
name "post"
/proc/device-tree/packages/gscsi:
#address-cells 00000001
linux,phandle 00dbd100 (14405888)
name "gscsi"
/proc/device-tree/packages/gscsi/disk:
device_type "block"
linux,phandle 00dc2d98 (14429592)
name "disk"
/proc/device-tree/packages/gscsi/tape:
device_type "byte"
linux,phandle 00dc4be8 (14437352)
name "tape"
/proc/device-tree/packages/boot-mgr:
linux,phandle 00dee418 (14607384)
name "boot-mgr"
/proc/device-tree/packages/elf-loader:
linux,phandle 00de2ee0 (14560992)
name "elf-loader"
/proc/device-tree/packages/iso-9660-files:
linux,phandle 00de1910 (14555408)
name "iso-9660-files"
/proc/device-tree/packages/iptest:
linux,phandle 00dc68e0 (14444768)
name "iptest"
/proc/device-tree/packages/udp:
linux,phandle 00d7f010 (14151696)
name "udp"
/proc/device-tree/packages/tape-label:
linux,phandle 00d5c3b8 (14009272)
name "tape-label"
/proc/device-tree/packages/cas:
linux,phandle 00dff6e8 (14677736)
name "cas"
/proc/device-tree/packages/net:
linux,phandle 00de71a8 (14578088)
name "net"
/proc/device-tree/packages/ping:
linux,phandle 00d8d678 (14210680)
name "ping"
/proc/device-tree/packages/bootp:
linux,phandle 00d95620 (14243360)
name "bootp"
/proc/device-tree/packages/utilities:
linux,phandle 00d28e00 (13798912)
name "utilities"
/proc/device-tree/packages/utilities/trace:
linux,phandle 00d40900 (13895936)
name "trace"
/proc/device-tree/packages/disk-label:
linux,phandle 00d57fd8 (13991896)
name "disk-label"
/proc/device-tree/packages/prep-boot:
linux,phandle 00dcc5e0 (14468576)
name "prep-boot"
/proc/device-tree/packages/ipv6:
linux,phandle 00d77550 (14120272)
name "ipv6"
/proc/device-tree/packages/terminal-emulator:
iso6429-1983-colors
linux,phandle 00e02070 (14688368)
name "terminal-emulator"
/proc/device-tree/packages/gen-iso-13346-files:
linux,phandle 00dd8a40 (14518848)
name "gen-iso-13346-files"
/proc/device-tree/packages/dev-tree:
linux,phandle 00cf2928 (13576488)
name "dev-tree"
/proc/device-tree/packages/tcp:
linux,phandle 00d86048 (14180424)
name "tcp"
/proc/device-tree/packages/iso-13346-files:
linux,phandle 00dcf198 (14479768)
name "iso-13346-files"
/proc/device-tree/packages/ipv4:
linux,phandle 00d70270 (14090864)
name "ipv4"
/proc/device-tree/packages/tftp:
linux,phandle 00dab1e8 (14332392)
name "tftp"
/proc/device-tree/packages/iscsi:
linux,phandle 00db5760 (14374752)
name "iscsi"
/proc/device-tree/packages/nls-support:
linux,phandle 00dfe8f0 (14674160)
name "nls-support"
/proc/device-tree/packages/dhcp:
linux,phandle 00d9ef28 (14282536)
name "dhcp"
/proc/device-tree/packages/chrp-loader:
linux,phandle 00de2d38 (14560568)
name "chrp-loader"
/proc/device-tree/packages/dynamic-reconfig:
linux,phandle 00e02168 (14688616)
name "dynamic-reconfig"
/proc/device-tree/cpus:
ibm,drc-names 00000020 43505520 31004350 55203900
43505520 31370043 50552032 35004350
55203333 00435055 20343100 43505520
34390043 50552035 37004350 55203635
00435055 20373300 43505520 38310043
50552038 39004350 55203937 00435055
20313035 00435055 20313133 00435055
20313231 00435055 20313239 00435055
[245 bytes total]
ibm,drc-types 00000020 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
43505500 43505500 43505500 43505500
[132 bytes total]
ibm,drc-indexes 00000020 10000000 10000008 10000010
10000018 10000020 10000028 10000030
10000038 10000040 10000048 10000050
10000058 10000060 10000068 10000070
10000078 10000080 10000088 10000090
10000098 100000a0 100000a8 100000b0
100000b8 100000c0 100000c8 100000d0
100000d8 100000e0 100000e8 100000f0
[132 bytes total]
smp-enabled
#address-cells 00000001
#size-cells 00000000
ibm,drc-power-domains
00000020 ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
[132 bytes total]
ibm,phandle fffffff8 (-8)
linux,phandle 00ca0558 (13239640)
name "cpus"
/proc/device-tree/cpus/PowerPC,POWER8@8:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000008 (268435464)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000008 00000009 0000000a 0000000b
0000000c 0000000d 0000000e 0000000f
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000008 0000ffff 00000009 0000ffff
0000000a 0000ffff 0000000b 0000ffff
0000000c 0000ffff 0000000d 0000ffff
0000000e 0000ffff 0000000f 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000008
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000008 (268435464)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00ca7a50 (13269584)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@30:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000030 (268435504)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000030 00000031 00000032 00000033
00000034 00000035 00000036 00000037
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000030 0000ffff 00000031 0000ffff
00000032 0000ffff 00000033 0000ffff
00000034 0000ffff 00000035 0000ffff
00000036 0000ffff 00000037 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000030 (48)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000030 (268435504)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00cac9b0 (13289904)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@20:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000020 (268435488)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000020 00000021 00000022 00000023
00000024 00000025 00000026 00000027
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000020 0000ffff 00000021 0000ffff
00000022 0000ffff 00000023 0000ffff
00000024 0000ffff 00000025 0000ffff
00000026 0000ffff 00000027 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000020 (32)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000020 (268435488)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00caa9f0 (13281776)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@10:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000010 (268435472)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000010 00000011 00000012 00000013
00000014 00000015 00000016 00000017
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000010 0000ffff 00000011 0000ffff
00000012 0000ffff 00000013 0000ffff
00000014 0000ffff 00000015 0000ffff
00000016 0000ffff 00000017 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000010 (16)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000010 (268435472)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00ca8a30 (13273648)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@0:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000000 (268435456)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000000 00000001 00000002 00000003
00000004 00000005 00000006 00000007
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000000 0000ffff 00000001 0000ffff
00000002 0000ffff 00000003 0000ffff
00000004 0000ffff 00000005 0000ffff
00000006 0000ffff 00000007 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000000
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000000 (268435456)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00ca67d0 (13264848)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@38:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000038 (268435512)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000038 00000039 0000003a 0000003b
0000003c 0000003d 0000003e 0000003f
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000038 0000ffff 00000039 0000ffff
0000003a 0000ffff 0000003b 0000ffff
0000003c 0000ffff 0000003d 0000ffff
0000003e 0000ffff 0000003f 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000038 (56)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000038 (268435512)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00cad990 (13293968)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@28:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000028 (268435496)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000028 00000029 0000002a 0000002b
0000002c 0000002d 0000002e 0000002f
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000028 0000ffff 00000029 0000ffff
0000002a 0000ffff 0000002b 0000ffff
0000002c 0000ffff 0000002d 0000ffff
0000002e 0000ffff 0000002f 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000028 (40)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000028 (268435496)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00cab9d0 (13285840)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
/proc/device-tree/cpus/PowerPC,POWER8@18:
bus-frequency 00000000
performance-monitor
00000000 00000001
ibm,pft-size 00000000 00000020
ibm,vmx 00000003
ibm,spurr 00000001
d-tlb-sets 00000004
ibm,nominal-tbf 1e848000 (512000000)
64-bit
timebase-frequency
1e848000 (512000000)
ibm,extended-clock-frequency
00000000 b45d1880
ibm,estimate-precision
66 72 65 00 00 00 00 0e 66 72 65 73 00 00 00 00fre.....fres....
0e 66 72 73 71 72 74 65 00 00 00 00 0e 66 72 73.frsqrte.....frs
71 72 74 65 73 00 00 00 00 0e qrtes.....
tlb-size 00000800 (2048)
ibm,pa-features 18 00 f6 3f c7 c0 80 f0 80 00 00 00 00 00 00 00...?............
00 00 80 00 80 00 80 00 80 00 ..........
graphics
ibm,slb-size 00000020 (32)
device_type "cpu"
tlb-sets 00000004
ibm,my-drc-index 10000018 (268435480)
reservation-granule-size
00000080 (128)
ibm,processor-storage-keys
0000001f 00000000
ibm,processor-vadd-size
00000044 (68)
slb-size 00000020 (32)
ibm,dfp 00000002
ibm,ppc-interrupt-server#s
00000018 00000019 0000001a 0000001b
0000001c 0000001d 0000001e 0000001f
ibm,vrma-page-sizes
00000018 00000010 0000000c 00000000
00000010 00000011 00000022 00000013
ibm,segment-page-sizes
0000000c 00000000 00000003 0000000c
00000000 00000010 00000007 00000018
00000038 00000010 00000110 00000002
00000010 00000001 00000018 00000008
00000018 00000100 00000001 00000018
00000000 00000022 00000120 00000001
00000022 00000003
i-cache-line-size
00000080 (128)
i-tlb-size 00000000
ibm,pa-optimizations
02001004 (33558532)
status "okay"
cpu-version 0f000004 (251658244)
ibm,purr 00000001
ibm,ppc-interrupt-gserver#s
00000018 0000ffff 00000019 0000ffff
0000001a 0000ffff 0000001b 0000ffff
0000001c 0000ffff 0000001d 0000ffff
0000001e 0000ffff 0000001f 0000ffff
i-tlb-sets 00000000
ibm,tbu40-offset ffffb452 a7000000
ibm,pi-features 01 00 a0 ...
ibm,negotiated-pa-features
01 00 00 ...
i-cache-block-size
00000080 (128)
32-64-bridge
ibm,extended-bus-frequency
00000000 00000000
reg 00000018 (24)
d-cache-block-size
00000080 (128)
clock-frequency b45d1880
ibm,phandle 10000018 (268435480)
ibm,processor-segment-sizes
0000001c 00000028 ffffffff ffffffff
linux,phandle 00ca9a10 (13277712)
d-cache-size 00010000 (65536)
i-cache-size 00008000 (32768)
general-purpose
ibm,raw-pi-features
01 00 00 ...
d-tlb-size 00000800 (2048)
name "PowerPC,POWER8"
ibm,processor-page-sizes
0000000c 00000010 00000018 00000022
d-cache-sets 00000008
i-cache-sets 00000004
d-cache-line-size
00000080 (128)
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-02 5:24 ` Michael Bringmann
@ 2017-06-06 9:48 ` Michael Ellerman
2017-06-06 16:16 ` Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Michael Ellerman @ 2017-06-06 9:48 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>
> When the 'numa.c' code is built with debug messages, and the system was
> given that configuration by pHyp, yes, I did.
>
>> What does it say?
>
> The debug message for each core thread would be something like,
>
> removing cpu 64 from node 0
> adding cpu 64 to node 8
>
> repeated for all 8 threads of the CPU, and usually with the messages
> for all of the CPUs coming out intermixed on the console/dmesg log.
OK. I meant what do you see at boot.
I'm curious how we're discovering node 0 and 8 at all if neither has any
memory or CPUs assigned at boot.
>> Right. So it's not that you're hot adding memory into a previously
>> unseen node as you implied in earlier mails.
>
> In the sense that the nodes were defined in the device tree, that is correct.
Where are they defined in the device tree? That's what I'm trying to understand.
> In the sense that those nodes are currently deleted from node_possible_map in
> 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
> node_online_map);', the nodes are no longer available to place memory or CPU.
Yeah I understand that part.
> Okay, I can try to insert code that extracts all of the nodes from the
> ibm,associativity-lookup-arrays property and merge them with the nodes
> put into the online map from the CPUs that were found previously during
> boot of the powerpc code.
Hmm, will that work?
Looking at PAPR it's not clear to me that it will work for nodes that
have no memory assigned at boot.
This property is used to duplicate the function of the
“ibm,associativity” property in a /memory node. Each “assigned” LMB
represented has an index valued between 0 and M-1 which is used as in
index into this table to select which associativity list to use for
the LMB. “unassigned” LMBs are place holders for potential DLPAR
additions, for which the associativity list index is meaningless and
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
is given the reserved value of -1. This static property, need only
contain values relevant for the LMBs presented in the
“ibm,dynamicreconfiguration-memory” node; for a dynamic LPAR addition
of a new LMB, the device tree fragment reported by the
ibm,configure-connector RTAS function is a /memory node, with the
inclusion of the “ibm,associativity” device tree property defined in
Section C.6.2.2‚ “Properties of the Children of Root‚” on page 1059.
>> What does your device tree look like? Can you send us the output of:
>>
>> $ lsprop /proc/device-tree
Thanks. I forgot that lsprop will truncate long properties, I actually
wanted to see all of the ibm,dynamic-memory property.
But looking at the code I see the only place we set a nid online is if
there is a CPU assigned to it:
static int __init parse_numa_properties(void)
{
...
for_each_present_cpu(i) {
...
cpu = of_get_cpu_node(i, NULL);
nid = of_node_to_nid_single(cpu);
...
node_set_online(nid);
}
Or for memory nodes (same function):
for_each_node_by_type(memory, "memory") {
...
nid = of_node_to_nid_single(memory);
...
node_set_online(nid);
...
}
Or for entries in ibm,dynamic-memory that are assigned:
static void __init parse_drconf_memory(struct device_node *memory)
{
...
for (; n != 0; --n) {
...
/* skip this block if the reserved bit is set in flags (0x80)
or if the block is not assigned to this partition (0x8) */
if ((drmem.flags & DRCONF_MEM_RESERVED)
|| !(drmem.flags & DRCONF_MEM_ASSIGNED))
continue;
...
do {
...
nid = of_drconf_to_nid_single(&drmem, &aa);
node_set_online(nid);
...
} while (--ranges);
}
}
So I don't see from that how we can even be aware that node 0 and 8
exist at boot based on that. Maybe there's another path I'm missing
though.
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-06 9:48 ` Michael Ellerman
@ 2017-06-06 16:16 ` Michael Bringmann
2017-06-07 12:08 ` Michael Ellerman
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-06-06 16:16 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
On 06/06/2017 04:48 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>
>> When the 'numa.c' code is built with debug messages, and the system was
>> given that configuration by pHyp, yes, I did.
>>
>>> What does it say?
>>
>> The debug message for each core thread would be something like,
>>
>> removing cpu 64 from node 0
>> adding cpu 64 to node 8
>>
>> repeated for all 8 threads of the CPU, and usually with the messages
>> for all of the CPUs coming out intermixed on the console/dmesg log.
>
> OK. I meant what do you see at boot.
Here is an example with nodes 0,2,6,7, node 0 starts out empty:
[ 0.000000] Initmem setup node 0
[ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
[ 0.000000] NODE_DATA(0) on node 7
[ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
[ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
[ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
[ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff]
[ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
[ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff]
[ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff]
[ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff]
[ 0.000000] Could not find start_pfn for node 0
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
[ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
[ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
[ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
[ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320
and,
[root@ltcalpine2-lp20 ~]# numactl --hardware
available: 4 nodes (0,2,6-7)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
node 2 size: 81792 MB
node 2 free: 81033 MB
node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
node 6 size: 133743 MB
node 6 free: 133097 MB
node 7 cpus: 48 49 50 51 52 53 54 55
node 7 size: 29877 MB
node 7 free: 29599 MB
node distances:
node 0 2 6 7
0: 10 40 40 40
2: 40 10 40 40
6: 40 40 10 20
7: 40 40 20 10
[root@ltcalpine2-lp20 ~]#
>
> I'm curious how we're discovering node 0 and 8 at all if neither has any
> memory or CPUs assigned at boot.
Both are in the memory associativity lookup arrays. And we are circling
back to the
>
>>> Right. So it's not that you're hot adding memory into a previously
>>> unseen node as you implied in earlier mails.
>>
>> In the sense that the nodes were defined in the device tree, that is correct.
>
> Where are they defined in the device tree? That's what I'm trying to understand.
The nodes for memory are defined one time in "ibm,associativity-lookup-arrays".
I wish that there was an official set of node properties in the device-tree,
instead of having them be the values of other properties.
>
>> In the sense that those nodes are currently deleted from node_possible_map in
>> 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map,
>> node_online_map);', the nodes are no longer available to place memory or CPU.
>
> Yeah I understand that part.
>
>> Okay, I can try to insert code that extracts all of the nodes from the
>> ibm,associativity-lookup-arrays property and merge them with the nodes
>> put into the online map from the CPUs that were found previously during
>> boot of the powerpc code.
>
> Hmm, will that work?
The nodes are defined in the associativity lookup array, so they have at least
been reserved for us by the pHyp. On the other hand, if we are only to use
nodes that have resources at boot, why are there extra node values specified?
What I am not 100% clear on -- and why I preferred letting all possible nodes
originally defined, still be possible for subsequent hot-add operations -- is
whether the nodes to be used for hot-added CPUs would always be a subset of
the nodes used for hot-added memory.
* The hot-added CPUs in Shared CPU configurations may be mapped to nodes by
the value returned to the kernel by the VPHN hcall.
* So far in my tests, this has not been a problem, but I could not be positive
from the PAPR.
>
> Looking at PAPR it's not clear to me that it will work for nodes that
> have no memory assigned at boot.
>
> This property is used to duplicate the function of the
> “ibm,associativity” property in a /memory node. Each “assigned” LMB
> represented has an index valued between 0 and M-1 which is used as in
> index into this table to select which associativity list to use for
> the LMB. “unassigned” LMBs are place holders for potential DLPAR
> additions, for which the associativity list index is meaningless and
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> is given the reserved value of -1. This static property, need only
> contain values relevant for the LMBs presented in the
> “ibm,dynamicreconfiguration-memory” node; for a dynamic LPAR addition
> of a new LMB, the device tree fragment reported by the
> ibm,configure-connector RTAS function is a /memory node, with the
> inclusion of the “ibm,associativity” device tree property defined in
> Section C.6.2.2‚ “Properties of the Children of Root‚” on page 1059.
I don't see any place that builds new /memory nodes in conjunction with
hot-added memory. The code for powerpc treats the definitions provided
by 'ibm,dynamic-reconfiguration-memory' as the primary reference wherever
hot-added memory comes into play. It looks like the '/memory' properties
are backups or used for the 'pseries' according to one comment.
>
>>> What does your device tree look like? Can you send us the output of:
>>>
>>> $ lsprop /proc/device-tree
>
> Thanks. I forgot that lsprop will truncate long properties, I actually
> wanted to see all of the ibm,dynamic-memory property.
>
> But looking at the code I see the only place we set a nid online is if
> there is a CPU assigned to it:
>
> static int __init parse_numa_properties(void)
> {
> ...
> for_each_present_cpu(i) {
> ...
> cpu = of_get_cpu_node(i, NULL);
> nid = of_node_to_nid_single(cpu);
> ...
> node_set_online(nid);
> }
>
> Or for memory nodes (same function):
>
> for_each_node_by_type(memory, "memory") {
> ...
> nid = of_node_to_nid_single(memory);
> ...
> node_set_online(nid);
> ...
> }
>
> Or for entries in ibm,dynamic-memory that are assigned:
>
> static void __init parse_drconf_memory(struct device_node *memory)
> {
> ...
> for (; n != 0; --n) {
> ...
> /* skip this block if the reserved bit is set in flags (0x80)
> or if the block is not assigned to this partition (0x8) */
> if ((drmem.flags & DRCONF_MEM_RESERVED)
> || !(drmem.flags & DRCONF_MEM_ASSIGNED))
> continue;
>
> ...
> do {
> ...
> nid = of_drconf_to_nid_single(&drmem, &aa);
> node_set_online(nid);
> ...
> } while (--ranges);
> }
> }
>
>
> So I don't see from that how we can even be aware that node 0 and 8
> exist at boot based on that. Maybe there's another path I'm missing
> though.
We don't 'fill in' the nodes, but we are aware that they exist per the 'ibm,associativity-lookup-arrays'
or the responsed provided by the pHyp to the VPHN hcall. We don't associate either of these resources
to them, but does that mean that the nodes do not exist?
The code currently says that only nodes booted with resources "exist" i.e. it can't hot-add new nodes,
but is that a just a problem of the kernel implementation? I think so.
However, this is the problem for users running systems that hot-add a lot of resources are concerned.
They see the associativity arrays (and 'hypinfo' table internal to the pHyp), and they ask why the
kernel only records new resources into the boot-time nodes, while pHyp appears to distribute across
all of the memory nodes specified to the LPAR of the kernel at boot.
I think that all of those nodes specified by the pHyp should exist to the kernel, and that we are
trying to find the best way to make them visible here.
>
> cheers
>
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-01 21:33 ` Reza Arbab
@ 2017-06-07 8:06 ` Balbir Singh
2017-06-07 12:07 ` Michael Ellerman
1 sibling, 0 replies; 33+ messages in thread
From: Balbir Singh @ 2017-06-07 8:06 UTC (permalink / raw)
To: Reza Arbab, Michael Ellerman
Cc: Michael Bringmann, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
On Thu, 2017-06-01 at 16:33 -0500, Reza Arbab wrote:
> On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
> > I don't think that's what the patch does. It just marks 32 (!?) nodes
> > as online. Or if you're talking about reverting 3af229f2071f that
> > leaves you with 256 possible nodes. Both of which are wasteful.
>
> To be clear, with Balbir's set the latter is no longer wasteful.
>
> > The right fix is to make sure any nodes which are present at boot
> > remain in the possible map, even if they don't have memory/CPUs
> > assigned at boot.
>
> I'm still hoping 3af229f2071f could indeed be reverted some day, but
> until then the following would follow your suggestion for our GPU nodes.
> What do you think?
>
> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
> void __init initmem_init(void)
> {
> int nid, cpu;
> + struct device_node *dn;
>
> max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
> max_pfn = max_low_pfn;
> @@ -911,6 +912,18 @@ void __init initmem_init(void)
> */
> nodes_and(node_possible_map, node_possible_map, node_online_map);
>
> + /*
> + * Consider an ibm,coherent-device-memory node possible. Even though
> + * it is not online at boot, it may be hotplugged later.
> + */
> + for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
> + nid = of_node_to_nid_single(dn);
> + if (nid < 0)
> + continue;
> +
> + node_set(nid, node_possible_map);
> + }
> +
I think it looks reasonable, although I'd really like to set a limit
in firmware on the number of nodes and fix memcg hotplug correctly in
the medium term.
Balbir Singh.
> for_each_online_node(nid) {
> unsigned long start_pfn, end_pfn;
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-01 21:33 ` Reza Arbab
2017-06-07 8:06 ` Balbir Singh
@ 2017-06-07 12:07 ` Michael Ellerman
1 sibling, 0 replies; 33+ messages in thread
From: Michael Ellerman @ 2017-06-07 12:07 UTC (permalink / raw)
To: Reza Arbab
Cc: Michael Bringmann, Balbir Singh, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, linuxppc-dev, Sebastian Andrzej Siewior
Reza Arbab <arbab@linux.vnet.ibm.com> writes:
> On Thu, Jun 01, 2017 at 07:36:31PM +1000, Michael Ellerman wrote:
>>I don't think that's what the patch does. It just marks 32 (!?) nodes
>>as online. Or if you're talking about reverting 3af229f2071f that
>>leaves you with 256 possible nodes. Both of which are wasteful.
>
> To be clear, with Balbir's set the latter is no longer wasteful.
AFAIK that series has been rejected.
And even so, it only fixes one known case where having a very large
possible map is wasteful, there could be others we haven't found, or
there could be more introduced in future.
So we will always prefer to keep the possible map as small as it can be.
>>The right fix is to make sure any nodes which are present at boot
>>remain in the possible map, even if they don't have memory/CPUs
>>assigned at boot.
>
> I'm still hoping 3af229f2071f could indeed be reverted some day, but
> until then the following would follow your suggestion for our GPU nodes.
> What do you think?
Yeah we can do something like that.
We might want to come up with a generic property, rather than looking
specifically for ibm,coherent-device-memory.
And we might also need a way for skiboot to tell us "in addition to
what is in the device tree at boot, up to N more nodes could be hot
plugged".
cheers
> --- a/arch/powerpc/mm/numa.c
> +++ b/arch/powerpc/mm/numa.c
> @@ -895,6 +895,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
> void __init initmem_init(void)
> {
> int nid, cpu;
> + struct device_node *dn;
>
> max_low_pfn = memblock_end_of_DRAM() >> PAGE_SHIFT;
> max_pfn = max_low_pfn;
> @@ -911,6 +912,18 @@ void __init initmem_init(void)
> */
> nodes_and(node_possible_map, node_possible_map, node_online_map);
>
> + /*
> + * Consider an ibm,coherent-device-memory node possible. Even though
> + * it is not online at boot, it may be hotplugged later.
> + */
> + for_each_compatible_node(dn, NULL, "ibm,coherent-device-memory") {
> + nid = of_node_to_nid_single(dn);
> + if (nid < 0)
> + continue;
> +
> + node_set(nid, node_possible_map);
> + }
> +
> for_each_online_node(nid) {
> unsigned long start_pfn, end_pfn;
>
>
> --
> Reza Arbab
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-06 16:16 ` Michael Bringmann
@ 2017-06-07 12:08 ` Michael Ellerman
2017-06-07 17:28 ` RESEND " Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Michael Ellerman @ 2017-06-07 12:08 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> On 06/06/2017 04:48 AM, Michael Ellerman wrote:
>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>>
>>> When the 'numa.c' code is built with debug messages, and the system was
>>> given that configuration by pHyp, yes, I did.
>>>
>>>> What does it say?
>>>
>>> The debug message for each core thread would be something like,
>>>
>>> removing cpu 64 from node 0
>>> adding cpu 64 to node 8
>>>
>>> repeated for all 8 threads of the CPU, and usually with the messages
>>> for all of the CPUs coming out intermixed on the console/dmesg log.
>>
>> OK. I meant what do you see at boot.
>
> Here is an example with nodes 0,2,6,7, node 0 starts out empty:
>
> [ 0.000000] Initmem setup node 0
> [ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
> [ 0.000000] NODE_DATA(0) on node 7
> [ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
> [ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
> [ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
> [ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff]
> [ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
> [ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]
>
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff]
> [ 0.000000] DMA32 empty
> [ 0.000000] Normal empty
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff]
> [ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff]
> [ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff]
> [ 0.000000] Could not find start_pfn for node 0
> [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
> [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
> [ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
> [ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
> [ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
> [ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320
>
> and,
>
> [root@ltcalpine2-lp20 ~]# numactl --hardware
> available: 4 nodes (0,2,6-7)
> node 0 cpus:
> node 0 size: 0 MB
> node 0 free: 0 MB
> node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
> node 2 size: 81792 MB
> node 2 free: 81033 MB
> node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
> node 6 size: 133743 MB
> node 6 free: 133097 MB
> node 7 cpus: 48 49 50 51 52 53 54 55
> node 7 size: 29877 MB
> node 7 free: 29599 MB
> node distances:
> node 0 2 6 7
> 0: 10 40 40 40
> 2: 40 10 40 40
> 6: 40 40 10 20
> 7: 40 40 20 10
> [root@ltcalpine2-lp20 ~]#
What kernel is that running?
And can you show me the full ibm,dynamic-memory and lookup-arrays
properties for that system?
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-07 12:08 ` Michael Ellerman
@ 2017-06-07 17:28 ` Michael Bringmann
2017-06-13 10:45 ` Michael Ellerman
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-06-07 17:28 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
[-- Attachment #1: Type: text/plain, Size: 3833 bytes --]
Here is the information from 2 different kernels. I have not been able to retrieve
the information matching yesterday's attachments, yet, as those dumps were
acquired in April.
Attached please find 2 dumps of similar material from kernels running with my
current patches (Linux 4.4, Linux 4.12).
On 06/07/2017 07:08 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>
>> On 06/06/2017 04:48 AM, Michael Ellerman wrote:
>>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>>>> On 06/01/2017 04:36 AM, Michael Ellerman wrote:
>>>>> Do you actually see mention of nodes 0 and 8 in the dmesg?
>>>>
>>>> When the 'numa.c' code is built with debug messages, and the system was
>>>> given that configuration by pHyp, yes, I did.
>>>>
>>>>> What does it say?
>>>>
>>>> The debug message for each core thread would be something like,
>>>>
>>>> removing cpu 64 from node 0
>>>> adding cpu 64 to node 8
>>>>
>>>> repeated for all 8 threads of the CPU, and usually with the messages
>>>> for all of the CPUs coming out intermixed on the console/dmesg log.
>>>
>>> OK. I meant what do you see at boot.
>>
>> Here is an example with nodes 0,2,6,7, node 0 starts out empty:
>>
>> [ 0.000000] Initmem setup node 0
>> [ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff]
>> [ 0.000000] NODE_DATA(0) on node 7
>> [ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff]
>> [ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff]
>> [ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff]
>> [ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff]
>> [ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff]
>> [ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff]
>>
>> [ 0.000000] Zone ranges:
>> [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff]
>> [ 0.000000] DMA32 empty
>> [ 0.000000] Normal empty
>> [ 0.000000] Movable zone start for each node
>> [ 0.000000] Early memory node ranges
>> [ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff]
>> [ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff]
>> [ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff]
>> [ 0.000000] Could not find start_pfn for node 0
>> [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
>> [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff]
>> [ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff]
>> [ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff]
>> [ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144
>> [ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320
>>
>> and,
>>
>> [root@ltcalpine2-lp20 ~]# numactl --hardware
>> available: 4 nodes (0,2,6-7)
>> node 0 cpus:
>> node 0 size: 0 MB
>> node 0 free: 0 MB
>> node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63
>> node 2 size: 81792 MB
>> node 2 free: 81033 MB
>> node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47
>> node 6 size: 133743 MB
>> node 6 free: 133097 MB
>> node 7 cpus: 48 49 50 51 52 53 54 55
>> node 7 size: 29877 MB
>> node 7 free: 29599 MB
>> node distances:
>> node 0 2 6 7
>> 0: 10 40 40 40
>> 2: 40 10 40 40
>> 6: 40 40 10 20
>> 7: 40 40 20 10
>> [root@ltcalpine2-lp20 ~]#
>
> What kernel is that running?
>
> And can you show me the full ibm,dynamic-memory and lookup-arrays
> properties for that system?
>
> cheers
>
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
[-- Attachment #2: 1.linux4.12-3nodes.txt --]
[-- Type: text/plain, Size: 4480 bytes --]
Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 4.12.0-rc3.wi91275_054c_060106.ppc64le+ on an ppc64le
ltcalpine2-lp20 login: root
Password:
Last login: Wed Jun 7 11:03:12 from oc1554177480.austin.ibm.com
[root@ltcalpine2-lp20 ~]# numactl -H
available: 3 nodes (0,2-3)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 48 49 50 51 52 53 54 55
node 2 size: 188668 MB
node 2 free: 187903 MB
node 3 cpus: 40 41 42 43 44 45 46 47 56 57 58 59 60 61 62 63
node 3 size: 56261 MB
node 3 free: 55324 MB
node distances:
node 0 2 3
0: 10 40 40
2: 40 10 20
3: 40 20 10
[root@ltcalpine2-lp20 ~]# cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,dynamic-memory
ibm,dynamic-memory
0000059e 00000000 20000000 80000002
00000000 00000001 00000008 00000000
30000000 80000003 00000000 00000001
00000008 00000000 40000000 80000004
00000000 00000001 00000008 00000000
50000000 80000005 00000000 00000001
00000008 00000000 60000000 80000006
00000000 00000001 00000008 00000000
[34516 bytes total]
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
00000003 00000004 00000000 00000000
00000000 00000000 00000000 00000001
00000002 00000002 00000000 00000001
00000003 00000003
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# cat ~/dmesg.log
[ 0.000000] Linux version 4.12.0-rc3.wi91275_054c_060106.ppc64le+ (root@ltcalpine2-lp20.aus.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #28 SMP Tue Jun 6 16:05:01 EDT 2017
[ 0.000000] Found initrd at 0xc00000000ce00000:0xc00000000eeb4b12
[ 0.000000] Using pSeries machine description
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 256 cpus.
[ 0.000000] CPU maps initialized for 8 threads per core
[ 0.000000] (thread shift is 3)
[ 0.000000] Freed 4784128 bytes for unused pacas
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x20
[ 0.000000] phys_mem_size = 0x3c00000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x27fc7aec18500249
[ 0.000000] possible = 0x7fffffff18500649
[ 0.000000] always = 0x0000000018100040
[ 0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[ 0.000000] mmu_features = 0x7c006001
[ 0.000000] firmware_features = 0x00000003c45bfc57
[ 0.000000] htab_hash_mask = 0x1ffffff
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x3bff7d5f00-0x3bff7dffff]
[ 0.000000] numa: NODE_DATA(0) on node 3
[ 0.000000] numa: NODE_DATA [mem 0x2e1fff5f00-0x2e1fffffff]
[ 0.000000] numa: NODE_DATA [mem 0x3bff7cbe00-0x3bff7d5eff]
[ 0.000000] Section 15357 and 15359 (node 3) have a circular dependency on usemap and pgdat allocations
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[ 0.000000] Memory hole size: 0MB
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 2: [mem 0x0000000000000000-0x0000002e1fffffff]
[ 0.000000] node 3: [mem 0x0000002e20000000-0x0000003bffffffff]
[ 0.000000] Could not find start_pfn for node 0
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000] On node 0 totalpages: 0
[ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x0000002e1fffffff]
[ 0.000000] On node 2 totalpages: 3022848
[ 0.000000] DMA zone: 2952 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 3022848 pages, LIFO batch:1
[ 0.000000] Initmem setup node 3 [mem 0x0000002e20000000-0x0000003bffffffff]
[ 0.000000] On node 3 totalpages: 909312
[ 0.000000] DMA zone: 888 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 909312 pages, LIFO batch:1
[root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]#
[-- Attachment #3: 2.linux4.4-4nodes.txt --]
[-- Type: text/plain, Size: 88606 bytes --]
Welcome to SUSE Linux Enterprise Server 12 SP3 Beta2 (ppc64le) - Kernel 4.4.63-2.1.wi91275.sp3.055a_27-default (hvc0).
ltcalpine2-lp9 login: root
Password:
Last login: Mon Jun 5 11:43:45 from 9.53.92.157
ltcalpine2-lp9:ß # numactl -H
available: 4 nodes (0-1,6-7)
node 0 cpus:
node 0 size: 20512 MB
node 0 free: 20371 MB
node 1 cpus:
node 1 size: 2815 MB
node 1 free: 2706 MB
node 6 cpus: 0 1 2 3 4 5 6 7
node 6 size: 134783 MB
node 6 free: 134068 MB
node 7 cpus:
node 7 size: 87210 MB
node 7 free: 87102 MB
node distances:
node 0 1 6 7
0: 10 20 40 40
1: 20 10 40 40
6: 40 40 10 20
7: 40 40 20 10
ltcalpine2-lp9:~ # cd
ltcalpine2-lp9:~ # cd /proc/device-tree/ibm,dynamic-reconfiguration-memory
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # ls
ibm,associativity-lookup-arrays ibm,memory-flags-mask linux,phandle
ibm,dynamic-memory ibm,memory-preservation-time name
ibm,lmb-size ibm,phandle
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
00000004 00000004 00000000 00000000 00000000 00000000
00000000 00000000 00000001 00000001 00000000 00000003
00000006 00000006 00000000 00000003 00000007 00000007
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,dynamic-memory
ibm,dynamic-memory
0000059e 00000000 20000000 80000002
00000000 00000000 00000008 00000000
30000000 80000003 00000000 00000000
00000008 00000000 40000000 80000004
00000000 00000000 00000008 00000000
50000000 80000005 00000000 00000000
00000008 00000000 60000000 80000006
00000000 00000000 00000008 00000000
70000000 80000007 00000000 00000000
00000008 00000000 80000000 80000008
00000000 00000000 00000008 00000000
90000000 80000009 00000000 00000000
00000008 00000000 a0000000 8000000a
00000000 00000000 00000008 00000000
b0000000 8000000b 00000000 00000000
00000008 00000000 c0000000 8000000c
00000000 00000000 00000008 00000000
d0000000 8000000d 00000000 00000000
00000008 00000000 e0000000 8000000e
00000000 00000000 00000008 00000000
f0000000 8000000f 00000000 00000000
00000008 00000001 00000000 80000010
00000000 00000000 00000008 00000001
10000000 80000011 00000000 00000000
00000008 00000001 20000000 80000012
00000000 00000000 00000008 00000001
30000000 80000013 00000000 00000000
00000008 00000001 40000000 80000014
00000000 00000000 00000008 00000001
50000000 80000015 00000000 00000000
00000008 00000001 60000000 80000016
00000000 00000000 00000008 00000001
70000000 80000017 00000000 00000000
00000008 00000001 80000000 80000018
00000000 00000000 00000008 00000001
90000000 80000019 00000000 00000000
00000008 00000001 a0000000 8000001a
00000000 00000000 00000008 00000001
b0000000 8000001b 00000000 00000000
00000008 00000001 c0000000 8000001c
00000000 00000000 00000008 00000001
d0000000 8000001d 00000000 00000000
00000008 00000001 e0000000 8000001e
00000000 00000000 00000008 00000001
f0000000 8000001f 00000000 00000000
00000008 00000002 00000000 80000020
00000000 00000000 00000008 00000002
10000000 80000021 00000000 00000000
00000008 00000002 20000000 80000022
00000000 00000000 00000008 00000002
30000000 80000023 00000000 00000000
00000008 00000002 40000000 80000024
00000000 00000000 00000008 00000002
50000000 80000025 00000000 00000000
00000008 00000002 60000000 80000026
00000000 00000000 00000008 00000002
70000000 80000027 00000000 00000000
00000008 00000002 80000000 80000028
00000000 00000000 00000008 00000002
90000000 80000029 00000000 00000000
00000008 00000002 a0000000 8000002a
00000000 00000000 00000008 00000002
b0000000 8000002b 00000000 00000000
00000008 00000002 c0000000 8000002c
00000000 00000000 00000008 00000002
d0000000 8000002d 00000000 00000000
00000008 00000002 e0000000 8000002e
00000000 00000000 00000008 00000002
f0000000 8000002f 00000000 00000000
00000008 00000003 00000000 80000030
00000000 00000000 00000008 00000003
10000000 80000031 00000000 00000000
00000008 00000003 20000000 80000032
00000000 00000000 00000008 00000003
30000000 80000033 00000000 00000000
00000008 00000003 40000000 80000034
00000000 00000000 00000008 00000003
50000000 80000035 00000000 00000000
00000008 00000003 60000000 80000036
00000000 00000000 00000008 00000003
70000000 80000037 00000000 00000000
00000008 00000003 80000000 80000038
00000000 00000000 00000008 00000003
90000000 80000039 00000000 00000000
00000008 00000003 a0000000 8000003a
00000000 00000000 00000008 00000003
b0000000 8000003b 00000000 00000000
00000008 00000003 c0000000 8000003c
00000000 00000000 00000008 00000003
d0000000 8000003d 00000000 00000000
00000008 00000003 e0000000 8000003e
00000000 00000000 00000008 00000003
f0000000 8000003f 00000000 00000000
00000008 00000004 00000000 80000040
00000000 00000000 00000008 00000004
10000000 80000041 00000000 00000000
00000008 00000004 20000000 80000042
00000000 00000000 00000008 00000004
30000000 80000043 00000000 00000000
00000008 00000004 40000000 80000044
00000000 00000000 00000008 00000004
50000000 80000045 00000000 00000000
00000008 00000004 60000000 80000046
00000000 00000000 00000008 00000004
70000000 80000047 00000000 00000000
00000008 00000004 80000000 80000048
00000000 00000000 00000008 00000004
90000000 80000049 00000000 00000000
00000008 00000004 a0000000 8000004a
00000000 00000000 00000008 00000004
b0000000 8000004b 00000000 00000000
00000008 00000004 c0000000 8000004c
00000000 00000000 00000008 00000004
d0000000 8000004d 00000000 00000000
00000008 00000004 e0000000 8000004e
00000000 00000000 00000008 00000004
f0000000 8000004f 00000000 00000000
00000008 00000005 00000000 80000050
00000000 00000000 00000008 00000005
10000000 80000051 00000000 00000001
00000008 00000005 20000000 80000052
00000000 00000001 00000008 00000005
30000000 80000053 00000000 00000001
00000008 00000005 40000000 80000054
00000000 00000001 00000008 00000005
50000000 80000055 00000000 00000001
00000008 00000005 60000000 80000056
00000000 00000001 00000008 00000005
70000000 80000057 00000000 00000001
00000008 00000005 80000000 80000058
00000000 00000001 00000008 00000005
90000000 80000059 00000000 00000001
00000008 00000005 a0000000 8000005a
00000000 00000001 00000008 00000005
b0000000 8000005b 00000000 00000001
00000008 00000005 c0000000 8000005c
00000000 00000002 00000008 00000005
d0000000 8000005d 00000000 00000002
00000008 00000005 e0000000 8000005e
00000000 00000002 00000008 00000005
f0000000 8000005f 00000000 00000002
00000008 00000006 00000000 80000060
00000000 00000002 00000008 00000006
10000000 80000061 00000000 00000002
00000008 00000006 20000000 80000062
00000000 00000002 00000008 00000006
30000000 80000063 00000000 00000002
00000008 00000006 40000000 80000064
00000000 00000002 00000008 00000006
50000000 80000065 00000000 00000002
00000008 00000006 60000000 80000066
00000000 00000002 00000008 00000006
70000000 80000067 00000000 00000002
00000008 00000006 80000000 80000068
00000000 00000002 00000008 00000006
90000000 80000069 00000000 00000002
00000008 00000006 a0000000 8000006a
00000000 00000002 00000008 00000006
b0000000 8000006b 00000000 00000002
00000008 00000006 c0000000 8000006c
00000000 00000002 00000008 00000006
d0000000 8000006d 00000000 00000002
00000008 00000006 e0000000 8000006e
00000000 00000002 00000008 00000006
f0000000 8000006f 00000000 00000002
00000008 00000007 00000000 80000070
00000000 00000002 00000008 00000007
10000000 80000071 00000000 00000002
00000008 00000007 20000000 80000072
00000000 00000002 00000008 00000007
30000000 80000073 00000000 00000002
00000008 00000007 40000000 80000074
00000000 00000002 00000008 00000007
50000000 80000075 00000000 00000002
00000008 00000007 60000000 80000076
00000000 00000002 00000008 00000007
70000000 80000077 00000000 00000002
00000008 00000007 80000000 80000078
00000000 00000002 00000008 00000007
90000000 80000079 00000000 00000002
00000008 00000007 a0000000 8000007a
00000000 00000002 00000008 00000007
b0000000 8000007b 00000000 00000002
00000008 00000007 c0000000 8000007c
00000000 00000002 00000008 00000007
d0000000 8000007d 00000000 00000002
00000008 00000007 e0000000 8000007e
00000000 00000002 00000008 00000007
f0000000 8000007f 00000000 00000002
00000008 00000008 00000000 80000080
00000000 00000002 00000008 00000008
10000000 80000081 00000000 00000002
00000008 00000008 20000000 80000082
00000000 00000002 00000008 00000008
30000000 80000083 00000000 00000002
00000008 00000008 40000000 80000084
00000000 00000002 00000008 00000008
50000000 80000085 00000000 00000002
00000008 00000008 60000000 80000086
00000000 00000002 00000008 00000008
70000000 80000087 00000000 00000002
00000008 00000008 80000000 80000088
00000000 00000002 00000008 00000008
90000000 80000089 00000000 00000002
00000008 00000008 a0000000 8000008a
00000000 00000002 00000008 00000008
b0000000 8000008b 00000000 00000002
00000008 00000008 c0000000 8000008c
00000000 00000002 00000008 00000008
d0000000 8000008d 00000000 00000002
00000008 00000008 e0000000 8000008e
00000000 00000002 00000008 00000008
f0000000 8000008f 00000000 00000002
00000008 00000009 00000000 80000090
00000000 00000002 00000008 00000009
10000000 80000091 00000000 00000002
00000008 00000009 20000000 80000092
00000000 00000002 00000008 00000009
30000000 80000093 00000000 00000002
00000008 00000009 40000000 80000094
00000000 00000002 00000008 00000009
50000000 80000095 00000000 00000002
00000008 00000009 60000000 80000096
00000000 00000002 00000008 00000009
70000000 80000097 00000000 00000002
00000008 00000009 80000000 80000098
00000000 00000002 00000008 00000009
90000000 80000099 00000000 00000002
00000008 00000009 a0000000 8000009a
00000000 00000002 00000008 00000009
b0000000 8000009b 00000000 00000002
00000008 00000009 c0000000 8000009c
00000000 00000002 00000008 00000009
d0000000 8000009d 00000000 00000002
00000008 00000009 e0000000 8000009e
00000000 00000002 00000008 00000009
f0000000 8000009f 00000000 00000002
00000008 0000000a 00000000 800000a0
00000000 00000002 00000008 0000000a
10000000 800000a1 00000000 00000002
00000008 0000000a 20000000 800000a2
00000000 00000002 00000008 0000000a
30000000 800000a3 00000000 00000002
00000008 0000000a 40000000 800000a4
00000000 00000002 00000008 0000000a
50000000 800000a5 00000000 00000002
00000008 0000000a 60000000 800000a6
00000000 00000002 00000008 0000000a
70000000 800000a7 00000000 00000002
00000008 0000000a 80000000 800000a8
00000000 00000002 00000008 0000000a
90000000 800000a9 00000000 00000002
00000008 0000000a a0000000 800000aa
00000000 00000002 00000008 0000000a
b0000000 800000ab 00000000 00000002
00000008 0000000a c0000000 800000ac
00000000 00000002 00000008 0000000a
d0000000 800000ad 00000000 00000002
00000008 0000000a e0000000 800000ae
00000000 00000002 00000008 0000000a
f0000000 800000af 00000000 00000002
00000008 0000000b 00000000 800000b0
00000000 00000002 00000008 0000000b
10000000 800000b1 00000000 00000002
00000008 0000000b 20000000 800000b2
00000000 00000002 00000008 0000000b
30000000 800000b3 00000000 00000002
00000008 0000000b 40000000 800000b4
00000000 00000002 00000008 0000000b
50000000 800000b5 00000000 00000002
00000008 0000000b 60000000 800000b6
00000000 00000002 00000008 0000000b
70000000 800000b7 00000000 00000002
00000008 0000000b 80000000 800000b8
00000000 00000002 00000008 0000000b
90000000 800000b9 00000000 00000002
00000008 0000000b a0000000 800000ba
00000000 00000002 00000008 0000000b
b0000000 800000bb 00000000 00000002
00000008 0000000b c0000000 800000bc
00000000 00000002 00000008 0000000b
d0000000 800000bd 00000000 00000002
00000008 0000000b e0000000 800000be
00000000 00000002 00000008 0000000b
f0000000 800000bf 00000000 00000002
00000008 0000000c 00000000 800000c0
00000000 00000002 00000008 0000000c
10000000 800000c1 00000000 00000002
00000008 0000000c 20000000 800000c2
00000000 00000002 00000008 0000000c
30000000 800000c3 00000000 00000002
00000008 0000000c 40000000 800000c4
00000000 00000002 00000008 0000000c
50000000 800000c5 00000000 00000002
00000008 0000000c 60000000 800000c6
00000000 00000002 00000008 0000000c
70000000 800000c7 00000000 00000002
00000008 0000000c 80000000 800000c8
00000000 00000002 00000008 0000000c
90000000 800000c9 00000000 00000002
00000008 0000000c a0000000 800000ca
00000000 00000002 00000008 0000000c
b0000000 800000cb 00000000 00000002
00000008 0000000c c0000000 800000cc
00000000 00000002 00000008 0000000c
d0000000 800000cd 00000000 00000002
00000008 0000000c e0000000 800000ce
00000000 00000002 00000008 0000000c
f0000000 800000cf 00000000 00000002
00000008 0000000d 00000000 800000d0
00000000 00000002 00000008 0000000d
10000000 800000d1 00000000 00000002
00000008 0000000d 20000000 800000d2
00000000 00000002 00000008 0000000d
30000000 800000d3 00000000 00000002
00000008 0000000d 40000000 800000d4
00000000 00000002 00000008 0000000d
50000000 800000d5 00000000 00000002
00000008 0000000d 60000000 800000d6
00000000 00000002 00000008 0000000d
70000000 800000d7 00000000 00000002
00000008 0000000d 80000000 800000d8
00000000 00000002 00000008 0000000d
90000000 800000d9 00000000 00000002
00000008 0000000d a0000000 800000da
00000000 00000002 00000008 0000000d
b0000000 800000db 00000000 00000002
00000008 0000000d c0000000 800000dc
00000000 00000002 00000008 0000000d
d0000000 800000dd 00000000 00000002
00000008 0000000d e0000000 800000de
00000000 00000002 00000008 0000000d
f0000000 800000df 00000000 00000002
00000008 0000000e 00000000 800000e0
00000000 00000002 00000008 0000000e
10000000 800000e1 00000000 00000002
00000008 0000000e 20000000 800000e2
00000000 00000002 00000008 0000000e
30000000 800000e3 00000000 00000002
00000008 0000000e 40000000 800000e4
00000000 00000002 00000008 0000000e
50000000 800000e5 00000000 00000002
00000008 0000000e 60000000 800000e6
00000000 00000002 00000008 0000000e
70000000 800000e7 00000000 00000002
00000008 0000000e 80000000 800000e8
00000000 00000002 00000008 0000000e
90000000 800000e9 00000000 00000002
00000008 0000000e a0000000 800000ea
00000000 00000002 00000008 0000000e
b0000000 800000eb 00000000 00000002
00000008 0000000e c0000000 800000ec
00000000 00000002 00000008 0000000e
d0000000 800000ed 00000000 00000002
00000008 0000000e e0000000 800000ee
00000000 00000002 00000008 0000000e
f0000000 800000ef 00000000 00000002
00000008 0000000f 00000000 800000f0
00000000 00000002 00000008 0000000f
10000000 800000f1 00000000 00000002
00000008 0000000f 20000000 800000f2
00000000 00000002 00000008 0000000f
30000000 800000f3 00000000 00000002
00000008 0000000f 40000000 800000f4
00000000 00000002 00000008 0000000f
50000000 800000f5 00000000 00000002
00000008 0000000f 60000000 800000f6
00000000 00000002 00000008 0000000f
70000000 800000f7 00000000 00000002
00000008 0000000f 80000000 800000f8
00000000 00000002 00000008 0000000f
90000000 800000f9 00000000 00000002
00000008 0000000f a0000000 800000fa
00000000 00000002 00000008 0000000f
b0000000 800000fb 00000000 00000002
00000008 0000000f c0000000 800000fc
00000000 00000002 00000008 0000000f
d0000000 800000fd 00000000 00000002
00000008 0000000f e0000000 800000fe
00000000 00000002 00000008 0000000f
f0000000 800000ff 00000000 00000002
00000008 00000010 00000000 80000100
00000000 00000002 00000008 00000010
10000000 80000101 00000000 00000002
00000008 00000010 20000000 80000102
00000000 00000002 00000008 00000010
30000000 80000103 00000000 00000002
00000008 00000010 40000000 80000104
00000000 00000002 00000008 00000010
50000000 80000105 00000000 00000002
00000008 00000010 60000000 80000106
00000000 00000002 00000008 00000010
70000000 80000107 00000000 00000002
00000008 00000010 80000000 80000108
00000000 00000002 00000008 00000010
90000000 80000109 00000000 00000002
00000008 00000010 a0000000 8000010a
00000000 00000002 00000008 00000010
b0000000 8000010b 00000000 00000002
00000008 00000010 c0000000 8000010c
00000000 00000002 00000008 00000010
d0000000 8000010d 00000000 00000002
00000008 00000010 e0000000 8000010e
00000000 00000002 00000008 00000010
f0000000 8000010f 00000000 00000002
00000008 00000011 00000000 80000110
00000000 00000002 00000008 00000011
10000000 80000111 00000000 00000002
00000008 00000011 20000000 80000112
00000000 00000002 00000008 00000011
30000000 80000113 00000000 00000002
00000008 00000011 40000000 80000114
00000000 00000002 00000008 00000011
50000000 80000115 00000000 00000002
00000008 00000011 60000000 80000116
00000000 00000002 00000008 00000011
70000000 80000117 00000000 00000002
00000008 00000011 80000000 80000118
00000000 00000002 00000008 00000011
90000000 80000119 00000000 00000002
00000008 00000011 a0000000 8000011a
00000000 00000002 00000008 00000011
b0000000 8000011b 00000000 00000002
00000008 00000011 c0000000 8000011c
00000000 00000002 00000008 00000011
d0000000 8000011d 00000000 00000002
00000008 00000011 e0000000 8000011e
00000000 00000002 00000008 00000011
f0000000 8000011f 00000000 00000002
00000008 00000012 00000000 80000120
00000000 00000002 00000008 00000012
10000000 80000121 00000000 00000002
00000008 00000012 20000000 80000122
00000000 00000002 00000008 00000012
30000000 80000123 00000000 00000002
00000008 00000012 40000000 80000124
00000000 00000002 00000008 00000012
50000000 80000125 00000000 00000002
00000008 00000012 60000000 80000126
00000000 00000002 00000008 00000012
70000000 80000127 00000000 00000002
00000008 00000012 80000000 80000128
00000000 00000002 00000008 00000012
90000000 80000129 00000000 00000002
00000008 00000012 a0000000 8000012a
00000000 00000002 00000008 00000012
b0000000 8000012b 00000000 00000002
00000008 00000012 c0000000 8000012c
00000000 00000002 00000008 00000012
d0000000 8000012d 00000000 00000002
00000008 00000012 e0000000 8000012e
00000000 00000002 00000008 00000012
f0000000 8000012f 00000000 00000002
00000008 00000013 00000000 80000130
00000000 00000002 00000008 00000013
10000000 80000131 00000000 00000002
00000008 00000013 20000000 80000132
00000000 00000002 00000008 00000013
30000000 80000133 00000000 00000002
00000008 00000013 40000000 80000134
00000000 00000002 00000008 00000013
50000000 80000135 00000000 00000002
00000008 00000013 60000000 80000136
00000000 00000002 00000008 00000013
70000000 80000137 00000000 00000002
00000008 00000013 80000000 80000138
00000000 00000002 00000008 00000013
90000000 80000139 00000000 00000002
00000008 00000013 a0000000 8000013a
00000000 00000002 00000008 00000013
b0000000 8000013b 00000000 00000002
00000008 00000013 c0000000 8000013c
00000000 00000002 00000008 00000013
d0000000 8000013d 00000000 00000002
00000008 00000013 e0000000 8000013e
00000000 00000002 00000008 00000013
f0000000 8000013f 00000000 00000002
00000008 00000014 00000000 80000140
00000000 00000002 00000008 00000014
10000000 80000141 00000000 00000002
00000008 00000014 20000000 80000142
00000000 00000002 00000008 00000014
30000000 80000143 00000000 00000002
00000008 00000014 40000000 80000144
00000000 00000002 00000008 00000014
50000000 80000145 00000000 00000002
00000008 00000014 60000000 80000146
00000000 00000002 00000008 00000014
70000000 80000147 00000000 00000002
00000008 00000014 80000000 80000148
00000000 00000002 00000008 00000014
90000000 80000149 00000000 00000002
00000008 00000014 a0000000 8000014a
00000000 00000002 00000008 00000014
b0000000 8000014b 00000000 00000002
00000008 00000014 c0000000 8000014c
00000000 00000002 00000008 00000014
d0000000 8000014d 00000000 00000002
00000008 00000014 e0000000 8000014e
00000000 00000002 00000008 00000014
f0000000 8000014f 00000000 00000002
00000008 00000015 00000000 80000150
00000000 00000002 00000008 00000015
10000000 80000151 00000000 00000002
00000008 00000015 20000000 80000152
00000000 00000002 00000008 00000015
30000000 80000153 00000000 00000002
00000008 00000015 40000000 80000154
00000000 00000002 00000008 00000015
50000000 80000155 00000000 00000002
00000008 00000015 60000000 80000156
00000000 00000002 00000008 00000015
70000000 80000157 00000000 00000002
00000008 00000015 80000000 80000158
00000000 00000002 00000008 00000015
90000000 80000159 00000000 00000002
00000008 00000015 a0000000 8000015a
00000000 00000002 00000008 00000015
b0000000 8000015b 00000000 00000002
00000008 00000015 c0000000 8000015c
00000000 00000002 00000008 00000015
d0000000 8000015d 00000000 00000002
00000008 00000015 e0000000 8000015e
00000000 00000002 00000008 00000015
f0000000 8000015f 00000000 00000002
00000008 00000016 00000000 80000160
00000000 00000002 00000008 00000016
10000000 80000161 00000000 00000002
00000008 00000016 20000000 80000162
00000000 00000002 00000008 00000016
30000000 80000163 00000000 00000002
00000008 00000016 40000000 80000164
00000000 00000002 00000008 00000016
50000000 80000165 00000000 00000002
00000008 00000016 60000000 80000166
00000000 00000002 00000008 00000016
70000000 80000167 00000000 00000002
00000008 00000016 80000000 80000168
00000000 00000002 00000008 00000016
90000000 80000169 00000000 00000002
00000008 00000016 a0000000 8000016a
00000000 00000002 00000008 00000016
b0000000 8000016b 00000000 00000002
00000008 00000016 c0000000 8000016c
00000000 00000002 00000008 00000016
d0000000 8000016d 00000000 00000002
00000008 00000016 e0000000 8000016e
00000000 00000002 00000008 00000016
f0000000 8000016f 00000000 00000002
00000008 00000017 00000000 80000170
00000000 00000002 00000008 00000017
10000000 80000171 00000000 00000002
00000008 00000017 20000000 80000172
00000000 00000002 00000008 00000017
30000000 80000173 00000000 00000002
00000008 00000017 40000000 80000174
00000000 00000002 00000008 00000017
50000000 80000175 00000000 00000002
00000008 00000017 60000000 80000176
00000000 00000002 00000008 00000017
70000000 80000177 00000000 00000002
00000008 00000017 80000000 80000178
00000000 00000002 00000008 00000017
90000000 80000179 00000000 00000002
00000008 00000017 a0000000 8000017a
00000000 00000002 00000008 00000017
b0000000 8000017b 00000000 00000002
00000008 00000017 c0000000 8000017c
00000000 00000002 00000008 00000017
d0000000 8000017d 00000000 00000002
00000008 00000017 e0000000 8000017e
00000000 00000002 00000008 00000017
f0000000 8000017f 00000000 00000002
00000008 00000018 00000000 80000180
00000000 00000002 00000008 00000018
10000000 80000181 00000000 00000002
00000008 00000018 20000000 80000182
00000000 00000002 00000008 00000018
30000000 80000183 00000000 00000002
00000008 00000018 40000000 80000184
00000000 00000002 00000008 00000018
50000000 80000185 00000000 00000002
00000008 00000018 60000000 80000186
00000000 00000002 00000008 00000018
70000000 80000187 00000000 00000002
00000008 00000018 80000000 80000188
00000000 00000002 00000008 00000018
90000000 80000189 00000000 00000002
00000008 00000018 a0000000 8000018a
00000000 00000002 00000008 00000018
b0000000 8000018b 00000000 00000002
00000008 00000018 c0000000 8000018c
00000000 00000002 00000008 00000018
d0000000 8000018d 00000000 00000002
00000008 00000018 e0000000 8000018e
00000000 00000002 00000008 00000018
f0000000 8000018f 00000000 00000002
00000008 00000019 00000000 80000190
00000000 00000002 00000008 00000019
10000000 80000191 00000000 00000002
00000008 00000019 20000000 80000192
00000000 00000002 00000008 00000019
30000000 80000193 00000000 00000002
00000008 00000019 40000000 80000194
00000000 00000002 00000008 00000019
50000000 80000195 00000000 00000002
00000008 00000019 60000000 80000196
00000000 00000002 00000008 00000019
70000000 80000197 00000000 00000002
00000008 00000019 80000000 80000198
00000000 00000002 00000008 00000019
90000000 80000199 00000000 00000002
00000008 00000019 a0000000 8000019a
00000000 00000002 00000008 00000019
b0000000 8000019b 00000000 00000002
00000008 00000019 c0000000 8000019c
00000000 00000002 00000008 00000019
d0000000 8000019d 00000000 00000002
00000008 00000019 e0000000 8000019e
00000000 00000002 00000008 00000019
f0000000 8000019f 00000000 00000002
00000008 0000001a 00000000 800001a0
00000000 00000002 00000008 0000001a
10000000 800001a1 00000000 00000002
00000008 0000001a 20000000 800001a2
00000000 00000002 00000008 0000001a
30000000 800001a3 00000000 00000002
00000008 0000001a 40000000 800001a4
00000000 00000002 00000008 0000001a
50000000 800001a5 00000000 00000002
00000008 0000001a 60000000 800001a6
00000000 00000002 00000008 0000001a
70000000 800001a7 00000000 00000002
00000008 0000001a 80000000 800001a8
00000000 00000002 00000008 0000001a
90000000 800001a9 00000000 00000002
00000008 0000001a a0000000 800001aa
00000000 00000002 00000008 0000001a
b0000000 800001ab 00000000 00000002
00000008 0000001a c0000000 800001ac
00000000 00000002 00000008 0000001a
d0000000 800001ad 00000000 00000002
00000008 0000001a e0000000 800001ae
00000000 00000002 00000008 0000001a
f0000000 800001af 00000000 00000002
00000008 0000001b 00000000 800001b0
00000000 00000002 00000008 0000001b
10000000 800001b1 00000000 00000002
00000008 0000001b 20000000 800001b2
00000000 00000002 00000008 0000001b
30000000 800001b3 00000000 00000002
00000008 0000001b 40000000 800001b4
00000000 00000002 00000008 0000001b
50000000 800001b5 00000000 00000002
00000008 0000001b 60000000 800001b6
00000000 00000002 00000008 0000001b
70000000 800001b7 00000000 00000002
00000008 0000001b 80000000 800001b8
00000000 00000002 00000008 0000001b
90000000 800001b9 00000000 00000002
00000008 0000001b a0000000 800001ba
00000000 00000002 00000008 0000001b
b0000000 800001bb 00000000 00000002
00000008 0000001b c0000000 800001bc
00000000 00000002 00000008 0000001b
d0000000 800001bd 00000000 00000002
00000008 0000001b e0000000 800001be
00000000 00000002 00000008 0000001b
f0000000 800001bf 00000000 00000002
00000008 0000001c 00000000 800001c0
00000000 00000002 00000008 0000001c
10000000 800001c1 00000000 00000002
00000008 0000001c 20000000 800001c2
00000000 00000002 00000008 0000001c
30000000 800001c3 00000000 00000002
00000008 0000001c 40000000 800001c4
00000000 00000002 00000008 0000001c
50000000 800001c5 00000000 00000002
00000008 0000001c 60000000 800001c6
00000000 00000002 00000008 0000001c
70000000 800001c7 00000000 00000002
00000008 0000001c 80000000 800001c8
00000000 00000002 00000008 0000001c
90000000 800001c9 00000000 00000002
00000008 0000001c a0000000 800001ca
00000000 00000002 00000008 0000001c
b0000000 800001cb 00000000 00000002
00000008 0000001c c0000000 800001cc
00000000 00000002 00000008 0000001c
d0000000 800001cd 00000000 00000002
00000008 0000001c e0000000 800001ce
00000000 00000002 00000008 0000001c
f0000000 800001cf 00000000 00000002
00000008 0000001d 00000000 800001d0
00000000 00000002 00000008 0000001d
10000000 800001d1 00000000 00000002
00000008 0000001d 20000000 800001d2
00000000 00000002 00000008 0000001d
30000000 800001d3 00000000 00000002
00000008 0000001d 40000000 800001d4
00000000 00000002 00000008 0000001d
50000000 800001d5 00000000 00000002
00000008 0000001d 60000000 800001d6
00000000 00000002 00000008 0000001d
70000000 800001d7 00000000 00000002
00000008 0000001d 80000000 800001d8
00000000 00000002 00000008 0000001d
90000000 800001d9 00000000 00000002
00000008 0000001d a0000000 800001da
00000000 00000002 00000008 0000001d
b0000000 800001db 00000000 00000002
00000008 0000001d c0000000 800001dc
00000000 00000002 00000008 0000001d
d0000000 800001dd 00000000 00000002
00000008 0000001d e0000000 800001de
00000000 00000002 00000008 0000001d
f0000000 800001df 00000000 00000002
00000008 0000001e 00000000 800001e0
00000000 00000002 00000008 0000001e
10000000 800001e1 00000000 00000002
00000008 0000001e 20000000 800001e2
00000000 00000002 00000008 0000001e
30000000 800001e3 00000000 00000002
00000008 0000001e 40000000 800001e4
00000000 00000002 00000008 0000001e
50000000 800001e5 00000000 00000002
00000008 0000001e 60000000 800001e6
00000000 00000002 00000008 0000001e
70000000 800001e7 00000000 00000002
00000008 0000001e 80000000 800001e8
00000000 00000002 00000008 0000001e
90000000 800001e9 00000000 00000002
00000008 0000001e a0000000 800001ea
00000000 00000002 00000008 0000001e
b0000000 800001eb 00000000 00000002
00000008 0000001e c0000000 800001ec
00000000 00000002 00000008 0000001e
d0000000 800001ed 00000000 00000002
00000008 0000001e e0000000 800001ee
00000000 00000002 00000008 0000001e
f0000000 800001ef 00000000 00000002
00000008 0000001f 00000000 800001f0
00000000 00000002 00000008 0000001f
10000000 800001f1 00000000 00000002
00000008 0000001f 20000000 800001f2
00000000 00000002 00000008 0000001f
30000000 800001f3 00000000 00000002
00000008 0000001f 40000000 800001f4
00000000 00000002 00000008 0000001f
50000000 800001f5 00000000 00000002
00000008 0000001f 60000000 800001f6
00000000 00000002 00000008 0000001f
70000000 800001f7 00000000 00000002
00000008 0000001f 80000000 800001f8
00000000 00000002 00000008 0000001f
90000000 800001f9 00000000 00000002
00000008 0000001f a0000000 800001fa
00000000 00000002 00000008 0000001f
b0000000 800001fb 00000000 00000002
00000008 0000001f c0000000 800001fc
00000000 00000002 00000008 0000001f
d0000000 800001fd 00000000 00000002
00000008 0000001f e0000000 800001fe
00000000 00000002 00000008 0000001f
f0000000 800001ff 00000000 00000002
00000008 00000020 00000000 80000200
00000000 00000002 00000008 00000020
10000000 80000201 00000000 00000002
00000008 00000020 20000000 80000202
00000000 00000002 00000008 00000020
30000000 80000203 00000000 00000002
00000008 00000020 40000000 80000204
00000000 00000002 00000008 00000020
50000000 80000205 00000000 00000002
00000008 00000020 60000000 80000206
00000000 00000002 00000008 00000020
70000000 80000207 00000000 00000002
00000008 00000020 80000000 80000208
00000000 00000002 00000008 00000020
90000000 80000209 00000000 00000002
00000008 00000020 a0000000 8000020a
00000000 00000002 00000008 00000020
b0000000 8000020b 00000000 00000002
00000008 00000020 c0000000 8000020c
00000000 00000002 00000008 00000020
d0000000 8000020d 00000000 00000002
00000008 00000020 e0000000 8000020e
00000000 00000002 00000008 00000020
f0000000 8000020f 00000000 00000002
00000008 00000021 00000000 80000210
00000000 00000002 00000008 00000021
10000000 80000211 00000000 00000002
00000008 00000021 20000000 80000212
00000000 00000002 00000008 00000021
30000000 80000213 00000000 00000002
00000008 00000021 40000000 80000214
00000000 00000002 00000008 00000021
50000000 80000215 00000000 00000002
00000008 00000021 60000000 80000216
00000000 00000002 00000008 00000021
70000000 80000217 00000000 00000002
00000008 00000021 80000000 80000218
00000000 00000002 00000008 00000021
90000000 80000219 00000000 00000002
00000008 00000021 a0000000 8000021a
00000000 00000002 00000008 00000021
b0000000 8000021b 00000000 00000002
00000008 00000021 c0000000 8000021c
00000000 00000002 00000008 00000021
d0000000 8000021d 00000000 00000002
00000008 00000021 e0000000 8000021e
00000000 00000002 00000008 00000021
f0000000 8000021f 00000000 00000002
00000008 00000022 00000000 80000220
00000000 00000002 00000008 00000022
10000000 80000221 00000000 00000002
00000008 00000022 20000000 80000222
00000000 00000002 00000008 00000022
30000000 80000223 00000000 00000002
00000008 00000022 40000000 80000224
00000000 00000002 00000008 00000022
50000000 80000225 00000000 00000002
00000008 00000022 60000000 80000226
00000000 00000002 00000008 00000022
70000000 80000227 00000000 00000002
00000008 00000022 80000000 80000228
00000000 00000002 00000008 00000022
90000000 80000229 00000000 00000002
00000008 00000022 a0000000 8000022a
00000000 00000002 00000008 00000022
b0000000 8000022b 00000000 00000002
00000008 00000022 c0000000 8000022c
00000000 00000002 00000008 00000022
d0000000 8000022d 00000000 00000002
00000008 00000022 e0000000 8000022e
00000000 00000002 00000008 00000022
f0000000 8000022f 00000000 00000002
00000008 00000023 00000000 80000230
00000000 00000002 00000008 00000023
10000000 80000231 00000000 00000002
00000008 00000023 20000000 80000232
00000000 00000002 00000008 00000023
30000000 80000233 00000000 00000002
00000008 00000023 40000000 80000234
00000000 00000002 00000008 00000023
50000000 80000235 00000000 00000002
00000008 00000023 60000000 80000236
00000000 00000002 00000008 00000023
70000000 80000237 00000000 00000002
00000008 00000023 80000000 80000238
00000000 00000002 00000008 00000023
90000000 80000239 00000000 00000002
00000008 00000023 a0000000 8000023a
00000000 00000002 00000008 00000023
b0000000 8000023b 00000000 00000002
00000008 00000023 c0000000 8000023c
00000000 00000002 00000008 00000023
d0000000 8000023d 00000000 00000002
00000008 00000023 e0000000 8000023e
00000000 00000002 00000008 00000023
f0000000 8000023f 00000000 00000002
00000008 00000024 00000000 80000240
00000000 00000002 00000008 00000024
10000000 80000241 00000000 00000002
00000008 00000024 20000000 80000242
00000000 00000002 00000008 00000024
30000000 80000243 00000000 00000002
00000008 00000024 40000000 80000244
00000000 00000002 00000008 00000024
50000000 80000245 00000000 00000002
00000008 00000024 60000000 80000246
00000000 00000002 00000008 00000024
70000000 80000247 00000000 00000002
00000008 00000024 80000000 80000248
00000000 00000002 00000008 00000024
90000000 80000249 00000000 00000002
00000008 00000024 a0000000 8000024a
00000000 00000002 00000008 00000024
b0000000 8000024b 00000000 00000002
00000008 00000024 c0000000 8000024c
00000000 00000002 00000008 00000024
d0000000 8000024d 00000000 00000002
00000008 00000024 e0000000 8000024e
00000000 00000002 00000008 00000024
f0000000 8000024f 00000000 00000002
00000008 00000025 00000000 80000250
00000000 00000002 00000008 00000025
10000000 80000251 00000000 00000002
00000008 00000025 20000000 80000252
00000000 00000002 00000008 00000025
30000000 80000253 00000000 00000002
00000008 00000025 40000000 80000254
00000000 00000002 00000008 00000025
50000000 80000255 00000000 00000002
00000008 00000025 60000000 80000256
00000000 00000002 00000008 00000025
70000000 80000257 00000000 00000002
00000008 00000025 80000000 80000258
00000000 00000002 00000008 00000025
90000000 80000259 00000000 00000002
00000008 00000025 a0000000 8000025a
00000000 00000002 00000008 00000025
b0000000 8000025b 00000000 00000002
00000008 00000025 c0000000 8000025c
00000000 00000002 00000008 00000025
d0000000 8000025d 00000000 00000002
00000008 00000025 e0000000 8000025e
00000000 00000002 00000008 00000025
f0000000 8000025f 00000000 00000002
00000008 00000026 00000000 80000260
00000000 00000002 00000008 00000026
10000000 80000261 00000000 00000002
00000008 00000026 20000000 80000262
00000000 00000002 00000008 00000026
30000000 80000263 00000000 00000002
00000008 00000026 40000000 80000264
00000000 00000002 00000008 00000026
50000000 80000265 00000000 00000002
00000008 00000026 60000000 80000266
00000000 00000002 00000008 00000026
70000000 80000267 00000000 00000002
00000008 00000026 80000000 80000268
00000000 00000002 00000008 00000026
90000000 80000269 00000000 00000002
00000008 00000026 a0000000 8000026a
00000000 00000002 00000008 00000026
b0000000 8000026b 00000000 00000003
00000008 00000026 c0000000 8000026c
00000000 00000003 00000008 00000026
d0000000 8000026d 00000000 00000003
00000008 00000026 e0000000 8000026e
00000000 00000003 00000008 00000026
f0000000 8000026f 00000000 00000003
00000008 00000027 00000000 80000270
00000000 00000003 00000008 00000027
10000000 80000271 00000000 00000003
00000008 00000027 20000000 80000272
00000000 00000003 00000008 00000027
30000000 80000273 00000000 00000003
00000008 00000027 40000000 80000274
00000000 00000003 00000008 00000027
50000000 80000275 00000000 00000003
00000008 00000027 60000000 80000276
00000000 00000003 00000008 00000027
70000000 80000277 00000000 00000003
00000008 00000027 80000000 80000278
00000000 00000003 00000008 00000027
90000000 80000279 00000000 00000003
00000008 00000027 a0000000 8000027a
00000000 00000003 00000008 00000027
b0000000 8000027b 00000000 00000003
00000008 00000027 c0000000 8000027c
00000000 00000003 00000008 00000027
d0000000 8000027d 00000000 00000003
00000008 00000027 e0000000 8000027e
00000000 00000003 00000008 00000027
f0000000 8000027f 00000000 00000003
00000008 00000028 00000000 80000280
00000000 00000003 00000008 00000028
10000000 80000281 00000000 00000003
00000008 00000028 20000000 80000282
00000000 00000003 00000008 00000028
30000000 80000283 00000000 00000003
00000008 00000028 40000000 80000284
00000000 00000003 00000008 00000028
50000000 80000285 00000000 00000003
00000008 00000028 60000000 80000286
00000000 00000003 00000008 00000028
70000000 80000287 00000000 00000003
00000008 00000028 80000000 80000288
00000000 00000003 00000008 00000028
90000000 80000289 00000000 00000003
00000008 00000028 a0000000 8000028a
00000000 00000003 00000008 00000028
b0000000 8000028b 00000000 00000003
00000008 00000028 c0000000 8000028c
00000000 00000003 00000008 00000028
d0000000 8000028d 00000000 00000003
00000008 00000028 e0000000 8000028e
00000000 00000003 00000008 00000028
f0000000 8000028f 00000000 00000003
00000008 00000029 00000000 80000290
00000000 00000003 00000008 00000029
10000000 80000291 00000000 00000003
00000008 00000029 20000000 80000292
00000000 00000003 00000008 00000029
30000000 80000293 00000000 00000003
00000008 00000029 40000000 80000294
00000000 00000003 00000008 00000029
50000000 80000295 00000000 00000003
00000008 00000029 60000000 80000296
00000000 00000003 00000008 00000029
70000000 80000297 00000000 00000003
00000008 00000029 80000000 80000298
00000000 00000003 00000008 00000029
90000000 80000299 00000000 00000003
00000008 00000029 a0000000 8000029a
00000000 00000003 00000008 00000029
b0000000 8000029b 00000000 00000003
00000008 00000029 c0000000 8000029c
00000000 00000003 00000008 00000029
d0000000 8000029d 00000000 00000003
00000008 00000029 e0000000 8000029e
00000000 00000003 00000008 00000029
f0000000 8000029f 00000000 00000003
00000008 0000002a 00000000 800002a0
00000000 00000003 00000008 0000002a
10000000 800002a1 00000000 00000003
00000008 0000002a 20000000 800002a2
00000000 00000003 00000008 0000002a
30000000 800002a3 00000000 00000003
00000008 0000002a 40000000 800002a4
00000000 00000003 00000008 0000002a
50000000 800002a5 00000000 00000003
00000008 0000002a 60000000 800002a6
00000000 00000003 00000008 0000002a
70000000 800002a7 00000000 00000003
00000008 0000002a 80000000 800002a8
00000000 00000003 00000008 0000002a
90000000 800002a9 00000000 00000003
00000008 0000002a a0000000 800002aa
00000000 00000003 00000008 0000002a
b0000000 800002ab 00000000 00000003
00000008 0000002a c0000000 800002ac
00000000 00000003 00000008 0000002a
d0000000 800002ad 00000000 00000003
00000008 0000002a e0000000 800002ae
00000000 00000003 00000008 0000002a
f0000000 800002af 00000000 00000003
00000008 0000002b 00000000 800002b0
00000000 00000003 00000008 0000002b
10000000 800002b1 00000000 00000003
00000008 0000002b 20000000 800002b2
00000000 00000003 00000008 0000002b
30000000 800002b3 00000000 00000003
00000008 0000002b 40000000 800002b4
00000000 00000003 00000008 0000002b
50000000 800002b5 00000000 00000003
00000008 0000002b 60000000 800002b6
00000000 00000003 00000008 0000002b
70000000 800002b7 00000000 00000003
00000008 0000002b 80000000 800002b8
00000000 00000003 00000008 0000002b
90000000 800002b9 00000000 00000003
00000008 0000002b a0000000 800002ba
00000000 00000003 00000008 0000002b
b0000000 800002bb 00000000 00000003
00000008 0000002b c0000000 800002bc
00000000 00000003 00000008 0000002b
d0000000 800002bd 00000000 00000003
00000008 0000002b e0000000 800002be
00000000 00000003 00000008 0000002b
f0000000 800002bf 00000000 00000003
00000008 0000002c 00000000 800002c0
00000000 00000003 00000008 0000002c
10000000 800002c1 00000000 00000003
00000008 0000002c 20000000 800002c2
00000000 00000003 00000008 0000002c
30000000 800002c3 00000000 00000003
00000008 0000002c 40000000 800002c4
00000000 00000003 00000008 0000002c
50000000 800002c5 00000000 00000003
00000008 0000002c 60000000 800002c6
00000000 00000003 00000008 0000002c
70000000 800002c7 00000000 00000003
00000008 0000002c 80000000 800002c8
00000000 00000003 00000008 0000002c
90000000 800002c9 00000000 00000003
00000008 0000002c a0000000 800002ca
00000000 00000003 00000008 0000002c
b0000000 800002cb 00000000 00000003
00000008 0000002c c0000000 800002cc
00000000 00000003 00000008 0000002c
d0000000 800002cd 00000000 00000003
00000008 0000002c e0000000 800002ce
00000000 00000003 00000008 0000002c
f0000000 800002cf 00000000 00000003
00000008 0000002d 00000000 800002d0
00000000 00000003 00000008 0000002d
10000000 800002d1 00000000 00000003
00000008 0000002d 20000000 800002d2
00000000 00000003 00000008 0000002d
30000000 800002d3 00000000 00000003
00000008 0000002d 40000000 800002d4
00000000 00000003 00000008 0000002d
50000000 800002d5 00000000 00000003
00000008 0000002d 60000000 800002d6
00000000 00000003 00000008 0000002d
70000000 800002d7 00000000 00000003
00000008 0000002d 80000000 800002d8
00000000 00000003 00000008 0000002d
90000000 800002d9 00000000 00000003
00000008 0000002d a0000000 800002da
00000000 00000003 00000008 0000002d
b0000000 800002db 00000000 00000003
00000008 0000002d c0000000 800002dc
00000000 00000003 00000008 0000002d
d0000000 800002dd 00000000 00000003
00000008 0000002d e0000000 800002de
00000000 00000003 00000008 0000002d
f0000000 800002df 00000000 00000003
00000008 0000002e 00000000 800002e0
00000000 00000003 00000008 0000002e
10000000 800002e1 00000000 00000003
00000008 0000002e 20000000 800002e2
00000000 00000003 00000008 0000002e
30000000 800002e3 00000000 00000003
00000008 0000002e 40000000 800002e4
00000000 00000003 00000008 0000002e
50000000 800002e5 00000000 00000003
00000008 0000002e 60000000 800002e6
00000000 00000003 00000008 0000002e
70000000 800002e7 00000000 00000003
00000008 0000002e 80000000 800002e8
00000000 00000003 00000008 0000002e
90000000 800002e9 00000000 00000003
00000008 0000002e a0000000 800002ea
00000000 00000003 00000008 0000002e
b0000000 800002eb 00000000 00000003
00000008 0000002e c0000000 800002ec
00000000 00000003 00000008 0000002e
d0000000 800002ed 00000000 00000003
00000008 0000002e e0000000 800002ee
00000000 00000003 00000008 0000002e
f0000000 800002ef 00000000 00000003
00000008 0000002f 00000000 800002f0
00000000 00000003 00000008 0000002f
10000000 800002f1 00000000 00000003
00000008 0000002f 20000000 800002f2
00000000 00000003 00000008 0000002f
30000000 800002f3 00000000 00000003
00000008 0000002f 40000000 800002f4
00000000 00000003 00000008 0000002f
50000000 800002f5 00000000 00000003
00000008 0000002f 60000000 800002f6
00000000 00000003 00000008 0000002f
70000000 800002f7 00000000 00000003
00000008 0000002f 80000000 800002f8
00000000 00000003 00000008 0000002f
90000000 800002f9 00000000 00000003
00000008 0000002f a0000000 800002fa
00000000 00000003 00000008 0000002f
b0000000 800002fb 00000000 00000003
00000008 0000002f c0000000 800002fc
00000000 00000003 00000008 0000002f
d0000000 800002fd 00000000 00000003
00000008 0000002f e0000000 800002fe
00000000 00000003 00000008 0000002f
f0000000 800002ff 00000000 00000003
00000008 00000030 00000000 80000300
00000000 00000003 00000008 00000030
10000000 80000301 00000000 00000003
00000008 00000030 20000000 80000302
00000000 00000003 00000008 00000030
30000000 80000303 00000000 00000003
00000008 00000030 40000000 80000304
00000000 00000003 00000008 00000030
50000000 80000305 00000000 00000003
00000008 00000030 60000000 80000306
00000000 00000003 00000008 00000030
70000000 80000307 00000000 00000003
00000008 00000030 80000000 80000308
00000000 00000003 00000008 00000030
90000000 80000309 00000000 00000003
00000008 00000030 a0000000 8000030a
00000000 00000003 00000008 00000030
b0000000 8000030b 00000000 00000003
00000008 00000030 c0000000 8000030c
00000000 00000003 00000008 00000030
d0000000 8000030d 00000000 00000003
00000008 00000030 e0000000 8000030e
00000000 00000003 00000008 00000030
f0000000 8000030f 00000000 00000003
00000008 00000031 00000000 80000310
00000000 00000003 00000008 00000031
10000000 80000311 00000000 00000003
00000008 00000031 20000000 80000312
00000000 00000003 00000008 00000031
30000000 80000313 00000000 00000003
00000008 00000031 40000000 80000314
00000000 00000003 00000008 00000031
50000000 80000315 00000000 00000003
00000008 00000031 60000000 80000316
00000000 00000003 00000008 00000031
70000000 80000317 00000000 00000003
00000008 00000031 80000000 80000318
00000000 00000003 00000008 00000031
90000000 80000319 00000000 00000003
00000008 00000031 a0000000 8000031a
00000000 00000003 00000008 00000031
b0000000 8000031b 00000000 00000003
00000008 00000031 c0000000 8000031c
00000000 00000003 00000008 00000031
d0000000 8000031d 00000000 00000003
00000008 00000031 e0000000 8000031e
00000000 00000003 00000008 00000031
f0000000 8000031f 00000000 00000003
00000008 00000032 00000000 80000320
00000000 00000003 00000008 00000032
10000000 80000321 00000000 00000003
00000008 00000032 20000000 80000322
00000000 00000003 00000008 00000032
30000000 80000323 00000000 00000003
00000008 00000032 40000000 80000324
00000000 00000003 00000008 00000032
50000000 80000325 00000000 00000003
00000008 00000032 60000000 80000326
00000000 00000003 00000008 00000032
70000000 80000327 00000000 00000003
00000008 00000032 80000000 80000328
00000000 00000003 00000008 00000032
90000000 80000329 00000000 00000003
00000008 00000032 a0000000 8000032a
00000000 00000003 00000008 00000032
b0000000 8000032b 00000000 00000003
00000008 00000032 c0000000 8000032c
00000000 00000003 00000008 00000032
d0000000 8000032d 00000000 00000003
00000008 00000032 e0000000 8000032e
00000000 00000003 00000008 00000032
f0000000 8000032f 00000000 00000003
00000008 00000033 00000000 80000330
00000000 00000003 00000008 00000033
10000000 80000331 00000000 00000003
00000008 00000033 20000000 80000332
00000000 00000003 00000008 00000033
30000000 80000333 00000000 00000003
00000008 00000033 40000000 80000334
00000000 00000003 00000008 00000033
50000000 80000335 00000000 00000003
00000008 00000033 60000000 80000336
00000000 00000003 00000008 00000033
70000000 80000337 00000000 00000003
00000008 00000033 80000000 80000338
00000000 00000003 00000008 00000033
90000000 80000339 00000000 00000003
00000008 00000033 a0000000 8000033a
00000000 00000003 00000008 00000033
b0000000 8000033b 00000000 00000003
00000008 00000033 c0000000 8000033c
00000000 00000003 00000008 00000033
d0000000 8000033d 00000000 00000003
00000008 00000033 e0000000 8000033e
00000000 00000003 00000008 00000033
f0000000 8000033f 00000000 00000003
00000008 00000034 00000000 80000340
00000000 00000003 00000008 00000034
10000000 80000341 00000000 00000003
00000008 00000034 20000000 80000342
00000000 00000003 00000008 00000034
30000000 80000343 00000000 00000003
00000008 00000034 40000000 80000344
00000000 00000003 00000008 00000034
50000000 80000345 00000000 00000003
00000008 00000034 60000000 80000346
00000000 00000003 00000008 00000034
70000000 80000347 00000000 00000003
00000008 00000034 80000000 80000348
00000000 00000003 00000008 00000034
90000000 80000349 00000000 00000003
00000008 00000034 a0000000 8000034a
00000000 00000003 00000008 00000034
b0000000 8000034b 00000000 00000003
00000008 00000034 c0000000 8000034c
00000000 00000003 00000008 00000034
d0000000 8000034d 00000000 00000003
00000008 00000034 e0000000 8000034e
00000000 00000003 00000008 00000034
f0000000 8000034f 00000000 00000003
00000008 00000035 00000000 80000350
00000000 00000003 00000008 00000035
10000000 80000351 00000000 00000003
00000008 00000035 20000000 80000352
00000000 00000003 00000008 00000035
30000000 80000353 00000000 00000003
00000008 00000035 40000000 80000354
00000000 00000003 00000008 00000035
50000000 80000355 00000000 00000003
00000008 00000035 60000000 80000356
00000000 00000003 00000008 00000035
70000000 80000357 00000000 00000003
00000008 00000035 80000000 80000358
00000000 00000003 00000008 00000035
90000000 80000359 00000000 00000003
00000008 00000035 a0000000 8000035a
00000000 00000003 00000008 00000035
b0000000 8000035b 00000000 00000003
00000008 00000035 c0000000 8000035c
00000000 00000003 00000008 00000035
d0000000 8000035d 00000000 00000003
00000008 00000035 e0000000 8000035e
00000000 00000003 00000008 00000035
f0000000 8000035f 00000000 00000003
00000008 00000036 00000000 80000360
00000000 00000003 00000008 00000036
10000000 80000361 00000000 00000003
00000008 00000036 20000000 80000362
00000000 00000003 00000008 00000036
30000000 80000363 00000000 00000003
00000008 00000036 40000000 80000364
00000000 00000003 00000008 00000036
50000000 80000365 00000000 00000003
00000008 00000036 60000000 80000366
00000000 00000003 00000008 00000036
70000000 80000367 00000000 00000003
00000008 00000036 80000000 80000368
00000000 00000003 00000008 00000036
90000000 80000369 00000000 00000003
00000008 00000036 a0000000 8000036a
00000000 00000003 00000008 00000036
b0000000 8000036b 00000000 00000003
00000008 00000036 c0000000 8000036c
00000000 00000003 00000008 00000036
d0000000 8000036d 00000000 00000003
00000008 00000036 e0000000 8000036e
00000000 00000003 00000008 00000036
f0000000 8000036f 00000000 00000003
00000008 00000037 00000000 80000370
00000000 00000003 00000008 00000037
10000000 80000371 00000000 00000003
00000008 00000037 20000000 80000372
00000000 00000003 00000008 00000037
30000000 80000373 00000000 00000003
00000008 00000037 40000000 80000374
00000000 00000003 00000008 00000037
50000000 80000375 00000000 00000003
00000008 00000037 60000000 80000376
00000000 00000003 00000008 00000037
70000000 80000377 00000000 00000003
00000008 00000037 80000000 80000378
00000000 00000003 00000008 00000037
90000000 80000379 00000000 00000003
00000008 00000037 a0000000 8000037a
00000000 00000003 00000008 00000037
b0000000 8000037b 00000000 00000003
00000008 00000037 c0000000 8000037c
00000000 00000003 00000008 00000037
d0000000 8000037d 00000000 00000003
00000008 00000037 e0000000 8000037e
00000000 00000003 00000008 00000037
f0000000 8000037f 00000000 00000003
00000008 00000038 00000000 80000380
00000000 00000003 00000008 00000038
10000000 80000381 00000000 00000003
00000008 00000038 20000000 80000382
00000000 00000003 00000008 00000038
30000000 80000383 00000000 00000003
00000008 00000038 40000000 80000384
00000000 00000003 00000008 00000038
50000000 80000385 00000000 00000003
00000008 00000038 60000000 80000386
00000000 00000003 00000008 00000038
70000000 80000387 00000000 00000003
00000008 00000038 80000000 80000388
00000000 00000003 00000008 00000038
90000000 80000389 00000000 00000003
00000008 00000038 a0000000 8000038a
00000000 00000003 00000008 00000038
b0000000 8000038b 00000000 00000003
00000008 00000038 c0000000 8000038c
00000000 00000003 00000008 00000038
d0000000 8000038d 00000000 00000003
00000008 00000038 e0000000 8000038e
00000000 00000003 00000008 00000038
f0000000 8000038f 00000000 00000003
00000008 00000039 00000000 80000390
00000000 00000003 00000008 00000039
10000000 80000391 00000000 00000003
00000008 00000039 20000000 80000392
00000000 00000003 00000008 00000039
30000000 80000393 00000000 00000003
00000008 00000039 40000000 80000394
00000000 00000003 00000008 00000039
50000000 80000395 00000000 00000003
00000008 00000039 60000000 80000396
00000000 00000003 00000008 00000039
70000000 80000397 00000000 00000003
00000008 00000039 80000000 80000398
00000000 00000003 00000008 00000039
90000000 80000399 00000000 00000003
00000008 00000039 a0000000 8000039a
00000000 00000003 00000008 00000039
b0000000 8000039b 00000000 00000003
00000008 00000039 c0000000 8000039c
00000000 00000003 00000008 00000039
d0000000 8000039d 00000000 00000003
00000008 00000039 e0000000 8000039e
00000000 00000003 00000008 00000039
f0000000 8000039f 00000000 00000003
00000008 0000003a 00000000 800003a0
00000000 00000003 00000008 0000003a
10000000 800003a1 00000000 00000003
00000008 0000003a 20000000 800003a2
00000000 00000003 00000008 0000003a
30000000 800003a3 00000000 00000003
00000008 0000003a 40000000 800003a4
00000000 00000003 00000008 0000003a
50000000 800003a5 00000000 00000003
00000008 0000003a 60000000 800003a6
00000000 00000003 00000008 0000003a
70000000 800003a7 00000000 00000003
00000008 0000003a 80000000 800003a8
00000000 00000003 00000008 0000003a
90000000 800003a9 00000000 00000003
00000008 0000003a a0000000 800003aa
00000000 00000003 00000008 0000003a
b0000000 800003ab 00000000 00000003
00000008 0000003a c0000000 800003ac
00000000 00000003 00000008 0000003a
d0000000 800003ad 00000000 00000003
00000008 0000003a e0000000 800003ae
00000000 00000003 00000008 0000003a
f0000000 800003af 00000000 00000003
00000008 0000003b 00000000 800003b0
00000000 00000003 00000008 0000003b
10000000 800003b1 00000000 00000003
00000008 0000003b 20000000 800003b2
00000000 00000003 00000008 0000003b
30000000 800003b3 00000000 00000003
00000008 0000003b 40000000 800003b4
00000000 00000003 00000008 0000003b
50000000 800003b5 00000000 00000003
00000008 0000003b 60000000 800003b6
00000000 00000003 00000008 0000003b
70000000 800003b7 00000000 00000003
00000008 0000003b 80000000 800003b8
00000000 00000003 00000008 0000003b
90000000 800003b9 00000000 00000003
00000008 0000003b a0000000 800003ba
00000000 00000003 00000008 0000003b
b0000000 800003bb 00000000 00000003
00000008 0000003b c0000000 800003bc
00000000 00000003 00000008 0000003b
d0000000 800003bd 00000000 00000003
00000008 0000003b e0000000 800003be
00000000 00000003 00000008 0000003b
f0000000 800003bf 00000000 00000003
00000008 0000003c 00000000 800003c0
00000000 ffffffff 00000000 0000003c
10000000 800003c1 00000000 ffffffff
00000000 0000003c 20000000 800003c2
00000000 ffffffff 00000000 0000003c
30000000 800003c3 00000000 ffffffff
00000000 0000003c 40000000 800003c4
00000000 ffffffff 00000000 0000003c
50000000 800003c5 00000000 ffffffff
00000000 0000003c 60000000 800003c6
00000000 ffffffff 00000000 0000003c
70000000 800003c7 00000000 ffffffff
00000000 0000003c 80000000 800003c8
00000000 ffffffff 00000000 0000003c
90000000 800003c9 00000000 ffffffff
00000000 0000003c a0000000 800003ca
00000000 ffffffff 00000000 0000003c
b0000000 800003cb 00000000 ffffffff
00000000 0000003c c0000000 800003cc
00000000 ffffffff 00000000 0000003c
d0000000 800003cd 00000000 ffffffff
00000000 0000003c e0000000 800003ce
00000000 ffffffff 00000000 0000003c
f0000000 800003cf 00000000 ffffffff
00000000 0000003d 00000000 800003d0
00000000 ffffffff 00000000 0000003d
10000000 800003d1 00000000 ffffffff
00000000 0000003d 20000000 800003d2
00000000 ffffffff 00000000 0000003d
30000000 800003d3 00000000 ffffffff
00000000 0000003d 40000000 800003d4
00000000 ffffffff 00000000 0000003d
50000000 800003d5 00000000 ffffffff
00000000 0000003d 60000000 800003d6
00000000 ffffffff 00000000 0000003d
70000000 800003d7 00000000 ffffffff
00000000 0000003d 80000000 800003d8
00000000 ffffffff 00000000 0000003d
90000000 800003d9 00000000 ffffffff
00000000 0000003d a0000000 800003da
00000000 ffffffff 00000000 0000003d
b0000000 800003db 00000000 ffffffff
00000000 0000003d c0000000 800003dc
00000000 ffffffff 00000000 0000003d
d0000000 800003dd 00000000 ffffffff
00000000 0000003d e0000000 800003de
00000000 ffffffff 00000000 0000003d
f0000000 800003df 00000000 ffffffff
00000000 0000003e 00000000 800003e0
00000000 ffffffff 00000000 0000003e
10000000 800003e1 00000000 ffffffff
00000000 0000003e 20000000 800003e2
00000000 ffffffff 00000000 0000003e
30000000 800003e3 00000000 ffffffff
00000000 0000003e 40000000 800003e4
00000000 ffffffff 00000000 0000003e
50000000 800003e5 00000000 ffffffff
00000000 0000003e 60000000 800003e6
00000000 ffffffff 00000000 0000003e
70000000 800003e7 00000000 ffffffff
00000000 0000003e 80000000 800003e8
00000000 ffffffff 00000000 0000003e
90000000 800003e9 00000000 ffffffff
00000000 0000003e a0000000 800003ea
00000000 ffffffff 00000000 0000003e
b0000000 800003eb 00000000 ffffffff
00000000 0000003e c0000000 800003ec
00000000 ffffffff 00000000 0000003e
d0000000 800003ed 00000000 ffffffff
00000000 0000003e e0000000 800003ee
00000000 ffffffff 00000000 0000003e
f0000000 800003ef 00000000 ffffffff
00000000 0000003f 00000000 800003f0
00000000 ffffffff 00000000 0000003f
10000000 800003f1 00000000 ffffffff
00000000 0000003f 20000000 800003f2
00000000 ffffffff 00000000 0000003f
30000000 800003f3 00000000 ffffffff
00000000 0000003f 40000000 800003f4
00000000 ffffffff 00000000 0000003f
50000000 800003f5 00000000 ffffffff
00000000 0000003f 60000000 800003f6
00000000 ffffffff 00000000 0000003f
70000000 800003f7 00000000 ffffffff
00000000 0000003f 80000000 800003f8
00000000 ffffffff 00000000 0000003f
90000000 800003f9 00000000 ffffffff
00000000 0000003f a0000000 800003fa
00000000 ffffffff 00000000 0000003f
b0000000 800003fb 00000000 ffffffff
00000000 0000003f c0000000 800003fc
00000000 ffffffff 00000000 0000003f
d0000000 800003fd 00000000 ffffffff
00000000 0000003f e0000000 800003fe
00000000 ffffffff 00000000 0000003f
f0000000 800003ff 00000000 ffffffff
00000000 00000040 00000000 80000400
00000000 ffffffff 00000000 00000040
10000000 80000401 00000000 ffffffff
00000000 00000040 20000000 80000402
00000000 ffffffff 00000000 00000040
30000000 80000403 00000000 ffffffff
00000000 00000040 40000000 80000404
00000000 ffffffff 00000000 00000040
50000000 80000405 00000000 ffffffff
00000000 00000040 60000000 80000406
00000000 ffffffff 00000000 00000040
70000000 80000407 00000000 ffffffff
00000000 00000040 80000000 80000408
00000000 ffffffff 00000000 00000040
90000000 80000409 00000000 ffffffff
00000000 00000040 a0000000 8000040a
00000000 ffffffff 00000000 00000040
b0000000 8000040b 00000000 ffffffff
00000000 00000040 c0000000 8000040c
00000000 ffffffff 00000000 00000040
d0000000 8000040d 00000000 ffffffff
00000000 00000040 e0000000 8000040e
00000000 ffffffff 00000000 00000040
f0000000 8000040f 00000000 ffffffff
00000000 00000041 00000000 80000410
00000000 ffffffff 00000000 00000041
10000000 80000411 00000000 ffffffff
00000000 00000041 20000000 80000412
00000000 ffffffff 00000000 00000041
30000000 80000413 00000000 ffffffff
00000000 00000041 40000000 80000414
00000000 ffffffff 00000000 00000041
50000000 80000415 00000000 ffffffff
00000000 00000041 60000000 80000416
00000000 ffffffff 00000000 00000041
70000000 80000417 00000000 ffffffff
00000000 00000041 80000000 80000418
00000000 ffffffff 00000000 00000041
90000000 80000419 00000000 ffffffff
00000000 00000041 a0000000 8000041a
00000000 ffffffff 00000000 00000041
b0000000 8000041b 00000000 ffffffff
00000000 00000041 c0000000 8000041c
00000000 ffffffff 00000000 00000041
d0000000 8000041d 00000000 ffffffff
00000000 00000041 e0000000 8000041e
00000000 ffffffff 00000000 00000041
f0000000 8000041f 00000000 ffffffff
00000000 00000042 00000000 80000420
00000000 ffffffff 00000000 00000042
10000000 80000421 00000000 ffffffff
00000000 00000042 20000000 80000422
00000000 ffffffff 00000000 00000042
30000000 80000423 00000000 ffffffff
00000000 00000042 40000000 80000424
00000000 ffffffff 00000000 00000042
50000000 80000425 00000000 ffffffff
00000000 00000042 60000000 80000426
00000000 ffffffff 00000000 00000042
70000000 80000427 00000000 ffffffff
00000000 00000042 80000000 80000428
00000000 ffffffff 00000000 00000042
90000000 80000429 00000000 ffffffff
00000000 00000042 a0000000 8000042a
00000000 ffffffff 00000000 00000042
b0000000 8000042b 00000000 ffffffff
00000000 00000042 c0000000 8000042c
00000000 ffffffff 00000000 00000042
d0000000 8000042d 00000000 ffffffff
00000000 00000042 e0000000 8000042e
00000000 ffffffff 00000000 00000042
f0000000 8000042f 00000000 ffffffff
00000000 00000043 00000000 80000430
00000000 ffffffff 00000000 00000043
10000000 80000431 00000000 ffffffff
00000000 00000043 20000000 80000432
00000000 ffffffff 00000000 00000043
30000000 80000433 00000000 ffffffff
00000000 00000043 40000000 80000434
00000000 ffffffff 00000000 00000043
50000000 80000435 00000000 ffffffff
00000000 00000043 60000000 80000436
00000000 ffffffff 00000000 00000043
70000000 80000437 00000000 ffffffff
00000000 00000043 80000000 80000438
00000000 ffffffff 00000000 00000043
90000000 80000439 00000000 ffffffff
00000000 00000043 a0000000 8000043a
00000000 ffffffff 00000000 00000043
b0000000 8000043b 00000000 ffffffff
00000000 00000043 c0000000 8000043c
00000000 ffffffff 00000000 00000043
d0000000 8000043d 00000000 ffffffff
00000000 00000043 e0000000 8000043e
00000000 ffffffff 00000000 00000043
f0000000 8000043f 00000000 ffffffff
00000000 00000044 00000000 80000440
00000000 ffffffff 00000000 00000044
10000000 80000441 00000000 ffffffff
00000000 00000044 20000000 80000442
00000000 ffffffff 00000000 00000044
30000000 80000443 00000000 ffffffff
00000000 00000044 40000000 80000444
00000000 ffffffff 00000000 00000044
50000000 80000445 00000000 ffffffff
00000000 00000044 60000000 80000446
00000000 ffffffff 00000000 00000044
70000000 80000447 00000000 ffffffff
00000000 00000044 80000000 80000448
00000000 ffffffff 00000000 00000044
90000000 80000449 00000000 ffffffff
00000000 00000044 a0000000 8000044a
00000000 ffffffff 00000000 00000044
b0000000 8000044b 00000000 ffffffff
00000000 00000044 c0000000 8000044c
00000000 ffffffff 00000000 00000044
d0000000 8000044d 00000000 ffffffff
00000000 00000044 e0000000 8000044e
00000000 ffffffff 00000000 00000044
f0000000 8000044f 00000000 ffffffff
00000000 00000045 00000000 80000450
00000000 ffffffff 00000000 00000045
10000000 80000451 00000000 ffffffff
00000000 00000045 20000000 80000452
00000000 ffffffff 00000000 00000045
30000000 80000453 00000000 ffffffff
00000000 00000045 40000000 80000454
00000000 ffffffff 00000000 00000045
50000000 80000455 00000000 ffffffff
00000000 00000045 60000000 80000456
00000000 ffffffff 00000000 00000045
70000000 80000457 00000000 ffffffff
00000000 00000045 80000000 80000458
00000000 ffffffff 00000000 00000045
90000000 80000459 00000000 ffffffff
00000000 00000045 a0000000 8000045a
00000000 ffffffff 00000000 00000045
b0000000 8000045b 00000000 ffffffff
00000000 00000045 c0000000 8000045c
00000000 ffffffff 00000000 00000045
d0000000 8000045d 00000000 ffffffff
00000000 00000045 e0000000 8000045e
00000000 ffffffff 00000000 00000045
f0000000 8000045f 00000000 ffffffff
00000000 00000046 00000000 80000460
00000000 ffffffff 00000000 00000046
10000000 80000461 00000000 ffffffff
00000000 00000046 20000000 80000462
00000000 ffffffff 00000000 00000046
30000000 80000463 00000000 ffffffff
00000000 00000046 40000000 80000464
00000000 ffffffff 00000000 00000046
50000000 80000465 00000000 ffffffff
00000000 00000046 60000000 80000466
00000000 ffffffff 00000000 00000046
70000000 80000467 00000000 ffffffff
00000000 00000046 80000000 80000468
00000000 ffffffff 00000000 00000046
90000000 80000469 00000000 ffffffff
00000000 00000046 a0000000 8000046a
00000000 ffffffff 00000000 00000046
b0000000 8000046b 00000000 ffffffff
00000000 00000046 c0000000 8000046c
00000000 ffffffff 00000000 00000046
d0000000 8000046d 00000000 ffffffff
00000000 00000046 e0000000 8000046e
00000000 ffffffff 00000000 00000046
f0000000 8000046f 00000000 ffffffff
00000000 00000047 00000000 80000470
00000000 ffffffff 00000000 00000047
10000000 80000471 00000000 ffffffff
00000000 00000047 20000000 80000472
00000000 ffffffff 00000000 00000047
30000000 80000473 00000000 ffffffff
00000000 00000047 40000000 80000474
00000000 ffffffff 00000000 00000047
50000000 80000475 00000000 ffffffff
00000000 00000047 60000000 80000476
00000000 ffffffff 00000000 00000047
70000000 80000477 00000000 ffffffff
00000000 00000047 80000000 80000478
00000000 ffffffff 00000000 00000047
90000000 80000479 00000000 ffffffff
00000000 00000047 a0000000 8000047a
00000000 ffffffff 00000000 00000047
b0000000 8000047b 00000000 ffffffff
00000000 00000047 c0000000 8000047c
00000000 ffffffff 00000000 00000047
d0000000 8000047d 00000000 ffffffff
00000000 00000047 e0000000 8000047e
00000000 ffffffff 00000000 00000047
f0000000 8000047f 00000000 ffffffff
00000000 00000048 00000000 80000480
00000000 ffffffff 00000000 00000048
10000000 80000481 00000000 ffffffff
00000000 00000048 20000000 80000482
00000000 ffffffff 00000000 00000048
30000000 80000483 00000000 ffffffff
00000000 00000048 40000000 80000484
00000000 ffffffff 00000000 00000048
50000000 80000485 00000000 ffffffff
00000000 00000048 60000000 80000486
00000000 ffffffff 00000000 00000048
70000000 80000487 00000000 ffffffff
00000000 00000048 80000000 80000488
00000000 ffffffff 00000000 00000048
90000000 80000489 00000000 ffffffff
00000000 00000048 a0000000 8000048a
00000000 ffffffff 00000000 00000048
b0000000 8000048b 00000000 ffffffff
00000000 00000048 c0000000 8000048c
00000000 ffffffff 00000000 00000048
d0000000 8000048d 00000000 ffffffff
00000000 00000048 e0000000 8000048e
00000000 ffffffff 00000000 00000048
f0000000 8000048f 00000000 ffffffff
00000000 00000049 00000000 80000490
00000000 ffffffff 00000000 00000049
10000000 80000491 00000000 ffffffff
00000000 00000049 20000000 80000492
00000000 ffffffff 00000000 00000049
30000000 80000493 00000000 ffffffff
00000000 00000049 40000000 80000494
00000000 ffffffff 00000000 00000049
50000000 80000495 00000000 ffffffff
00000000 00000049 60000000 80000496
00000000 ffffffff 00000000 00000049
70000000 80000497 00000000 ffffffff
00000000 00000049 80000000 80000498
00000000 ffffffff 00000000 00000049
90000000 80000499 00000000 ffffffff
00000000 00000049 a0000000 8000049a
00000000 ffffffff 00000000 00000049
b0000000 8000049b 00000000 ffffffff
00000000 00000049 c0000000 8000049c
00000000 ffffffff 00000000 00000049
d0000000 8000049d 00000000 ffffffff
00000000 00000049 e0000000 8000049e
00000000 ffffffff 00000000 00000049
f0000000 8000049f 00000000 ffffffff
00000000 0000004a 00000000 800004a0
00000000 ffffffff 00000000 0000004a
10000000 800004a1 00000000 ffffffff
00000000 0000004a 20000000 800004a2
00000000 ffffffff 00000000 0000004a
30000000 800004a3 00000000 ffffffff
00000000 0000004a 40000000 800004a4
00000000 ffffffff 00000000 0000004a
50000000 800004a5 00000000 ffffffff
00000000 0000004a 60000000 800004a6
00000000 ffffffff 00000000 0000004a
70000000 800004a7 00000000 ffffffff
00000000 0000004a 80000000 800004a8
00000000 ffffffff 00000000 0000004a
90000000 800004a9 00000000 ffffffff
00000000 0000004a a0000000 800004aa
00000000 ffffffff 00000000 0000004a
b0000000 800004ab 00000000 ffffffff
00000000 0000004a c0000000 800004ac
00000000 ffffffff 00000000 0000004a
d0000000 800004ad 00000000 ffffffff
00000000 0000004a e0000000 800004ae
00000000 ffffffff 00000000 0000004a
f0000000 800004af 00000000 ffffffff
00000000 0000004b 00000000 800004b0
00000000 ffffffff 00000000 0000004b
10000000 800004b1 00000000 ffffffff
00000000 0000004b 20000000 800004b2
00000000 ffffffff 00000000 0000004b
30000000 800004b3 00000000 ffffffff
00000000 0000004b 40000000 800004b4
00000000 ffffffff 00000000 0000004b
50000000 800004b5 00000000 ffffffff
00000000 0000004b 60000000 800004b6
00000000 ffffffff 00000000 0000004b
70000000 800004b7 00000000 ffffffff
00000000 0000004b 80000000 800004b8
00000000 ffffffff 00000000 0000004b
90000000 800004b9 00000000 ffffffff
00000000 0000004b a0000000 800004ba
00000000 ffffffff 00000000 0000004b
b0000000 800004bb 00000000 ffffffff
00000000 0000004b c0000000 800004bc
00000000 ffffffff 00000000 0000004b
d0000000 800004bd 00000000 ffffffff
00000000 0000004b e0000000 800004be
00000000 ffffffff 00000000 0000004b
f0000000 800004bf 00000000 ffffffff
00000000 0000004c 00000000 800004c0
00000000 ffffffff 00000000 0000004c
10000000 800004c1 00000000 ffffffff
00000000 0000004c 20000000 800004c2
00000000 ffffffff 00000000 0000004c
30000000 800004c3 00000000 ffffffff
00000000 0000004c 40000000 800004c4
00000000 ffffffff 00000000 0000004c
50000000 800004c5 00000000 ffffffff
00000000 0000004c 60000000 800004c6
00000000 ffffffff 00000000 0000004c
70000000 800004c7 00000000 ffffffff
00000000 0000004c 80000000 800004c8
00000000 ffffffff 00000000 0000004c
90000000 800004c9 00000000 ffffffff
00000000 0000004c a0000000 800004ca
00000000 ffffffff 00000000 0000004c
b0000000 800004cb 00000000 ffffffff
00000000 0000004c c0000000 800004cc
00000000 ffffffff 00000000 0000004c
d0000000 800004cd 00000000 ffffffff
00000000 0000004c e0000000 800004ce
00000000 ffffffff 00000000 0000004c
f0000000 800004cf 00000000 ffffffff
00000000 0000004d 00000000 800004d0
00000000 ffffffff 00000000 0000004d
10000000 800004d1 00000000 ffffffff
00000000 0000004d 20000000 800004d2
00000000 ffffffff 00000000 0000004d
30000000 800004d3 00000000 ffffffff
00000000 0000004d 40000000 800004d4
00000000 ffffffff 00000000 0000004d
50000000 800004d5 00000000 ffffffff
00000000 0000004d 60000000 800004d6
00000000 ffffffff 00000000 0000004d
70000000 800004d7 00000000 ffffffff
00000000 0000004d 80000000 800004d8
00000000 ffffffff 00000000 0000004d
90000000 800004d9 00000000 ffffffff
00000000 0000004d a0000000 800004da
00000000 ffffffff 00000000 0000004d
b0000000 800004db 00000000 ffffffff
00000000 0000004d c0000000 800004dc
00000000 ffffffff 00000000 0000004d
d0000000 800004dd 00000000 ffffffff
00000000 0000004d e0000000 800004de
00000000 ffffffff 00000000 0000004d
f0000000 800004df 00000000 ffffffff
00000000 0000004e 00000000 800004e0
00000000 ffffffff 00000000 0000004e
10000000 800004e1 00000000 ffffffff
00000000 0000004e 20000000 800004e2
00000000 ffffffff 00000000 0000004e
30000000 800004e3 00000000 ffffffff
00000000 0000004e 40000000 800004e4
00000000 ffffffff 00000000 0000004e
50000000 800004e5 00000000 ffffffff
00000000 0000004e 60000000 800004e6
00000000 ffffffff 00000000 0000004e
70000000 800004e7 00000000 ffffffff
00000000 0000004e 80000000 800004e8
00000000 ffffffff 00000000 0000004e
90000000 800004e9 00000000 ffffffff
00000000 0000004e a0000000 800004ea
00000000 ffffffff 00000000 0000004e
b0000000 800004eb 00000000 ffffffff
00000000 0000004e c0000000 800004ec
00000000 ffffffff 00000000 0000004e
d0000000 800004ed 00000000 ffffffff
00000000 0000004e e0000000 800004ee
00000000 ffffffff 00000000 0000004e
f0000000 800004ef 00000000 ffffffff
00000000 0000004f 00000000 800004f0
00000000 ffffffff 00000000 0000004f
10000000 800004f1 00000000 ffffffff
00000000 0000004f 20000000 800004f2
00000000 ffffffff 00000000 0000004f
30000000 800004f3 00000000 ffffffff
00000000 0000004f 40000000 800004f4
00000000 ffffffff 00000000 0000004f
50000000 800004f5 00000000 ffffffff
00000000 0000004f 60000000 800004f6
00000000 ffffffff 00000000 0000004f
70000000 800004f7 00000000 ffffffff
00000000 0000004f 80000000 800004f8
00000000 ffffffff 00000000 0000004f
90000000 800004f9 00000000 ffffffff
00000000 0000004f a0000000 800004fa
00000000 ffffffff 00000000 0000004f
b0000000 800004fb 00000000 ffffffff
00000000 0000004f c0000000 800004fc
00000000 ffffffff 00000000 0000004f
d0000000 800004fd 00000000 ffffffff
00000000 0000004f e0000000 800004fe
00000000 ffffffff 00000000 0000004f
f0000000 800004ff 00000000 ffffffff
00000000 00000050 00000000 80000500
00000000 ffffffff 00000000 00000050
10000000 80000501 00000000 ffffffff
00000000 00000050 20000000 80000502
00000000 ffffffff 00000000 00000050
30000000 80000503 00000000 ffffffff
00000000 00000050 40000000 80000504
00000000 ffffffff 00000000 00000050
50000000 80000505 00000000 ffffffff
00000000 00000050 60000000 80000506
00000000 ffffffff 00000000 00000050
70000000 80000507 00000000 ffffffff
00000000 00000050 80000000 80000508
00000000 ffffffff 00000000 00000050
90000000 80000509 00000000 ffffffff
00000000 00000050 a0000000 8000050a
00000000 ffffffff 00000000 00000050
b0000000 8000050b 00000000 ffffffff
00000000 00000050 c0000000 8000050c
00000000 ffffffff 00000000 00000050
d0000000 8000050d 00000000 ffffffff
00000000 00000050 e0000000 8000050e
00000000 ffffffff 00000000 00000050
f0000000 8000050f 00000000 ffffffff
00000000 00000051 00000000 80000510
00000000 ffffffff 00000000 00000051
10000000 80000511 00000000 ffffffff
00000000 00000051 20000000 80000512
00000000 ffffffff 00000000 00000051
30000000 80000513 00000000 ffffffff
00000000 00000051 40000000 80000514
00000000 ffffffff 00000000 00000051
50000000 80000515 00000000 ffffffff
00000000 00000051 60000000 80000516
00000000 ffffffff 00000000 00000051
70000000 80000517 00000000 ffffffff
00000000 00000051 80000000 80000518
00000000 ffffffff 00000000 00000051
90000000 80000519 00000000 ffffffff
00000000 00000051 a0000000 8000051a
00000000 ffffffff 00000000 00000051
b0000000 8000051b 00000000 ffffffff
00000000 00000051 c0000000 8000051c
00000000 ffffffff 00000000 00000051
d0000000 8000051d 00000000 ffffffff
00000000 00000051 e0000000 8000051e
00000000 ffffffff 00000000 00000051
f0000000 8000051f 00000000 ffffffff
00000000 00000052 00000000 80000520
00000000 ffffffff 00000000 00000052
10000000 80000521 00000000 ffffffff
00000000 00000052 20000000 80000522
00000000 ffffffff 00000000 00000052
30000000 80000523 00000000 ffffffff
00000000 00000052 40000000 80000524
00000000 ffffffff 00000000 00000052
50000000 80000525 00000000 ffffffff
00000000 00000052 60000000 80000526
00000000 ffffffff 00000000 00000052
70000000 80000527 00000000 ffffffff
00000000 00000052 80000000 80000528
00000000 ffffffff 00000000 00000052
90000000 80000529 00000000 ffffffff
00000000 00000052 a0000000 8000052a
00000000 ffffffff 00000000 00000052
b0000000 8000052b 00000000 ffffffff
00000000 00000052 c0000000 8000052c
00000000 ffffffff 00000000 00000052
d0000000 8000052d 00000000 ffffffff
00000000 00000052 e0000000 8000052e
00000000 ffffffff 00000000 00000052
f0000000 8000052f 00000000 ffffffff
00000000 00000053 00000000 80000530
00000000 ffffffff 00000000 00000053
10000000 80000531 00000000 ffffffff
00000000 00000053 20000000 80000532
00000000 ffffffff 00000000 00000053
30000000 80000533 00000000 ffffffff
00000000 00000053 40000000 80000534
00000000 ffffffff 00000000 00000053
50000000 80000535 00000000 ffffffff
00000000 00000053 60000000 80000536
00000000 ffffffff 00000000 00000053
70000000 80000537 00000000 ffffffff
00000000 00000053 80000000 80000538
00000000 ffffffff 00000000 00000053
90000000 80000539 00000000 ffffffff
00000000 00000053 a0000000 8000053a
00000000 ffffffff 00000000 00000053
b0000000 8000053b 00000000 ffffffff
00000000 00000053 c0000000 8000053c
00000000 ffffffff 00000000 00000053
d0000000 8000053d 00000000 ffffffff
00000000 00000053 e0000000 8000053e
00000000 ffffffff 00000000 00000053
f0000000 8000053f 00000000 ffffffff
00000000 00000054 00000000 80000540
00000000 ffffffff 00000000 00000054
10000000 80000541 00000000 ffffffff
00000000 00000054 20000000 80000542
00000000 ffffffff 00000000 00000054
30000000 80000543 00000000 ffffffff
00000000 00000054 40000000 80000544
00000000 ffffffff 00000000 00000054
50000000 80000545 00000000 ffffffff
00000000 00000054 60000000 80000546
00000000 ffffffff 00000000 00000054
70000000 80000547 00000000 ffffffff
00000000 00000054 80000000 80000548
00000000 ffffffff 00000000 00000054
90000000 80000549 00000000 ffffffff
00000000 00000054 a0000000 8000054a
00000000 ffffffff 00000000 00000054
b0000000 8000054b 00000000 ffffffff
00000000 00000054 c0000000 8000054c
00000000 ffffffff 00000000 00000054
d0000000 8000054d 00000000 ffffffff
00000000 00000054 e0000000 8000054e
00000000 ffffffff 00000000 00000054
f0000000 8000054f 00000000 ffffffff
00000000 00000055 00000000 80000550
00000000 ffffffff 00000000 00000055
10000000 80000551 00000000 ffffffff
00000000 00000055 20000000 80000552
00000000 ffffffff 00000000 00000055
30000000 80000553 00000000 ffffffff
00000000 00000055 40000000 80000554
00000000 ffffffff 00000000 00000055
50000000 80000555 00000000 ffffffff
00000000 00000055 60000000 80000556
00000000 ffffffff 00000000 00000055
70000000 80000557 00000000 ffffffff
00000000 00000055 80000000 80000558
00000000 ffffffff 00000000 00000055
90000000 80000559 00000000 ffffffff
00000000 00000055 a0000000 8000055a
00000000 ffffffff 00000000 00000055
b0000000 8000055b 00000000 ffffffff
00000000 00000055 c0000000 8000055c
00000000 ffffffff 00000000 00000055
d0000000 8000055d 00000000 ffffffff
00000000 00000055 e0000000 8000055e
00000000 ffffffff 00000000 00000055
f0000000 8000055f 00000000 ffffffff
00000000 00000056 00000000 80000560
00000000 ffffffff 00000000 00000056
10000000 80000561 00000000 ffffffff
00000000 00000056 20000000 80000562
00000000 ffffffff 00000000 00000056
30000000 80000563 00000000 ffffffff
00000000 00000056 40000000 80000564
00000000 ffffffff 00000000 00000056
50000000 80000565 00000000 ffffffff
00000000 00000056 60000000 80000566
00000000 ffffffff 00000000 00000056
70000000 80000567 00000000 ffffffff
00000000 00000056 80000000 80000568
00000000 ffffffff 00000000 00000056
90000000 80000569 00000000 ffffffff
00000000 00000056 a0000000 8000056a
00000000 ffffffff 00000000 00000056
b0000000 8000056b 00000000 ffffffff
00000000 00000056 c0000000 8000056c
00000000 ffffffff 00000000 00000056
d0000000 8000056d 00000000 ffffffff
00000000 00000056 e0000000 8000056e
00000000 ffffffff 00000000 00000056
f0000000 8000056f 00000000 ffffffff
00000000 00000057 00000000 80000570
00000000 ffffffff 00000000 00000057
10000000 80000571 00000000 ffffffff
00000000 00000057 20000000 80000572
00000000 ffffffff 00000000 00000057
30000000 80000573 00000000 ffffffff
00000000 00000057 40000000 80000574
00000000 ffffffff 00000000 00000057
50000000 80000575 00000000 ffffffff
00000000 00000057 60000000 80000576
00000000 ffffffff 00000000 00000057
70000000 80000577 00000000 ffffffff
00000000 00000057 80000000 80000578
00000000 ffffffff 00000000 00000057
90000000 80000579 00000000 ffffffff
00000000 00000057 a0000000 8000057a
00000000 ffffffff 00000000 00000057
b0000000 8000057b 00000000 ffffffff
00000000 00000057 c0000000 8000057c
00000000 ffffffff 00000000 00000057
d0000000 8000057d 00000000 ffffffff
00000000 00000057 e0000000 8000057e
00000000 ffffffff 00000000 00000057
f0000000 8000057f 00000000 ffffffff
00000000 00000058 00000000 80000580
00000000 ffffffff 00000000 00000058
10000000 80000581 00000000 ffffffff
00000000 00000058 20000000 80000582
00000000 ffffffff 00000000 00000058
30000000 80000583 00000000 ffffffff
00000000 00000058 40000000 80000584
00000000 ffffffff 00000000 00000058
50000000 80000585 00000000 ffffffff
00000000 00000058 60000000 80000586
00000000 ffffffff 00000000 00000058
70000000 80000587 00000000 ffffffff
00000000 00000058 80000000 80000588
00000000 ffffffff 00000000 00000058
90000000 80000589 00000000 ffffffff
00000000 00000058 a0000000 8000058a
00000000 ffffffff 00000000 00000058
b0000000 8000058b 00000000 ffffffff
00000000 00000058 c0000000 8000058c
00000000 ffffffff 00000000 00000058
d0000000 8000058d 00000000 ffffffff
00000000 00000058 e0000000 8000058e
00000000 ffffffff 00000000 00000058
f0000000 8000058f 00000000 ffffffff
00000000 00000059 00000000 80000590
00000000 ffffffff 00000000 00000059
10000000 80000591 00000000 ffffffff
00000000 00000059 20000000 80000592
00000000 ffffffff 00000000 00000059
30000000 80000593 00000000 ffffffff
00000000 00000059 40000000 80000594
00000000 ffffffff 00000000 00000059
50000000 80000595 00000000 ffffffff
00000000 00000059 60000000 80000596
00000000 ffffffff 00000000 00000059
70000000 80000597 00000000 ffffffff
00000000 00000059 80000000 80000598
00000000 ffffffff 00000000 00000059
90000000 80000599 00000000 ffffffff
00000000 00000059 a0000000 8000059a
00000000 ffffffff 00000000 00000059
b0000000 8000059b 00000000 ffffffff
00000000 00000059 c0000000 8000059c
00000000 ffffffff 00000000 00000059
d0000000 8000059d 00000000 ffffffff
00000000 00000059 e0000000 8000059e
00000000 ffffffff 00000000 00000059
f0000000 8000059f 00000000 ffffffff
00000000
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory #
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # cat ~/dmesg.log
[ 0.000000] Linux version 4.4.63-2.1.wi91275.sp3.055a_27-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Tue May 2 10:56:31 EDT 2017 (27e2849)
[ 0.000000] Node 0 Memory: 0x0-0x510000000
[ 0.000000] Node 1 Memory: 0x510000000-0x5c0000000
[ 0.000000] Node 6 Memory: 0x5c0000000-0x26b0000000
[ 0.000000] Node 7 Memory: 0x26b0000000-0x3c00000000
[ 0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x50fffffff]
[ 0.000000] numa: NODE_DATA [mem 0x50fff6100-0x50fffffff]
[ 0.000000] numa: Initmem setup node 1 [mem 0x510000000-0x5bfffffff]
[ 0.000000] numa: NODE_DATA [mem 0x5bfff6100-0x5bfffffff]
[ 0.000000] numa: Initmem setup node 6 [mem 0x5c0000000-0x26afffffff]
[ 0.000000] numa: NODE_DATA [mem 0x26afff6100-0x26afffffff]
[ 0.000000] numa: Initmem setup node 7 [mem 0x26b0000000-0x3bffffffff]
[ 0.000000] numa: NODE_DATA [mem 0x3bffbe2100-0x3bffbebfff]
[ 0.000000] Section 15357 and 15359 (node 7) have a circular dependency on usemap and pgdat allocations
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000
[ 0.000000] Memory hole size: 0MB
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000050fffffff]
[ 0.000000] node 1: [mem 0x0000000510000000-0x00000005bfffffff]
[ 0.000000] node 6: [mem 0x00000005c0000000-0x00000026afffffff]
[ 0.000000] node 7: [mem 0x00000026b0000000-0x0000003bffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000050fffffff]
[ 0.000000] On node 0 totalpages: 331776
[ 0.000000] DMA zone: 324 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 331776 pages, LIFO batch:1
[ 0.000000] Initmem setup node 1 [mem 0x0000000510000000-0x00000005bfffffff]
[ 0.000000] On node 1 totalpages: 45056
[ 0.000000] DMA zone: 44 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 45056 pages, LIFO batch:1
[ 0.000000] Initmem setup node 6 [mem 0x00000005c0000000-0x00000026afffffff]
[ 0.000000] On node 6 totalpages: 2158592
[ 0.000000] DMA zone: 2108 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 2158592 pages, LIFO batch:1
[ 0.000000] Initmem setup node 7 [mem 0x00000026b0000000-0x0000003bffffffff]
[ 0.000000] On node 7 totalpages: 1396736
[ 0.000000] DMA zone: 1364 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 1396736 pages, LIFO batch:1
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory #
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-07 17:28 ` RESEND " Michael Bringmann
@ 2017-06-13 10:45 ` Michael Ellerman
2017-06-13 22:21 ` Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Michael Ellerman @ 2017-06-13 10:45 UTC (permalink / raw)
To: Michael Bringmann, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
> Here is the information from 2 different kernels. I have not been able to retrieve
> the information matching yesterday's attachments, yet, as those dumps were
> acquired in April.
>
> Attached please find 2 dumps of similar material from kernels running with my
> current patches (Linux 4.4, Linux 4.12).
OK thanks.
I'd actually like to see the dmesg output from a kernel *without* your
patches.
Looking at the device tree properties:
ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
ibm,associativity-lookup-arrays
00000004 = 4 arrays
00000004 = of 4 entries each
00000000 00000000 00000000 00000000
00000000 00000000 00000001 00000001
00000000 00000003 00000006 00000006
00000000 00000003 00000007 00000007
Which does tell us that nodes 0, 1, 6 and 7 exist.
So your idea of looking at that and setting any node found in there
online should work.
My only worry is that behaviour appears to be completely undocumented in
PAPR, ie. PAPR explicitly says that property only needs to contain
values for LMBs present at boot.
But possibly we can talk to the PowerVM/PAPR guys and have that changed
so that it becomes something we can rely on.
cheers
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-13 10:45 ` Michael Ellerman
@ 2017-06-13 22:21 ` Michael Bringmann
[not found] ` <CAKTCnz==4vA4PevndTLF7p2PPKCM8ZkqpuCXPp9kFcjn+CqUKA@mail.gmail.com>
0 siblings, 1 reply; 33+ messages in thread
From: Michael Bringmann @ 2017-06-13 22:21 UTC (permalink / raw)
To: Michael Ellerman, Reza Arbab
Cc: Balbir Singh, linux-kernel, Paul Mackerras, Aneesh Kumar K.V,
Bharata B Rao, Shailendra Singh, Thomas Gleixner, linuxppc-dev,
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
On a related note, we are discussing the addition of 2 new device-tree properties
with Pete Heyrman and his fellows that should simplify the determination of the
set of required nodes.
* One property would provide the total/max number of nodes needed by the kernel
on the current hardware.
* A second property would provide the total/max number of nodes that the kernel
could use on any system to which it could be migrated.
These properties aren't available, yet, and it takes time to define new properties
in the PAPR and have them implemented in pHyp and the kernel. As an intermediary
step, the systems which are doing a lot of dynamic hot-add/hot-remove configuration
could provide equivalent information to the PowerPC kernel with a command line
parameter. The 'numa.c' code would then read this value and fill in the necessary
entries in the 'node_possible_map'.
Would you foresee any problems with using such a feature?
Thanks.
On 06/13/2017 05:45 AM, Michael Ellerman wrote:
> Michael Bringmann <mwb@linux.vnet.ibm.com> writes:
>
>> Here is the information from 2 different kernels. I have not been able to retrieve
>> the information matching yesterday's attachments, yet, as those dumps were
>> acquired in April.
>>
>> Attached please find 2 dumps of similar material from kernels running with my
>> current patches (Linux 4.4, Linux 4.12).
>
> OK thanks.
>
> I'd actually like to see the dmesg output from a kernel *without* your
> patches.
>
> Looking at the device tree properties:
>
> ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays
> ibm,associativity-lookup-arrays
> 00000004 = 4 arrays
> 00000004 = of 4 entries each
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000001 00000001
> 00000000 00000003 00000006 00000006
> 00000000 00000003 00000007 00000007
>
>
> Which does tell us that nodes 0, 1, 6 and 7 exist.
>
> So your idea of looking at that and setting any node found in there
> online should work.
>
> My only worry is that behaviour appears to be completely undocumented in
> PAPR, ie. PAPR explicitly says that property only needs to contain
> values for LMBs present at boot.
>
> But possibly we can talk to the PowerVM/PAPR guys and have that changed
> so that it becomes something we can rely on.
>
> cheers
>
>
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
[not found] ` <CAKTCnz==4vA4PevndTLF7p2PPKCM8ZkqpuCXPp9kFcjn+CqUKA@mail.gmail.com>
@ 2017-06-14 5:27 ` Balbir Singh
2017-06-14 13:41 ` Michael Bringmann
0 siblings, 1 reply; 33+ messages in thread
From: Balbir Singh @ 2017-06-14 5:27 UTC (permalink / raw)
To: Michael Bringmann
Cc: Michael Ellerman, Reza Arbab, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, open list:LINUX FOR POWERPC (32-BIT AND 64-BIT),
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote:
>
>
> On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com>
> wrote:
>>
>> On a related note, we are discussing the addition of 2 new device-tree
>> properties
>> with Pete Heyrman and his fellows that should simplify the determination
>> of the
>> set of required nodes.
>>
>> * One property would provide the total/max number of nodes needed by the
>> kernel
>> on the current hardware.
>
>
Yes, that would be nice to have
>
>>
>> * A second property would provide the total/max number of nodes that the
>> kernel
>> could use on any system to which it could be migrated.
>>
>
Not sure about this one, are you suggesting more memory can be added
depending on the migration target?
>
>
>>
>> These properties aren't available, yet, and it takes time to define new
>> properties
>> in the PAPR and have them implemented in pHyp and the kernel. As an
>> intermediary
>> step, the systems which are doing a lot of dynamic hot-add/hot-remove
>> configuration
>> could provide equivalent information to the PowerPC kernel with a command
>> line
>> parameter. The 'numa.c' code would then read this value and fill in the
>> necessary
>> entries in the 'node_possible_map'.
>>
>> Would you foresee any problems with using such a feature?
>
>
Sorry my mailer goofed up, resending
Balbir Singh
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: RESEND Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc
2017-06-14 5:27 ` Balbir Singh
@ 2017-06-14 13:41 ` Michael Bringmann
0 siblings, 0 replies; 33+ messages in thread
From: Michael Bringmann @ 2017-06-14 13:41 UTC (permalink / raw)
To: Balbir Singh
Cc: Michael Ellerman, Reza Arbab, linux-kernel, Paul Mackerras,
Aneesh Kumar K.V, Bharata B Rao, Shailendra Singh,
Thomas Gleixner, open list:LINUX FOR POWERPC (32-BIT AND 64-BIT),
Sebastian Andrzej Siewior, Michael Bringmann from Kernel Team
Hello:
On 06/14/2017 12:27 AM, Balbir Singh wrote:
> On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote:
>>
>>
>> On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com>
>> wrote:
>>>
>>> On a related note, we are discussing the addition of 2 new device-tree
>>> properties
>>> with Pete Heyrman and his fellows that should simplify the determination
>>> of the
>>> set of required nodes.
>>>
>>> * One property would provide the total/max number of nodes needed by the
>>> kernel
>>> on the current hardware.
>>
>>
>
> Yes, that would be nice to have
>
>>
>>>
>>> * A second property would provide the total/max number of nodes that the
>>> kernel
>>> could use on any system to which it could be migrated.
>>>
>>
>
> Not sure about this one, are you suggesting more memory can be added
> depending on the migration target?
We would use only one of these numbers to allocate nodes. I have only been
on the periphery of the discussions, so I can not communicate the full
reasoning as to why both measures would be needed. We would like to have
the first number for node allocation/initialization, but if only the second
value were provided, we would likely need to use it.
>>
>>
>>>
>>> These properties aren't available, yet, and it takes time to define new
>>> properties
>>> in the PAPR and have them implemented in pHyp and the kernel. As an
>>> intermediary
>>> step, the systems which are doing a lot of dynamic hot-add/hot-remove
>>> configuration
>>> could provide equivalent information to the PowerPC kernel with a command
>>> line
>>> parameter. The 'numa.c' code would then read this value and fill in the
>>> necessary
>>> entries in the 'node_possible_map'.
>>>
>>> Would you foresee any problems with using such a feature?
>>
>>
>
> Sorry my mailer goofed up, resending
>
> Balbir Singh
>
Thanks.
--
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line 363-5196
External: (512) 286-5196
Cell: (512) 466-0650
mwb@linux.vnet.ibm.com
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2017-06-14 13:41 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-23 15:15 [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc Michael Bringmann
2017-05-23 15:52 ` Reza Arbab
2017-05-23 20:05 ` Michael Bringmann
2017-05-23 21:49 ` Reza Arbab
2017-05-23 22:41 ` Michael Bringmann
2017-05-23 22:44 ` Michael Bringmann
2017-05-24 11:19 ` Michael Ellerman
2017-05-24 23:55 ` Michael Bringmann
2017-05-25 6:19 ` Michael Ellerman
2017-05-25 13:40 ` Michael Bringmann
2017-05-25 15:10 ` Reza Arbab
2017-05-25 15:26 ` Michael Bringmann
2017-05-26 3:46 ` Balbir Singh
2017-05-26 3:46 ` Michael Ellerman
2017-05-26 12:31 ` Michael Bringmann
2017-05-26 14:31 ` Reza Arbab
2017-05-29 5:32 ` Michael Ellerman
2017-05-31 14:05 ` Michael Bringmann
2017-06-01 9:36 ` Michael Ellerman
2017-06-01 21:33 ` Reza Arbab
2017-06-07 8:06 ` Balbir Singh
2017-06-07 12:07 ` Michael Ellerman
2017-06-02 5:24 ` Michael Bringmann
2017-06-06 9:48 ` Michael Ellerman
2017-06-06 16:16 ` Michael Bringmann
2017-06-07 12:08 ` Michael Ellerman
2017-06-07 17:28 ` RESEND " Michael Bringmann
2017-06-13 10:45 ` Michael Ellerman
2017-06-13 22:21 ` Michael Bringmann
[not found] ` <CAKTCnz==4vA4PevndTLF7p2PPKCM8ZkqpuCXPp9kFcjn+CqUKA@mail.gmail.com>
2017-06-14 5:27 ` Balbir Singh
2017-06-14 13:41 ` Michael Bringmann
2017-05-24 14:36 ` Reza Arbab
2017-05-24 23:41 ` Michael Bringmann
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).